-
GRPO Step by Step
Group Relative Policy Optimization for reinforcement learning with verifiable rewards (RLVR), with formulas and worked examples.
-
The Details of Flash Attention
From the online-softmax derivation to eight CUDA kernels — a 121.8× speedup, and the two optimizations that made things slower.
-
Tensor Parallel + Sequence Parallel — A Deep Dive
-
ZeRO-1 Distributed Optimizer - A Deep Dive
-
Optimizing softmax on GPU