Group Relative Policy Optimization for reinforcement learning with verifiable rewards (RLVR), with formulas and worked examples.
One-sentence summary: GRPO is PPO’s clipped update, except the learned critic is replaced by the average reward of several answers to the same question, plus a KL “leash” that keeps the model close to where it started.
GRPO was introduced in the DeepSeekMath paper and later used to train DeepSeek-R1. In reinforcement learning with verifiable rewards (RLVR) you have:
The goal is to change the weights \(\theta\) so the model produces high-reward answers more often. The reward comes from a rule-based checker rather than a learned reward model. That is what “verifiable” means.
Standard PPO needs two networks:
For LLMs, the critic is usually another model as big as the policy. That roughly doubles memory and compute, and it is hard to estimate the “value” of a half-written answer.
GRPO’s idea: sample a group of answers to the same question and use the group’s own mean reward as the baseline. No critic is needed.
For each question \(q\), sample \(G\) answers from the old policy, which is the model as it was before this update:
\[\lbrace o_1, o_2, \dots, o_G \rbrace \sim \pi_{\theta_{old}}(O \mid q)\]Running example. \(q\) = “What is \(\int_0^1 x^2\,dx\)?”, ground truth \(= 1/3\), \(G = 4\):
| Answer | Final answer | Verifier | Reward \(r_i\) |
|---|---|---|---|
| \(o_1\) | 1/3 | ✅ correct | 1 |
| \(o_2\) | 1/2 | ❌ wrong | 0 |
| \(o_3\) | 1/3 | ✅ correct | 1 |
| \(o_4\) | 0 | ❌ wrong | 0 |
Instead of using a critic, GRPO normalizes each reward within its group:
\[\hat{A}_i = \frac{r_i - \mathrm{mean}(r_1, \dots, r_G)}{\mathrm{std}(r_1, \dots, r_G)}\]With our rewards \([1, 0, 1, 0]\): mean \(= 0.5\), std \(\approx 0.577\) (sample standard deviation, which is what torch.std computes by default).
This is “grading on a curve”: for this question, answers better than the group average are pushed up and worse ones are pushed down. Why both the mean and the std are needed is covered in the normalization deep dive.
A quick sanity check in code:
import torch
r = torch.tensor([1., 0., 1., 0.])
adv = (r - r.mean()) / (r.std() + 1e-4)
print(adv) # tensor([ 0.8659, -0.8659, 0.8659, -0.8659])
Try [1, 1, 1, 1] to see a zero-variance group, or [1, 0, 0, 0] to see a single correct answer get a big advantage.
This is the objective from the DeepSeekMath paper. The expectation is over questions \(q \sim P(Q)\) and groups of answers \(\lbrace o_i \rbrace_{i=1}^{G} \sim \pi_{\theta_{old}}(O \mid q)\):
\[\small \mathcal{J}_{GRPO}(\theta) = \mathbb{E}\Bigg[ \frac{1}{G} \sum_{i=1}^{G} \frac{1}{\lvert o_i \rvert} \sum_{t=1}^{\lvert o_i \rvert} \Big( \min\big( \rho_{i,t}\hat{A}_{i,t},\, \mathrm{clip}(\rho_{i,t}, 1-\varepsilon, 1+\varepsilon)\hat{A}_{i,t} \big) - \beta\, \mathbb{D}_{KL}\big[\pi_\theta \Vert \pi_{ref}\big] \Big) \Bigg]\]The probability ratio is:
\[\rho_{i,t} = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,\lt t})}{\pi_{\theta_{old}}(o_{i,t} \mid q, o_{i,\lt t})}\]The per-token KL estimator, which is always \(\ge 0\) and equals \(0\) only when the two models agree:
\[\begin{aligned} \mathbb{D}_{KL}\big[\pi_\theta \,\Vert\, \pi_{ref}\big] = {} & \frac{\pi_{ref}(o_{i,t} \mid q, o_{i,\lt t})}{\pi_\theta(o_{i,t} \mid q, o_{i,\lt t})} \\[4pt] & - \log \frac{\pi_{ref}(o_{i,t} \mid q, o_{i,\lt t})}{\pi_\theta(o_{i,t} \mid q, o_{i,\lt t})} - 1 \end{aligned}\]| Piece | Meaning |
|---|---|
| \(\mathbb{E}_{q,\ o_i}\) | Average over training questions and the \(G\) answers sampled for each one |
| \(\frac{1}{G}\sum_{i=1}^{G}\) | Average over the answers in the group |
| \(\frac{1}{\lvert o_i \rvert}\sum_{t=1}^{\lvert o_i \rvert}\) | Average over the tokens of answer \(i\), so long answers don’t dominate |
| \(\min(\rho\hat{A},\ \mathrm{clip}(\rho)\hat{A})\) | PPO’s clipped update: reward good tokens and penalize bad ones, but limit how far one batch can move them |
| \(\beta\,\mathbb{D}_{KL}\) | Penalty for drifting too far from the reference model \(\pi_{ref}\) (usually the SFT model before RL) |
\(\rho_{i,t}\) compares how likely the current model is to write the token with how likely the sampling model was. It is needed because GRPO reuses the same samples for several gradient steps while the model keeps changing. See the deep dive.
\(\mathrm{clip}(\rho, 1-\varepsilon, 1+\varepsilon)\) limits the ratio to e.g. \([0.8, 1.2]\) when \(\varepsilon = 0.2\). Together with the \(\min\), this means: once a good token’s probability has gone up enough, or a bad token’s has gone down enough, stop pushing it in this batch.
Example: a token in \(o_1\) (\(\hat{A} = +0.87\)) whose probability went from \(0.5\) to \(0.9\):
\[\begin{aligned} \rho &= \frac{0.9}{0.5} = 1.8 \\ \mathrm{clip}(1.8,\ 0.8,\ 1.2) &= 1.2 \\ \min(1.8 \times 0.87,\ 1.2 \times 0.87) &= \min(1.57,\ 1.04) \\ &= 1.04 \end{aligned}\]Maximizing reward alone can lead to strange or degenerate text that happens to fool the verifier. The KL term keeps the model close to \(\pi_{ref}\), so it stays fluent. A larger \(\beta\) keeps it closer to the reference, and a smaller \(\beta\) lets it chase reward more.
The \(\frac{1}{G}\) averages over answers. The \(\frac{1}{\lvert o_i \rvert}\) averages over tokens, so each answer counts about equally regardless of length. This has a side effect on long wrong answers, covered in response length.
| Answer | Reward | \(\hat{A}\) | Effect on the model |
|---|---|---|---|
| \(o_1\) “…1/3” | 1 | +0.87 | Make these tokens more likely (with clipping) |
| \(o_2\) “…1/2” | 0 | −0.87 | Make these tokens less likely (with clipping) |
| \(o_3\) “…1/3” | 1 | +0.87 | More likely |
| \(o_4\) “…0” | 0 | −0.87 | Less likely |
\(t\) is the position of a token inside answer \(i\): the 1st token, the 2nd, and so on up to \(\lvert o_i \rvert\).
The model writes one token at a time, and the update changes each token’s probability separately, so every token needs an advantage value.
This is the usual RLVR setup. The verifier only checks the final answer, so every token in the answer gets the same advantage:
\[\hat{A}_{i,1} = \hat{A}_{i,2} = \dots = \hat{A}_{i,\lvert o_i \rvert} = \frac{r_i - \mathrm{mean}(r)}{\mathrm{std}(r)}\]| \(t\) | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
|---|---|---|---|---|---|---|---|
| token | x^3 | /3 | from | 0 | to 1 | = | 1/3 |
| \(\hat{A}_{1,t}\) | +0.87 | +0.87 | +0.87 | +0.87 | +0.87 | +0.87 | +0.87 |
Weakness: in a wrong answer, every token is penalized, including steps that were actually correct.
A reward model scores each reasoning step. Answer \(i\) has \(K_i\) steps, and step \(j\) ends at token position \(\mathrm{index}(j)\). Normalize each step reward using all step rewards \(R\) in the group:
\[\tilde{r}_i^{\,\mathrm{index}(j)} = \frac{r_i^{\,\mathrm{index}(j)} - \mathrm{mean}(R)}{\mathrm{std}(R)}\]The advantage of token \(t\) is the sum of the normalized rewards of all steps that end at or after \(t\):
\[\hat{A}_{i,t} = \sum_{\mathrm{index}(j) \ge t} \tilde{r}_i^{\,\mathrm{index}(j)}\]A token gets credit for its own step and every later step, but not for earlier steps, because it can’t affect what was already written.
Example (normalized step rewards are made up for illustration):
| Step \(j\) | Tokens | Ends at | Quality | \(\tilde{r}\) |
|---|---|---|---|---|
| 1 | \(t = 1\) to \(3\) “antiderivative is x^3/3” | 3 | ✅ good | +0.5 |
| 2 | \(t = 4\) to \(6\) “plug in 1 and 0” | 6 | ✅ good | +0.4 |
| 3 | \(t = 7\) to \(8\) “= 1/2” | 8 | ❌ arithmetic error | −1.2 |
| \(t\) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
|---|---|---|---|---|---|---|---|---|
| \(\hat{A}_{1,t}\) | −0.3 | −0.3 | −0.3 | −0.8 | −0.8 | −0.8 | −1.2 | −1.2 |
The faulty final step gets the biggest penalty, and the good early steps are penalized much less.
\(\rho_{i,t}\) measures how much the model’s probability for one token has changed since the answers were sampled.
\[\rho_{i,t} = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,\lt t})}{\pi_{\theta_{old}}(o_{i,t} \mid q, o_{i,\lt t})}\]Read it as: “Given the question and the tokens before position \(t\), how likely is the model to write token \(o_{i,t}\) now, compared with when the answer was sampled?”
| Value | Meaning |
|---|---|
| \(\rho_{i,t} = 1\) | No change yet |
| \(\rho_{i,t} \gt 1\) | The new model makes this token more likely |
| \(\rho_{i,t} \lt 1\) | The new model makes this token less likely |
Unlike the outcome advantage, \(\rho_{i,t}\) is almost always different for each token.
On the first step, \(\rho = 1\) but the gradient isn’t zero. When \(\theta = \theta_{old}\), every ratio equals 1, but its gradient is:
\[\nabla_\theta \rho_{i,t} \Big|_{\theta = \theta_{old}} = \frac{\nabla_\theta \pi_\theta(o_{i,t} \mid \cdot)}{\pi_{\theta_{old}}(o_{i,t} \mid \cdot)} = \nabla_\theta \log \pi_\theta(o_{i,t} \mid \cdot)\]So the update is \(\hat{A}_{i,t}\,\nabla_\theta \log \pi_\theta\), the basic policy gradient. On later steps \(\rho\) moves away from 1 and clipping starts to matter.
Here \(\varepsilon = 0.2\).
| Advantage | Ratio | What happens | Gradient? |
|---|---|---|---|
| \(\hat{A} \gt 0\) (good) | \(\rho \le 1.2\) | Keep raising this token’s probability | Yes |
| \(\hat{A} \gt 0\) (good) | \(\rho \gt 1.2\) | Already raised enough, so stop | No (clipped) |
| \(\hat{A} \lt 0\) (bad) | \(\rho \ge 0.8\) | Keep lowering this token’s probability | Yes |
| \(\hat{A} \lt 0\) (bad) | \(\rho \lt 0.8\) | Already lowered enough, so stop | No (clipped) |
Clipping only stops changes that went too far in the intended direction. If a good token became less likely (\(\rho \lt 0.8\) with \(\hat{A} \gt 0\)), nothing is clipped and the gradient pulls it back.
Correct answer \(o_1\), \(\hat{A}_1 = +0.87\):
| \(t\) | Token | \(\pi_{\theta_{old}}\) | \(\pi_\theta\) | \(\rho_{1,t}\) | Clipped \(\rho\) | Term \(= \min(\cdot)\) | Gradient? |
|---|---|---|---|---|---|---|---|
| 1 | x^3 | 0.60 | 0.66 | 1.10 | 1.10 | \(1.10 \times 0.87 = 0.957\) | Yes, keep raising |
| 6 | = | 0.95 | 0.96 | 1.01 | 1.01 | \(1.01 \times 0.87 = 0.879\) | Yes, but tiny change |
| 7 | 1/3 | 0.40 | 0.56 | 1.40 | 1.20 | \(\min(1.218,\ 1.044) = 1.044\) | No, clipped |
Wrong answer \(o_2\), \(\hat{A}_2 = -0.87\):
| \(t\) | Token | \(\pi_{\theta_{old}}\) | \(\pi_\theta\) | \(\rho_{2,t}\) | Clipped \(\rho\) | Term \(= \min(\cdot)\) | Gradient? |
|---|---|---|---|---|---|---|---|
| 6 | = | 0.90 | 0.93 | 1.03 | 1.03 | \(1.03 \times (-0.87) = -0.899\) | Yes, push down |
| 7 | 1/2 | 0.30 | 0.21 | 0.70 | 0.80 | \(\min(-0.609,\ -0.696) = -0.696\) | No, clipped |
An answer’s probability is the product of its token probabilities, so a whole-answer ratio would be:
\[\frac{\pi_\theta(o_i \mid q)}{\pi_{\theta_{old}}(o_i \mid q)} = \prod_{t=1}^{\lvert o_i \rvert} \rho_{i,t}\]For 1,000 tokens each with ratio 1.01, that is \(1.01^{1000} \approx 21{,}000\). Whole-answer ratios explode or vanish quickly. Per-token ratios stay near 1 and can be clipped one at a time. (Newer variants like GSPO use a length-normalized answer-level ratio, but standard GRPO works per token.)
Use log-probabilities for numerical stability: \(\rho_{i,t} = \exp\big(\log \pi_\theta(o_{i,t} \mid \cdot) - \log \pi_{\theta_{old}}(o_{i,t} \mid \cdot)\big)\).
# old_logp: [G, T] log-probs saved at sampling time (no gradient)
# new_logp: [G, T] log-probs from the current model (with gradient)
# ref_logp: [G, T] log-probs from the frozen reference model
# adv: [G, 1] group-normalized advantage, broadcast to every token
# mask: [G, T] 1 for real answer tokens, 0 for padding
# rho_{i,t}
ratio = torch.exp(new_logp - old_logp)
unclipped = ratio * adv
clipped = torch.clamp(ratio, 1 - eps, 1 + eps) * adv
policy_term = torch.min(unclipped, clipped) # [G, T]
# per-token KL estimator, with log_r = log(pi_ref / pi_theta)
log_r = ref_logp - new_logp
kl = torch.exp(log_r) - log_r - 1
per_token = policy_term - beta * kl
# average over tokens (1/|o_i|), then over answers (1/G)
per_answer = (per_token * mask).sum(1) / mask.sum(1)
loss = -per_answer.mean() # minus because we maximize J
Advantage normalization does two separate things, and each solves a different problem:
\[\hat{A}_i = \frac{\overbrace{r_i - \mathrm{mean}(r)}^{\text{1. subtract the mean}}}{\underbrace{\mathrm{std}(r)}_{\text{2. divide by the std}}}\]Problem: raw rewards push every answer up. If we used the raw reward as the advantage, \(\hat{A}_i = r_i\), the update would be:
\[\nabla_\theta \mathcal{J} \approx \frac{1}{G}\sum_{i=1}^{G} r_i \, \nabla_\theta \log \pi_\theta(o_i \mid q)\]Take a reward with a format bonus, \(\lambda = 0.5\):
| Answer | Quality | Raw \(r_i\) | Using raw \(r_i\) | Using \(r_i - \text{mean}\) (mean = 0.75) |
|---|---|---|---|---|
| \(o_1\) | correct + formatted | 1.5 | push up strongly | +0.75, push up |
| \(o_2\) | correct only | 1.0 | push up | +0.25, push up a little |
| \(o_3\) | wrong but formatted | 0.5 | push up ❌ | −0.25, push down ✅ |
| \(o_4\) | neither | 0 | no change ❌ | −0.75, push down ✅ |
With raw rewards, no answer is ever pushed down, and the wrong answer \(o_3\) is actually reinforced. After subtracting the mean, worse-than-average answers get a negative advantage and are pushed down directly.
Problem: question difficulty gets mixed into the signal.
| Question | Rewards | Raw: what the model learns | Mean-subtracted |
|---|---|---|---|
| Easy: “2 + 2 = ?” | \([1, 1, 1, 1]\) | All 4 answers pushed up strongly, even though nothing new was learned | \([0, 0, 0, 0]\): no update, correct |
| Hard: an olympiad problem | \([1, 0, 0, 0]\) | 1 answer pushed up, same strength as each answer above | \([+0.75, -0.25, -0.25, -0.25]\): the rare success stands out |
With raw rewards, the easy question produces 4× more update than the hard one, even though it teaches nothing. The mean acts as a baseline: “what this model usually scores on this question.” Only results better or worse than that count as news.
This baseline is exactly the job PPO’s critic \(V_\psi\) does. GRPO gets it for free from the group average, which is the whole point of the method.
Why a baseline doesn’t make the gradient wrong. For any baseline \(b\) that doesn’t depend on the sampled answer:
\[\begin{aligned} \mathbb{E}_{o \sim \pi_\theta}\big[ b \, \nabla_\theta \log \pi_\theta(o) \big] &= b \sum_{o} \pi_\theta(o) \frac{\nabla_\theta \pi_\theta(o)}{\pi_\theta(o)} \\ &= b \, \nabla_\theta \underbrace{\sum_{o} \pi_\theta(o)}_{=\,1} = b \, \nabla_\theta 1 = 0 \end{aligned}\]Subtracting \(b\) removes zero from the expected gradient, so on average the update points the same way. But it removes a lot of variance (noise), so each batch gives a much more reliable direction.
Technically the group mean includes \(r_i\) itself, so it depends slightly on the sampled answer. With \(G \ge 4\) the effect is small, and in practice it works well.
Problem: different questions have different reward spreads.
| Question | Rewards | Mean-subtracted only | After dividing by std |
|---|---|---|---|
| A (0/1 correctness) | \([1, 0, 1, 0]\) | \([+0.5, -0.5, +0.5, -0.5]\) | \([+0.87, -0.87, +0.87, -0.87]\) |
| B (partial credit) | \([0.6, 0.5, 0.6, 0.5]\) | \([+0.05, -0.05, +0.05, -0.05]\) | \([+0.87, -0.87, +0.87, -0.87]\) |
Both questions have the same pattern: two answers better, two worse. Without the std, question B’s signal is 10× weaker and is effectively ignored. After dividing by the std, both give the same strength of update.
Benefit: the reward’s scale stops mattering. For any scale \(a \gt 0\) and shift \(c\):
\[\begin{aligned} \frac{(a r_i + c) - \mathrm{mean}(a r + c)}{\mathrm{std}(a r + c)} &= \frac{a\,(r_i - \mathrm{mean}(r))}{a\,\mathrm{std}(r)} \\[4pt] &= \frac{r_i - \mathrm{mean}(r)}{\mathrm{std}(r)} \end{aligned}\]Changing rewards from \(\lbrace 0, 1 \rbrace\) to \(\lbrace 0, 10 \rbrace\) gives the same advantages. You can design rewards freely, and the same learning rate and clip range (\(\varepsilon = 0.2\)) keep working, since the clip range assumes advantages of a predictable size.
Benefit: rare outcomes get a strong signal. With \(G = 8\) (sample std):
Hard question, 1 of 8 correct: \(r = [1, 0, 0, 0, 0, 0, 0, 0]\), mean \(= 0.125\), std \(\approx 0.354\)
\[\begin{aligned} \hat{A}_{\text{correct}} &= \frac{1 - 0.125}{0.354} \approx +2.47 \\[4pt] \hat{A}_{\text{wrong}} &= \frac{0 - 0.125}{0.354} \approx -0.35 \end{aligned}\]Easy question, 7 of 8 correct: \(r = [1, 1, 1, 1, 1, 1, 1, 0]\), mean \(= 0.875\), std \(\approx 0.354\)
\[\hat{A}_{\text{correct}} \approx +0.35, \quad \hat{A}_{\text{wrong}} \approx -2.47\]The surprising result gets the big signal: the one success on a hard question is strongly reinforced, and the one failure on an easy question is strongly penalized.
The “Dr. GRPO” paper points out a question-difficulty bias:
So GRPO quietly gives more weight to very easy and very hard questions. A tiny std can also amplify noise, for example from a slightly flaky verifier. When std = 0, the code needs an \(\epsilon\) to avoid dividing by zero, and every advantage ends up 0.
| Method | Advantage | Note |
|---|---|---|
| Original GRPO | \(\dfrac{r_i - \mathrm{mean}(r)}{\mathrm{std}(r)}\) | Same scale for every question; biased toward very easy and very hard questions |
| Dr. GRPO | \(r_i - \mathrm{mean}(r)\) | Removes the difficulty bias; more sensitive to reward scale |
| Batch-level std | \(\dfrac{r_i - \mathrm{mean}_{\text{group}}(r)}{\mathrm{std}_{\text{batch}}(r)}\) | Per-question baseline, one shared scale for the whole batch |
Subtracting the mean is the essential part: it is the baseline that replaces the critic. Dividing by the std is a convenience for keeping scales consistent, and people still debate whether it should be used.
| Step | Problem it solves | Real example |
|---|---|---|
| Subtract the mean | With raw rewards, wrong answers are never pushed down, and easy questions dominate | \([1,1,1,1] \to [0,0,0,0]\): nothing to learn, no update |
| Divide by the std | Signal size depends on reward scale and spread | \([0.6, 0.5, 0.6, 0.5]\) gets the same ±0.87 as \([1, 0, 1, 0]\) |
| Together | A per-question, scale-free “better or worse than usual?” | A lone success on a hard question gets \(+2.47\) |
| Question | Metric |
|---|---|
| Is the model getting better? | Mean training reward, held-out accuracy |
| Is there anything left to learn from? | Fraction of groups where every answer got the same reward |
| Is the model changing too fast? | KL to the reference, clip fraction |
| Is the model breaking down? | Entropy, response length, real samples |
\[\frac{1}{G}\sum_{i=1}^{G} \hat{A}_i = \frac{1}{G}\sum_{i=1}^{G} \frac{r_i - \mathrm{mean}(r)}{\mathrm{std}(r)} = 0\]Don’t trust the loss curve. Advantages are normalized within each group, so they average to zero:
On the first gradient step every \(\rho_{i,t} = 1\), so the policy part of the loss is near 0 whether the model learns or not. Watch reward and evaluation curves instead.
\(B\) is the number of questions in the batch. With a 0/1 reward this is the training accuracy on the questions just sampled. A healthy curve rises noisily and then flattens, so look at a moving average.
A rising reward only shows the model is getting better at pleasing the verifier. Whether that means solving problems correctly is the topic of Part 2.
If all \(G\) answers get the same reward:
\[\begin{aligned} \mathrm{std}(r_1, \dots, r_G) = 0 \quad\Rightarrow\quad \hat{A}_i &= \frac{r_i - \mathrm{mean}(r)}{0 + \epsilon} \\[4pt] &= 0 \ \text{ for every } i \end{aligned}\]Code adds a small \(\epsilon\) (e.g. \(10^{-4}\)) to avoid dividing by zero. Every advantage is zero, so the question contributes nothing to the gradient. If the model solves a question with probability \(p\) per try:
\[P(\text{zero variance}) = p^{G} + (1-p)^{G}\]| Difficulty | \(p\) | \(p^8 + (1-p)^8\) | Meaning (\(G = 8\)) |
|---|---|---|---|
| Very easy | 0.95 | 0.66 | 66% of groups wasted |
| Easy | 0.90 | 0.43 | 43% wasted |
| Medium | 0.50 | 0.008 | Almost every group helps |
| Hard | 0.10 | 0.43 | 43% wasted |
| Very hard | 0.02 | 0.85 | 85% wasted |
Always evaluate on a held-out set. Sample \(n\) answers per question; \(c_j\) is the number correct for question \(j\), out of \(N\) questions.
\[\text{pass@1} = \frac{1}{N} \sum_{j=1}^{N} \frac{c_j}{n}\]pass@k is the chance that at least one of \(k\) answers is correct (unbiased estimator):
\[\text{pass@}k = \mathbb{E}_{\text{questions}}\left[ 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}} \right]\]maj@k: sample \(k\) answers, take the most common final answer, and check it.
Example, one question with \(n = 8\), \(c = 2\):
\[\begin{aligned} \text{pass@1} &= \frac{2}{8} = 0.25 \\[4pt] \text{pass@4} &= 1 - \frac{\binom{6}{4}}{\binom{8}{4}} = 1 - \frac{15}{70} = 0.786 \end{aligned}\]RLVR often raises pass@1 a lot while pass@k for large \(k\) barely moves, and it can even drop. RL mostly makes the model more reliable at things it could already sometimes do. If pass@1 rises while pass@64 falls, the model is losing diversity (see entropy).
where both probabilities are those of token \(o_{i,t}\) given its context.
where \(\cdot\) stands for the context \(q, o_{i,\lt t}\).
Length bias. Because of the \(\frac{1}{\lvert o_i \rvert}\) factor, a wrong answer’s penalty is spread over \(\lvert o_i \rvert\) tokens, so each token in a long wrong answer is punished less than in a short one. This quietly favors long wrong answers. The “Dr. GRPO” paper fixes this by normalizing with a constant instead of \(\lvert o_i \rvert\).
| What you see | Likely cause | What to do |
|---|---|---|
| Training reward ↑, held-out accuracy ↑ | Healthy | Keep going |
| Training reward ↑, held-out flat or ↓ | Reward hacking or overfitting | Read samples, tighten the verifier |
| Reward flat, zero-variance fraction high | Data too easy or too hard | Filter by difficulty |
| Entropy → 0, zero-variance ↑, pass@k ↓ | Entropy collapse | Raise temperature, lower LR, asymmetric clipping |
| KL spikes suddenly | New behavior, maybe a hack | Read samples from that step |
| Length ↑, accuracy flat, truncation ↑ | Rambling or loops | Length penalty, length-bias fix, check verifier |
| Clip fraction high, reward unstable | Updates too big | Lower LR, fewer epochs per batch |
Most important habit: print about 5 real samples every N steps and read them. Every metric can look fine while the model does something silly.
GRPO maximizes the reward you wrote, not what you meant. If the verifier has a loophole that is easier than solving the problem, gradient ascent will find it (Goodhart’s law).
Group normalization amplifies a loophole. If one answer finds a trick the verifier wrongly accepts:
| Answer | Real quality | Verifier reward | \(\hat{A}\) |
|---|---|---|---|
| \(o_1\) | Wrong, but uses the trick | 1 | +1.5 |
| \(o_2\) | Wrong | 0 | −0.5 |
| \(o_3\) | Wrong | 0 | −0.5 |
| \(o_4\) | Wrong | 0 | −0.5 |
(mean = 0.25, sample std = 0.5)
The hack gets the largest advantage in the group and will appear more often in the next batch.
Math
| Hack | Example | Why a naive verifier is fooled |
|---|---|---|
| Multiple answers | \boxed{1/3} ... \boxed{1/2} ... \boxed{0} | Accepts if any box matches |
| Hedging | “The answer is 1/3 or 1/2” | A regex finds “1/3” somewhere |
| No reasoning | <think></think>\boxed{1/3} | Correct on easy questions, teaches nothing |
| Parser confusion | Odd format that crashes the comparison | Verifier bug defaults to “correct” |
Code
| Hack | Example |
|---|---|
| Hardcoding tests | if x == 3: return 9 for every visible test |
| Early exit | sys.exit(0) so the runner reports “passed” |
| Editing tests | Changing or deleting test files if writable |
| Catching everything | try: ... except: pass |
Defense 1: A strict verifier (most important).
import re
from sympy import simplify, sympify
def correctness_reward(response: str, ground_truth: str) -> float:
answers = re.findall(r"\\boxed\{([^{}]*)\}", response)
# Exactly one final answer, no hedging with multiple boxes
if len(answers) != 1:
return 0.0
pred = answers[0].strip()
try:
diff = simplify(sympify(pred) - sympify(ground_truth))
return 1.0 if diff == 0 else 0.0
except Exception:
# Anything unparseable is wrong, never right by default
return 0.0
1/3 vs 0.333) and test them.Defense 2: The KL penalty.
\[\mathcal{J} = \text{(clipped policy term)} - \beta\, \mathbb{D}_{KL}\big[\pi_\theta \,\Vert\, \pi_{ref}\big]\]Strange hacked outputs are usually unlikely under the reference model, so they cost a lot of KL. But \(\beta\) only slows hacking, and a large \(\beta\) also slows real learning. Many recent RLVR setups use a tiny \(\beta\) or \(\beta = 0\) and rely on a strict verifier.
Defense 3: Length limits. Set a maximum length. A truncated answer has no final answer and gets 0. Optionally penalize answers near the limit (DAPO’s “soft overlong” penalty).
Defense 4: An independent check. Evaluate with a different, stricter check than training: another parser, hidden tests, or a human spot-check. Training reward ↑ while the independent check stays flat means you’ve found a hack.
Defense 5: Read the highest-reward samples. Hacks show up first among the answers with the highest reward.
| Layer | Purpose |
|---|---|
| Strict verifier | Close the loophole itself |
| Sandbox and hidden tests | Stop code hacks |
| KL penalty | Make strange outputs costly |
| Length limits | Stop rambling |
| Independent eval | Catch unexpected hacks |
| Reading samples | Catch everything else |
You often want a fixed structure, like the DeepSeek-R1 template:
<think>
...reasoning...
</think>
<answer>
\boxed{1/3}
</answer>
DeepSeek-R1-Zero used exactly this: an accuracy reward plus a format reward.
import re
FORMAT = re.compile(
r"^<think>\n.*?\n</think>\n<answer>\n.*?\n</answer>$", re.DOTALL
)
def format_reward(response: str) -> float:
return 1.0 if FORMAT.match(response.strip()) else 0.0
| Answer | Correct? | Formatted? | \(r^{\text{acc}}\) | \(r^{\text{fmt}}\) |
|---|---|---|---|---|
| \(o_1\) | ✅ | ✅ | 1 | 1 |
| \(o_2\) | ✅ | ❌ | 1 | 0 |
| \(o_3\) | ❌ | ✅ | 0 | 1 |
| \(o_4\) | ❌ | ❌ | 0 | 0 |
✅ Good: \(\lambda = 0.5\). Rewards \(1.5, 1.0, 0.5, 0\); mean \(= 0.75\), std \(= \sqrt{1.25/3} \approx 0.645\).
| Answer | \(r_i\) | \(\hat{A}_i\) |
|---|---|---|
| \(o_1\) correct + formatted | 1.5 | +1.16 |
| \(o_2\) correct only | 1.0 | +0.39 |
| \(o_3\) formatted only | 0.5 | −0.39 |
| \(o_4\) neither | 0 | −1.16 |
Every correct answer beats every wrong one, and formatting breaks ties.
❌ Bad: \(\lambda = 2\). Rewards \(3, 1, 2, 0\); mean \(= 1.5\), std \(= \sqrt{5/3} \approx 1.29\).
| Answer | \(r_i\) | \(\hat{A}_i\) |
|---|---|---|
| \(o_1\) correct + formatted | 3 | +1.16 |
| \(o_2\) correct only | 1 | −0.39 ❌ |
| \(o_3\) formatted only | 2 | +0.39 ❌ |
| \(o_4\) neither | 0 | −1.16 |
A wrong but well-formatted answer is now pushed up, and a correct answer is pushed down. That is a reward hack you created yourself.
Rule of thumb for 0/1 rewards: the best possible format bonus must never outweigh correctness.
\[\lambda \cdot \max(r^{\text{fmt}}) \lt \min \text{ gap in } r^{\text{acc}} \quad\Rightarrow\quad \lambda \lt 1\]Adding the same constant \(c\) to every reward in a group doesn’t change the advantages:
\[\frac{(r_i + c) - \mathrm{mean}(r + c)}{\mathrm{std}(r + c)} = \frac{r_i - \mathrm{mean}(r)}{\mathrm{std}(r)}\]Once every answer in a group is formatted correctly, the format reward is the same for all of them and contributes nothing. In the curves, the format reward rises to about 1.0 early and stays flat, after which correctness drives learning. If it stays low for long, the template may be too strict. A short SFT warm-up or an example in the prompt helps.
| Method | Formula | Pros and cons |
|---|---|---|
| Weighted sum | \(r = r^{acc} + \lambda r^{fmt}\) | Simple, early signal; \(\lambda\) needs care |
| Gate | \(r = r^{acc} \cdot r^{fmt}\) | Very strict; no signal until the format is learned |
| Penalty | \(r = r^{acc} - \lambda(1 - r^{fmt})\) | Same ranking as the sum, shifted so “unformatted” is negative |
| Schedule | \(\lambda\) decreases over training | Strong format signal early, correctness later |
<think></think> passes the template. You can require a minimum length, but don’t reward length directly, or the model will pad.^...$ so text outside the template counts as a failure.