Every Loss Mask Is a Reward

Policy-gradient training of LLM agents routinely excludes rollouts from the loss. Responses that hit the length limit are masked out (DAPO’s overlong filtering^[2]), trajectories that exhaust their turn budget or time out are masked (DeepSWE’s compact filtering^[3], Mini-o3’s over-turn masking^[7]), and asynchronous systems drop rollouts that arrive too stale to be worth training on. The stated rationale is usually noise control: a truncated attempt might have been about to succeed, so scoring it as a failure injects a wrong label. This reasoning is intuitive and widely available as a switch. We will argue that it changes the objective in a precise way, and that whether the change is harmful depends on three things the rationale never mentions: which outcomes the environment has, over whom the baseline is computed, and whether the task’s budget is part of the objective.

The purpose of this post is to lay out a research plan before running it. We have a derivation we believe, a set of symptoms in the literature that the derivation explains, and a set of experiments that would confirm or falsify it. We have not run the experiments. Writing the predictions down first is the point: if the curves in the last section come out the other way, that is what we will report.

The claims, in the order we will defend them:

• Masking is paying. For any trajectory-level inclusion rule, the expected update of the masked loss is $\nabla\mathbb{E}[R\,\mathbb{1}\{X\}] - b\,\nabla P(X)$. When excluded trajectories would otherwise score zero, this is $\nabla J + b\,\nabla P(\neg X)$: the baseline, harmless under full inclusion, becomes a reward of size $b$ for being excluded.
• The baseline term is decided by predictability. A mask that is predictable from the prefix, in the sense $\mathbb{E}[m_t\mid h_t,a_t]=\mathbb{E}[m_t\mid h_t]$, keeps the baseline zero-mean; it still reweights which states are trained, and that is not free. A mask that depends on how the trajectory ends makes the baseline a reward. Truncation, turn caps, format failures, and staleness drops are all of the second kind.
• It can rescale, reweight, or reverse. In an environment with only two outcomes (solved, or capped) the masked update is $\sum_x (1-J_x)\nabla J_x$: a per-prompt rescaling toward hard prompts, with no change of direction. With a third outcome, a submitted failure, the subsidy competes with the failure penalty. And when the excluded trajectories include unobserved successes, a two-strategy construction reverses the sign: the surviving data asks for fewer thorough attempts while the true objective asks for more.
• Reweighting cannot repair it. Truncation is deterministic censoring with inclusion probability zero or one. Horvitz–Thompson corrections need a known positive probability. The only exact fixes are not to exclude, or to randomize the exclusion so a probability exists.
• It is partly already visible. Skywork-OR1 decomposed overall and non-truncated success rates, compared advantage masks, saw truncation and clip ratio rise under the mask, and chose not to mask^[4]. MicroCoder found length growth increasing with the masking rate, and also that a conditional mask beat both full masking and no masking in its setting^[5]. Environment steps rise through DeepSWE training^[3], and staleness thresholds bias toward short responses^[11]. Each of these measures the symptom along one axis; the identity separates three (mask timing, baseline population, budget semantics), and that is the part we claim.
• Each mask has an unbiased replacement. In the budget regime, what the mask removes is not noise but the signal “do not hit the cap”; the variance it buys can be bought legitimately with a baseline conditioned on the turn index reached, which is prefix-measurable and therefore unbiased. In the infrastructure regime the replacements are not dropping, or dropping with a known probability and reweighting. The experiments test whether these replacements match or beat masking at matched compute; if they do not, the paper reports a diagnosis and nothing more.

Revision note. A first draft of this plan predicted a drift toward the cap on ALFWorld. In a two-outcome environment the identity predicts only a rescaling (see the caution in the reversal section), so that prediction was inconsistent with our own math. This revision separates the regimes, weakens the “causal masks are free” claim to what the algebra supports, replaces the imputation arm with a matched-update control, adds a three-outcome environment, and corrects the compute budget. We record the change rather than hide it.

Before any math, it is worth being concrete about how common this is, because the identity below is only interesting if the switch it describes is on somewhere. We read the current main branches of the frameworks we might run on. Two things matter and differ across implementations: whether the switch is a default or an opt-in, and the population over which the baseline is computed.

Three rows deserve a note. The rLLM row is the DeepSWE recipe, and it removes masked episodes before the group statistics are computed, which puts it in the amplified regime of Corollary 2 below. The prime-rl row is the closest thing to a mask decided before each token is sampled, but it is not quite that: the token’s weight version is fixed at sampling, while whether it counts as stale depends on the training step at which the trajectory is consumed, which depends on how long the trajectory ran. We found no implementation that is purely predictable in the sense of Corollary 3. And the caps bind: in a recent SWE-smith run with an 80-turn budget, 84% of trajectories hit the turn cap^[14], although the same report notes that successful trajectories had a median of 8 turns and that tightening the length cap did not affect successful runs. In that run the long tail was mostly failures; we return to this in the discussion.

We work with a single prompt and a policy $\pi_\theta$ that generates a trajectory $\tau$ of action tokens interleaved with environment observations. The objective is the terminal reward, and the estimator is REINFORCE with a baseline, which is what GRPO reduces to on a single step.

• $R$: terminal reward, here binary. Partial credit does not change the argument.
• $b$: the baseline. In GRPO it is the group mean; we first treat it as a constant equal to the population mean $J$, then relax this.
• $h_t$: the prefix before token $t$ (prompt, earlier actions, earlier observations).
• $X(\tau)$: an inclusion event. $\mathbb{1}\{X\}=1$ means the trajectory enters the loss. “Not truncated”, “arrived before the staleness limit”, and “parsed correctly” are all instances.

The one fact we use repeatedly is the score identity: for any prefix, $\mathbb{E}_{a_t\sim\pi(\cdot\mid h_t)}[\nabla\log\pi(a_t\mid h_t)]=0$. Summed over a trajectory it gives $\mathbb{E}[S(\tau)]=0$, which is exactly why a baseline is free: $\mathbb{E}[b\,S]=0$. Everything below is about what happens when the expectation is no longer over all trajectories.

Apply the inclusion rule to the loss and take the expectation over the full policy distribution. The two factors of $(R-b)$ separate:

Both terms are REINFORCE identities, $\mathbb{E}[f(\tau)S(\tau)]=\nabla\mathbb{E}[f(\tau)]$, applied to $f=R\,\mathbb{1}\{X\}$ and $f=\mathbb{1}\{X\}$. The identity holds for any on-policy sample, any inclusion rule without its own parameter dependence, and any baseline that does not depend on the trajectory being scored. What it means depends on what the excluded trajectories would have scored, and there are two regimes.

Regime A: budget failure, reward known. This is the convention that motivates masking in the first place: an excluded trajectory would have been scored $R=0$. Then $R\,\mathbb{1}\{X\}=R$ almost surely, the first term is $\nabla J$, and

The masked estimator is the unbiased gradient plus a term that carries no information about reward. Its direction is “make exclusion more likely” and its magnitude is $b$. Equivalently, since a trajectory with advantage zero contributes nothing, masking a trajectory is identical to assigning it $R:=b$. We find this phrasing more useful than the gradient one because it says what the policy is being taught: an excluded attempt is worth exactly an average attempt. The equivalence is exact only with the baseline and every other statistic (mean, std, loss denominator, regularizers) held at the values they take under masking; if one imputes $b$ into the reward path and recomputes the group mean, the imputed sample regains a nonzero advantage and the two are no longer the same computation.

Regime B: outcome never observed. When the infrastructure cancels or discards a rollout before its result exists, or discards a finished one, the excluded set contains successes and $\mathbb{E}[R\,\mathbb{1}\{X\}]\ne J$. The masked estimator is then the exact gradient of the survivors’ objective: with $b$ set to the survivors’ mean, it is $\nabla\big(\mathbb{E}[R\,\mathbb{1}\{X\}]/\mathbb{E}[\mathbb{1}\{X\}]\big)$ up to scale: the success rate among surviving attempts. This regime does not need the $R=0$ convention, and it is where the sign reversal below lives. The two regimes share the identity but not the ground truth, so we keep them apart in the experiments.

Corollary 1: the feedback loop

The subsidy is $b$, the mean reward. As training succeeds, $b$ rises, so the reward for being excluded rises with it. A policy at 30% success pays truncation 0.3; the same policy at 60% pays it 0.6. This is the opposite of a self-correcting effect. It is the same structural property that makes drift under training–inference mismatch compound in Score Centering^[1]: a term whose magnitude is tied to the thing being optimized.

Corollary 2: survivor baselines amplify it

If the group mean is computed only over surviving members, which is what happens when masked episodes are removed before grouping (rLLM) or when masked_reward_policy: exclude is set (NeMo-RL), the baseline is itself selection-biased. Writing $N_X$ for the number of survivors in a group of $G$, the expected update becomes $\kappa\,\nabla J+\beta\,\nabla P(\neg X)$ with $\beta/\kappa\to J/P(X)=\mathbb{E}[R\mid X]$ for large $G$. So the total coefficient on $\nabla P(\neg X)$ is the survivors’ mean reward, and relative to the all-population baseline the extra term is

which grows with the exclusion rate itself. In a two-outcome environment this coefficient is degenerate: every survivor has $R=1$, the survivors’ advantage is zero, and the signal vanishes rather than drifting. Under std-normalized advantages with binary rewards the per-prompt subsidy is $b/\sigma=\sqrt{p/(1-p)}$, largest on nearly-solved prompts. We flag these as derived, not yet measured; the ablation on baseline population below is designed to separate them.

Corollary 3: predictable masks keep the baseline neutral; outcome-dependent masks do not

Formally, let $m_t$ be the inclusion indicator for token $t$. The precise condition is on conditional expectations, not on causality as such: if $\mathbb{E}[m_t\mid h_t,a_t]=\mathbb{E}[m_t\mid h_t]$, then $\mathbb{E}[m_t\,b\,s_t]=0$ and $\mathbb{E}[\sum_t m_t\,(R-b)\,s_t]=\sum_t\mathbb{E}[m_t\,Q_t\,s_t]$, the policy gradient reweighted by $m_t$. In the tabular case this is an ascent direction for a behavior-weighted objective^[18]; with shared parameters, reweighting states can lower $J$, and the mask can delete the signal from exactly the states that matter. A mask that depends on the future is not automatically biased either: what matters is whether it is correlated with the action given the prefix. Truncation, turn caps, and duration-based drops plainly are, because the action affects the length. So the design rule is weaker than “causal masks are free”: predictable masks cost signal but not bias; outcome-dependent masks cost both. DAPO’s dynamic sampling is outcome-dependent and harmless for a different reason: it excludes whole groups whose advantages are already zero.

A bias of size $b$ on a direction that is unrelated to reward sounds like a nuisance. It is worth showing that it can flip the sign of the update in a setting that is not exotic. The construction lives in Regime B: a two-strategy coding agent under an asynchronous trainer that drops rollouts older than a staleness threshold, so that the excluded set contains successes.

Let $p$ be the probability of choosing Thorough. The true objective is $J(p)=0.6+0.3p$, increasing in $p$. The trainer sees the kept distribution, whose success rate is

The surviving data asks for fewer thorough attempts at every $p$. We checked this by exact enumeration at $p=\sigma(\theta)=0.5$: the full-distribution gradient with respect to $\theta$ is $+0.075$, the survivor estimate with a population-mean baseline is $-0.017$, and inverse-probability weighting with the known keep probabilities recovers $+0.075$. The numbers in the table are ours, not measured from any system; the point is that the structure, successes that finish late and failures that surface early, is the ordinary shape of careful software work.

The same identity has a second, quieter prediction for truncation masks in Regime A, provided the environment has a third outcome. A failing attempt that is masked when it runs out the clock receives advantage 0; one that submits a wrong answer receives $-b$. So on tasks the agent is going to fail anyway, the trained behavior is to keep going rather than to submit. Succeed fast, fail slow is the implied reward schedule, and it is the signature we will look for in turn-count distributions conditioned on outcome.

The natural instinct is importance weighting: if a trajectory survives with probability $c(\tau)$, weight it by $1/c$. Horvitz–Thompson corrections of this kind exist for randomized token subsampling^[10]. They need $c(\tau)>0$ and known. Truncation is deterministic censoring: a trajectory that reaches the cap is excluded with probability one, and a trajectory that does not is included with probability one. There is no weight that recovers the contribution of a sample that was never observed. The same holds for a hard staleness cutoff.

This leaves two exact options. Do not exclude: score the capped rollout with the reward its budget implies, or keep the late rollout and handle its staleness as an off-policy problem, which is what partial rollouts plus per-token corrections already do^[1]. Or randomize the exclusion so that a probability exists: when the scheduler would drop a group, keep it with a known probability $a$ and weight it by $1/a$. The second option is the STOPD trick^[13] applied to trajectories rather than tokens; it is unbiased in expectation,

and it is not free: the variance inflation is $1/a-1$. Whether that price is worth paying relative to simply waiting for stragglers is an empirical question we will put in the plan.

None of the remedies is new machinery. Each is the choice that the identity says is consistent with a stated objective, and the identity says that masking is consistent with none of them. That is a statement about objectives, not about training curves: a consistent estimator can still lose to an inconsistent one on variance or effective step size, and MicroCoder reports a conditional mask beating both full masking and no masking in its setting^[5]. Which one trains better is what the experiments are for. But the identity also says why masking can look helpful, and that points at the replacement.

What masking removes in the budget regime is signal, not noise

In Regime A the contribution of the excluded trajectories to the unmasked estimator is

whose mean is not zero. It is the gradient that teaches the policy not to hit the cap. Masking deletes it, and in doing so also deletes the variance of a $-b$ advantage spread over the thousands of tokens of a long failed trajectory. So the variance benefit of masking and its bias are the same term; there is no version of outcome-dependent masking that keeps one and drops the other. The legitimate way to buy that variance reduction is a better baseline.

The replacement: a baseline conditioned on the turn reached

“This trajectory is already long” is information that exists in the prefix. Let $\ell_t$ be the turn index of token $t$, and define

Because $b_t$ depends only on $h_t$ (the event “this rollout reached turn $\ell_t$”) and on other, independent rollouts, $\mathbb{E}[b_t\,s_t]=0$ and the estimator stays unbiased for $\nabla J$. Its effect is exactly the one masking was reaching for: among rollouts that reached a late turn, most end at the cap with $R=0$, so $b_t$ is small there and a long failed trajectory’s advantage $0-b_t$ is small too. The comparison with masking is one line: masking sets that advantage to $0$ (biased); the baseline sets it to $R-\mathbb{E}[R\mid\ell_t]$ (unbiased). The variance it removes is $\mathrm{Var}(\mathbb{E}[R\mid\ell])$, which is measurable offline from logged rollouts before any training; a conditional leave-one-out baseline is only better than the group mean when that explained variance exceeds roughly one over the bucket count, which is why the shrinkage is part of the definition and not an option.

Two things this replacement is not. It is not a process reward: it changes no optimum, so the agent cannot game it by changing its length. And it is not new machinery: it is the state-dependent baseline of textbook REINFORCE with the cheapest possible state feature. If it matches masking on success and removes the cap-hit drift at matched compute, the recommendation is a one-line default: never mask budget failures; score them zero and baseline on the turn reached.

The first row is a few lines in the advantage function of every framework we checked. The second is a scheduler change; Miles’ async buffer already holds whole groups and has a single _drop_stale branch where a coin flip and a weight would go. The third is a rollout-time change that must be labelled as optimizing a different objective. In all three, the design principle is the same: to discard, decide before sampling; to reduce variance, change the baseline; to change the objective, change the reward and say so. None of the three needs an outcome-dependent mask.

We organize the plan around three questions, in increasing cost. Each has a measurement that could come out against us. We state the arms, the controls, and what we would conclude from each outcome before seeing any of them.

Question 1: Is the term present in real gradients?

Experiment setup. Fix a checkpoint. Collect a few thousand multi-turn rollouts with per-token log-probabilities, rewards, truncation flags, and weight versions. Miles and slime dump all of this with --save-debug-rollout-data and replay it at a fixed checkpoint with --load-debug-rollout-data --debug-train-only; swapping the sample filter changes the masking rule without regenerating anything. We will use SWE-smith rollouts from a Qwen3-4B or 8B mini-swe-agent run with a 40–80 turn cap, where the cap is known to bind for most trajectories^[14], and ALFWorld rollouts as a cheap second environment.

Outcome census first. Before any gradient, count what the rollouts actually are: solved, submitted and failed, budget failure (cap or context limit), and infrastructure no-result (cancelled, timed out, dropped). The census decides which regime dominates in a realistic configuration, and therefore which ground truth applies. If budget failures are 80% of the non-solved mass and infrastructure no-results are 2%, Regime A is the story and the asynchronous experiments are secondary; if the reverse, the Fast/Thorough construction is the one to chase.

Arms. Advantages recomputed under: (a) no mask, $R=0$ on cap (the reference); (b) mask with baseline over all $G$; (c) mask with baseline over survivors; (d) mask only a random fraction $\rho\in\{0.3,0.6,1.0\}$ of capped rollouts; (e) no mask, $R=0$ with the turn-reached baseline. All statistics (mean, std, denominators) are computed once from the unmasked batch and frozen, so that the arms differ only in the mask or baseline. Arm (e) also yields the offline number that decides whether the replacement can work at all: the per-token advantage variance under (a) versus (e), which should drop by $\mathrm{Var}(\mathbb{E}[R\mid\ell])/\mathrm{Var}(R)$. For each, the full-batch policy gradient at the fixed $\theta$, flattened after backward on a single device to avoid tensor-parallel gathers.

Measurements. Cosine between each arm and the reference; the ratio of norms; the projection of the difference onto the estimated $\nabla P(\text{cap})$, obtained by REINFORCE on the cap indicator from the same rollouts; the same quantities bucketed by turn index and by prompt difficulty. On two-outcome data (ALFWorld) the algebra fixes the answer in advance: per prompt the masked gradient must be $(1-J_x)$ times the reference and the survivor-baseline variant must be zero; we use this as a correctness check of the replay pipeline, not as a finding.

What we learn. The identity is algebra and is not what is being tested. The informative numbers are, on three-outcome data, the cosine gap between (a) and (b) after removing the norm rescaling, and the fraction of that gap explained by the $\nabla P(\text{cap})$ direction. If the direction gap is within the seed-to-seed noise of (a), the effect at this checkpoint is a rescaling and reweighting, not a distortion, and we say so before spending anything on Phase 3.

Question 2: Does the term move behavior?

Experiment setup. The direction claim needs a third outcome, so the main environments are three-outcome ones. The cheapest is single-turn RLVR with a length cap (Qwen3-1.7B on GSM8K/MATH with the cap chosen so that 20–40% of responses are truncated at initialization), which is exactly the setting of^[4] and^[5] and gives truncated / wrong / right. The agentic one is WebShop in verl-agent^[15] with its 15-step cap, which gives bought wrong / bought right / stepped out. ALFWorld (Qwen2.5-1.5B, 50-step cap, 150 iterations on two H100s, public curves) is kept as a two-outcome control: there the identity predicts a per-prompt rescaling by $1-J_x$ and a shift of gradient share toward hard prompts, and the survivor-baseline arm should lose its signal. The default convention in all three is $R=0$, so the other arms are one-line edits to the reward and mask.

Arms (3 seeds each). (i) $R=0$ on cap, no mask; (ii) loss-mask on cap; (iii) matched-update control: the mask arm with its learning rate rescaled each step so that its update norm matches arm (i)’s, which separates a change of direction from a change of effective step size; (iv) mask with Dr. GRPO constant normalization^[8] (norm_adv_by_std_in_grpo=False, loss_agg_mode=seq-mean-token-sum-norm) to remove the length-normalization bias as a confound; (v) mask with survivor-only baseline; (vi) the proposed replacement: $R=0$ on cap with the turn-reached baseline. Arm (vi) is the one the paper stands or falls on: it must match or beat (ii) on success at matched compute while behaving like (i) on cap-hit rate and length.

Measurements. Cap-hit rate, submitted-failure rate and episode length per iteration; success rate; clip ratio; update norm; per-token advantage variance; the running value of $b$, so that we can plot the rate of change of the cap-hit rate against $b$ rather than against steps; on ALFWorld, gradient share by difficulty bucket.

Question 3: Does it suppress slow successes under asynchronous drops?

Experiment setup. Miles with the SWE-smith mini-swe-agent recipe^[14] scaled to Qwen3-4B or 8B on a single node, fully asynchronous. The staleness drop is the default --max-weight-staleness handler and the buffer holds complete groups, so the first experiment uses groups that are finished but stale: their rewards exist, and the only question is whether they enter the loss and with what weight. This prototype tests the selection mechanism only. Its rollout and sandbox costs are already paid, so it cannot show a saving in straggler waiting or sandbox time; those claims belong to the in-flight variant, which protects a group before it is cancelled and is a separate, harder experiment. We will choose a task subset with success rate between 0.2 and 0.5 so that $b$ is large enough for the subsidy to be visible.

Arms. (i) drop stale groups (default); (ii) never drop; (iii) randomized protection with $a\in\{0.25,0.5\}$ and the weight $1/a$ applied as a loss weight on the whole group, with one coin per group memoized by group id so that a group that passes through the drop branch more than once is not re-drawn; (iv) a gradient oracle computed on the same cohort of rollouts and the same learner checkpoint with every group included, not a separately generated synchronous run. All arms use the same off-policy correction (per-token truncated importance ratios plus score centering^[1]), so that differences are due to selection and not to the correction. An unbiased gradient is not an unbiased Adam step; we report both the gradient cosine and the training outcome.

Measurements. Turn-count distributions conditioned on outcome (the “succeed fast, fail slow” signature); the rate at which successful trajectories ran the test suite before submitting, as a proxy for thoroughness; gradient cosine to the synchronous reference; wall-clock and sandbox cost per unit of reward improvement, so that protection is charged for the stragglers it keeps alive.

Ablations

The ablations exist to pin the effect to the mechanism and to separate it from the two confounds we know about: length-normalization bias^[8][9] and the genuine noise reduction that masking provides early in training.

These are predictions, written before the runs. We give them with the observation that would falsify each.

1. P1 (algebra check, Q1). On two-outcome ALFWorld data with frozen statistics, the masked gradient equals $(1-J_x)$ times the reference per prompt to numerical precision, and the survivor-baseline variant is zero. Falsified if not; that would mean a statistic is not frozen or the implementation differs from the model, which must be fixed before anything else is run.
2. P2 (direction, Q1). The difference between the masked and unmasked gradients has cosine above 0.5 with the estimated $\nabla P(\text{cap})$, and its norm scales with $b$ across task subsets. Falsified if the difference is isotropic relative to $\nabla P(\text{cap})$, or if its norm is flat in $b$.
3. P3 (drift, Q2, three-outcome environments). The cap-hit rate under masking stops falling, or rises, once $b$ exceeds roughly 0.3, while under $R=0$ and under the matched-update control it keeps falling; the submitted-failure rate under masking falls faster than under $R=0$. Falsified if the mask and $R=0$ arms are indistinguishable in cap-hit rate at matched $b$ across three seeds, or if the matched-update control reproduces the drift.
4. P4 (amplification, Q2 ablation). The survivor-baseline arm drifts faster than the all-$G$ arm, with a gap that grows with the cap-hit rate. Falsified if the two baselines produce the same curves.
5. P5 (signature, Q3). Under staleness drops, successful trajectories get shorter and failed trajectories get longer over training relative to the never-drop arm, and the fraction of successes that ran tests before submitting falls. Falsified if the outcome-conditioned turn distributions do not separate, or separate in the other direction.
6. P6 (rescue, Q3). Randomized protection at $a=0.5$ recovers the never-drop arm’s final success within seed noise at lower wall-clock than never-drop, and at $a=0.1$ it does not, because the variance inflation dominates. Falsified if never-drop is cheaper or if no $a<1$ matches it.
7. P7 (the replacement, Q2). Offline, the turn-reached baseline cuts per-token advantage variance by at least 20% relative to the group mean on three-outcome data. Online, arm (vi) matches or beats the mask arm on final success at matched compute, with a cap-hit rate and episode length that track the $R=0$ arm. Falsified if the offline variance reduction is below 10% (then the fix has no lever and we fall back to reporting the diagnosis), or if the mask arm beats arm (vi) on success by more than seed noise at matched update magnitude (then masking’s benefit is not the variance we modelled, which is itself the finding).

We also expect one result that is not a confirmation: the mask arm should match or beat $R=0$ on success rate early in training, when $b$ is small and the label noise removed by masking is real. Reports that found overlong filtering helpful at short caps and unhelpful at long ones^[6] are consistent with the crossover we would predict, and MicroCoder’s conditional mask^[5] shows that a partial mask can beat both ends. If the mask arm wins throughout at matched update magnitude, the identity is still true but the noise benefit dominates the subsidy at these scales, and the honest headline is narrower: a diagnosis, not a fix.

What the identity does and does not say. It is a statement about the expected update at a fixed parameter. It does not by itself predict training dynamics, and the drift we predict depends on $\nabla P(\text{excluded})$ being a cheap direction for the policy to move along. For length and turn count it plainly is; for format failures it may not be. We also note that masking removes real noise early, so the net effect over a run can be positive for a while, and the identity does not say that it must turn negative: the extra term grows with $b$, but its effect also depends on $\|\nabla P(\text{excluded})\|$, on its angle with the task gradient, on the variance removed, and on the optimizer. The identity fixes the sign of the extra term, not the sign of the net effect.

Relation to prior observations. Skywork-OR1 §3.2.3 is the closest prior work^[4]: it decomposed overall and non-truncated success rates, compared advantage masks on truncated samples, observed a steadily rising clip ratio and more truncation under the mask, and chose not to mask. MicroCoder found that a higher masking rate accelerates length growth with a roughly dose-dependent effect, and that a conditional mask outperformed both ends^[5]. The overlong-filtering ablation in^[6] found gains at short caps that vanish at long ones. DeepSWE’s report shows environment steps rising through training under compact filtering^[3]. Applied Compute documented the short-response bias of staleness thresholds^[11]. Our reading is that these are one mechanism seen from several angles, each along one axis; what we add is the separation of mask timing, baseline population and budget semantics, and predictions in a multi-turn agent that no single prior report makes. The experiments above are designed to test whether that reading is right.

Counter-evidence we carry with us. The SWE-smith run with 84% cap hits^[14] also reports that successful trajectories had a median of 8 turns and that tightening the length cap did not affect successful runs. That is evidence for Regime A (the long tail is failures) and against the idea that slow successes were being deleted in that configuration. The outcome census in Q1 exists to settle this before Phase 3 is funded.

Relation to score centering. The two terms are different objects that share a shape. Score centering^[1] cancels a drift that arises because the sampler and trainer disagree at every prefix; it is a property of the probabilities. The term here arises because the trainer never sees some trajectories; it is a property of the data pipeline. A pipeline can have both, and the remedies compose: centering handles what arrives late, and not dropping ensures that it arrives.

What would change our mind. If P2 fails, the bias exists but points somewhere the policy cannot easily follow, and the practical consequence is small. If P3 fails under constant normalization, the length drift reported in the literature is the normalization bias of^[9], not ours. If the matched-update control reproduces the drift, the effect is effective step size, not direction, and the right framing is “masking is an uncontrolled learning-rate schedule”. If P5 fails, the asynchronous story reduces to the known short-response bias with nothing new added. We would publish any of these as the result.

Scope limits we accept going in. ALFWorld has two outcomes, no tests and no notion of thoroughness; it can show the rescaling and the reweighting, not the direction change, which is why it is a control. Small SWE models may have success rates so low that $b$, and with it the effect, is negligible, which is why Q3 restricts to a mid-difficulty subset. And frontier training stacks may already use penalties or bootstraps rather than masks, in which case this is a cleanup of open-source defaults rather than a change to frontier practice. We think that is still worth doing: the defaults are what most agentic RL runs use. Finally, none of the remedies is guaranteed better by the algebra alone: the turn-reached baseline is unbiased but only helps if turn index explains enough reward variance; predictable masks still reweight states; keeping stale groups costs off-policy risk and trainer time; randomized protection trades bias for variance and cost. The deliverable is the replacement rule and the matched-compute evidence for it (P7, P5, P6); the diagnosis is what makes the rule principled, not the product.

The body of the post fixes the arms, controls, and measurements. This appendix fixes the order of operations, the code paths we will touch, and the compute we expect to spend, so that the plan can be executed as written and audited afterwards. Numbers for ALFWorld come from the published verl-agent recipe and are reliable; numbers for SWE runs are projections from comparable configurations and are to be recalibrated after the pilot.

A. Phases and go/no-go gates

B. Engineering checklist

Phases 0–1, verl-agent (WebShop, ALFWorld) and a single-turn RLVR recipe. ALFWorld recipe: Qwen2.5-1.5B-Instruct, env.max_steps=50, group 8 × 16 groups, 150 iterations, two H100s; WebShop with the 15-step cap from the same repository; single-turn: Qwen3-1.7B on GSM8K/MATH with the length cap set so that 20–40% of initial responses truncate. Code touches: (a) the reward on a cap hit (already $R=0$); (b) zeroing response_mask for cap-hit episodes to obtain the mask arm; (c) the matched-update control: after each backward, scale the mask arm’s learning rate by the ratio of the $R=0$ arm’s update norm to its own, measured on the same batch; (d) the survivor-baseline arm filters before the mean inside compute_grpo_outcome_advantage; (e) the turn-reached baseline: for each rollout $i$ and turn $\ell$, compute the leave-one-out mean of $R_j$ over group members $j\ne i$ with $\text{len}_j\ge\ell$, shrink it toward the group mean with weight $n_\ell/(n_\ell+\kappa)$ ($\kappa=2$), and broadcast $R_i-b_{i,\ell}$ to the tokens of turn $\ell$; about fifteen lines in the advantage function, using the per-token turn index verl-agent already carries. For offline replays all statistics are computed once from the unmasked batch and frozen. Controls: norm_adv_by_std_in_grpo=False and loss_agg_mode=seq-mean-token-sum-norm.

Phases 2–3, Miles (SWE-smith). Start from the public mini-swe-agent recipe^[14] with Qwen3-8B, a 40-turn cap, 64k context, $n=8$, 16 prompts per step. Task subset: run the pilot over the SWE-smith training set and keep tasks with success rate in $[0.2,\,0.5]$, roughly one to two thousand. Code touches: (a) --rollout-sample-filter-path pointing at mask_truncated_zero_reward gives the mask arm, and omitting it gives $R=0$; (b) the RESCUE hook replaces the else-branch of fully_async_data_buffer.py::_drop_stale with a Bernoulli($a$) keep, drawn once per group id and memoized, and attaches $1/a$ as a loss weight on the whole group (not a reward multiplier, which later normalization could undo); (c) never-drop raises --max-weight-staleness; all three async arms enable the same truncated importance sampling plus score centering; (d) offline replay uses --save-debug-rollout-data and --load-debug-rollout-data --debug-train-only, with a post-backward hook that flattens the gradient for the cosine computation; the Q3 oracle is the same cohort replayed with every group included at the same checkpoint.

Logging contract. Every online run logs per step: the outcome census (solved / submitted failure / budget failure / infra no-result), cap-hit rate, mean and outcome-conditioned turn counts, success rate, clip ratio, update norm, the realized baseline $b$, staleness statistics where applicable, and wall-clock. All rate-of-change claims are plotted against $b$ as well as against steps.

C. Compute budget

Units are H100-hours. For a B200 cluster, halve only the LLM-bound items (Phases 0–1, the offline replay); the SWE items are sandbox-bound and do not shrink with the accelerator.

SWE step-cost note. With 128 rollouts per step averaging 25 turns and contexts growing to 30k tokens, generation is about 1.2M tokens per step and the trainer sees about 4M tokens. On one 8×H100 node the LLM side is one to two minutes, but rollouts are bounded by sandbox and tool latency; we project three to five minutes per step, so 300 steps is roughly 20 hours, or about 150 H100-hours per run. A comparable public configuration runs Qwen3-8B for 1000 steps on four B200s. The first twenty steps of the pilot calibrate this number, and Phase 3 is re-budgeted from the measured seconds per step.

Totals. Phases 0–1 alone (the round that tests the replacement rule in the budget regime and decides whether Phase 3 runs): about 1,850 H100-hours, or ten days on one 8×H100 node; the single-turn arms alone, which already test P3 and P7, are about 150 and fit on one GPU in a week. Adding the SWE pilot, the offline replay and Phase 3 with drop / never-drop / $a=0.5$ at two seeds: about 3,700 H100-hours. Full version (all ablations, three seeds, the $a$ sweep): about 4,600 H100-hours. The SWE items are projections and are re-budgeted after the pilot.

D. Allocation on a 24-node H100 cluster

With 192 GPUs the full program is about a day of saturated compute, so GPU time stops being the constraint. The plan below spends the surplus on confidence rather than on more arms, and it keeps the gate: Phase 3 is scheduled only after Phases 0–1 show a direction effect that the matched-update control does not reproduce.

GPU wall-clock for the whole program is four to five days; with pilot calibration, code changes, and analysis, two weeks is a realistic cycle. Upgrades worth buying with the surplus: five seeds instead of three for the headline arms ($R=0$ vs. mask; drop vs. never-drop); 500–1000 steps for the SWE arms, since the subsidy scales with $b$ and the crossover may sit past step 300; the 30B-A3B replication at the scale where the 84% cap-hit rate was measured^[14]; a cap-tightness ablation (40 vs. 80 turns) on SWE, which ALFWorld cannot provide; and a fully-run synchronous reference as the gradient oracle for Q3. Not worth buying: more benchmarks or more model families, which dilute engineering time without touching the claim.

E. Infrastructure prerequisites

• Sandbox CPU. 128 concurrent rollouts per SWE node; 15 nodes plus the 30B runs means roughly 2,000–2,500 concurrent containers. Budget 2–3k CPU cores, a local registry with the SWE-smith images cached, and a few hundred GB of container-layer storage per node. This is the component most likely to delay the program.
• Wall-clock scheduling. Asynchronous arms finish at different times because of stragglers; the scheduler must not wait on the slowest run before starting analysis.
• Storage. Rollout dumps are 1–2 GB per step; dumping the first 50 steps of 15 runs is about 1–1.5 TB.
• Discipline. One variable per comparison, the pre-registered predictions unchanged, the stopping conditions honored. The failure mode that abundant compute invites is adding arms until an effect appears, which is the one thing this post is designed to prevent.