← All posts

On-Policy Merge Stabilizes Multi-Task Reinforcement Learning

Bowen Wang
Nov 5, 2026
Contents
Multi-task RLMerge-afterMOPDOn-Policy MergeCascade RLevery step sums the gradients of all domainsmerged far from every policy that produced datanew student rollouts, teacher logits, no rewardrendezvous when a trainer drifts δ nats from the anchoreach stage starts from the last; nothing holds the earlier domainsdomain Adomain Bdomain Ctraining time →
Multi-task RLlocal optimumCascade RLMerge-aftermerge lands on the ridgeMOPDstudentOn-Policy Mergetrainer collapsesrolled back at the mergeglobal minimum
Figure 1. Five ways to train one model on several RL domains. Left: when each method combines the domains. Right: the methods play one at a time on the same loss landscape over a slice of weight space, each starting from a base model high on the left. Small rings mark each domain’s own optimum. Multi-task RL descends on the mixed loss and settles in a local optimum. Cascade RL trains the domains in turn; each stage pulls the model away from what the earlier ones reached, and it ends far from the global basin. Merge-after averages five finished specialists once, and the merge lands on a ridge. MOPD distils the same specialists into a new student trained from the base, which stops at its teachers’ level. On-Policy Merge (OPM) lets the trainers move apart only as far as a KL budget δ\delta allows, then merges them, round after round. Its path passes the local optimum and reaches the global minimum. When one trainer collapses, the next merge rolls it back. The landscape is an illustrative schematic of a conflict case; when domains help each other, multi-task RL can be faster.

Frontier models now acquire most of their breadth from reinforcement learning on many domains at once. A single policy is trained on competition math, instruction following, tool use, search, terminal work and software-engineering episodes, with one optimizer and batches that mix rollouts from all of them. The mixed batch rests on an assumption that is rarely stated: the domains’ updates can simply be added. At every step, what happens to a domain is its own gradient plus whatever the other domains’ gradients do to it.

When domains agree, that second term is free transfer. When they conflict, it is a tax, and it falls on the domains least able to pay: those with a small share of the batch, small gradients, or episodes slow enough that the run moves on without them. The usual remedies each trade this problem for another. Training domains in sequence forgets the earlier ones. Training specialists apart and merging their weights once lands tens to hundreds of nats from every policy that produced data ,. Distilling the specialists into one student with multi-teacher on-policy distillation (MOPD), the recipe in recent frontier reports ,,, needs a second round of rollouts in every domain and at best matches its teachers.

On-Policy Merge (OPM) keeps the specialists’ independence and gives up only the distance. Each domain trains on-policy on its own trainer, from a shared anchor. As soon as any trainer has drifted a fixed KL budget δ\delta from the anchor, all of them merge: the anchor moves to the point whose policy on each domain is closest in KL to that domain’s trainer, measured on the rollouts the trainer has just collected. Every trainer reloads it and continues. Train apart, meet often.

To second order the merge has a closed form, a Fisher-weighted average θ+=θˉ+(∑iFi)+∑iFiΔi\theta^{+}=\bar\theta+\big(\sum_iF_i\big)^{+}\sum_iF_i\Delta_i, and the choice of Fisher is what does the work. With FiF_i taken on trainer ii’s own on-policy rollouts, its policy gradient lies in the range of FiF_i: a change that domain’s Fisher cannot see is a change its return cannot feel. Whenever the trainers’ updates can be reconciled, the merged model therefore keeps every domain’s own first-order gain and drops every cross-domain term. This is the sense in which OPM stabilizes multi-task RL. Each domain improves whenever its own trainer does, whatever the batch shares, gradient scales or speeds of the others, and a domain that stalls or breaks is held in place rather than dragged.

Decoupling removes transfer of both signs, so OPM is not a faster optimizer, and when domains help each other, summing their updates does better. What OPM buys is a per-domain guarantee at a measurable price. With a guarded solver, one merge of a 9B model takes about ten minutes and needs no environment interaction. Merging five released 4B RL specialists once with the OPM rule, with no training and no student rollouts, beats task arithmetic by 1.7 points and the released MOPD model by 1.1 points on an eight-benchmark mean. Plain summation still wins by 0.9, as the theory predicts for math, science and code, which help each other.

The rest of the post places OPM among the existing recipes, derives the merge and what it guarantees, shows how to make it cheap, and tests it: first on those released specialists, then against strong multi-task RL, merging after training and MOPD on eight frontier-style domains at 9B. The 9B results are marked as pending until the runs finish.

#Four ways to get one model

Every method for combining RL domains makes two choices: how often the domains’ progress is combined, and by what rule. Figure 1 shows the first choice; this section is about the second, and Table 1 sets both side by side.

Multi-task RL combines at every step, in parameter space: one optimizer receives the sum of the domains’ gradients, weighted by their token shares in the batch. With plain SGD this is exactly the same as running one trainer per domain and summing their updates after every step (Appendix A). Combining every KK steps instead gives periodic summing or averaging, the federated-averaging and DiLoCo family ,. Combining once, after training, gives merge-after methods: task arithmetic, TIES and DARE ,,. Cascade RL trains the domains one after another ,. In these terms it is multi-task RL with every share but one set to zero, which is why nothing holds the other domains in place while one trains.

MOPD changes the geometry instead of the frequency. The student minimises the sum of reverse KLs to the teachers on its own rollouts, and to second order this solves a Fisher-weighted merge , with the Fisher measured on the student’s states and paid for with fresh rollouts in every domain. Distral and Divide-and-Conquer RL use the same KL barycenter as their shared policy and reach it by distillation ,.

OPM takes the remaining combination. It combines during training, as often as a KL budget demands, in KL geometry, with the Fisher measured on each trainer’s own states. Table 1 lists what each method pays.

MethodHow oftenGeometryReward when combiningWhat it pays
Multi-task RLEvery stepEuclidean, one shared optimizerYesDomains interfere through the shared step; weak or slow domains lose their share.
Periodic sum or averageEvery KK stepsEuclideanYesThe sum presumes no domain sees another’s update; the average presumes every domain sees every update equally.
Merge-after (TA, TIES, DARE)OnceEuclideanNoThe merged model is far from every policy that produced data, and pays for the whole drift at once.
MOPDOnce, then a distillation stageKL, student statesNo, unless an outcome reward is addedA second rollout stage in every domain; capped at the teachers.
OPMWhen a trainer’s KL reaches δ\deltaKL, trainer statesYes, in every local stepRemoves positive transfer along with interference; one model copy per domain.

The rules are related exactly, not by analogy (Appendix A, Table A2). Summing is OPM when the domains’ Fishers are mutually orthogonal. Averaging is OPM when they are all equal. With one natural-gradient step per trainer, the OPM merge is a natural policy gradient step on the summed objective . Task arithmetic with a coefficient between 1/n1/n and 1 interpolates between the first two presumptions. OPM measures, direction by direction, which one holds; Figure 2 shows what that looks like in weight space.

#On-Policy Merge

Algorithm 1 On-Policy Merge
Input: base model θ0\theta_0; for each domain ii: its data, verifier, rollout workers and hyperparameters, and a drift ceiling δi\delta_i
  1. θˉ←θ0\bar\theta\leftarrow\theta_0
  2. θi←θˉ\theta_i\leftarrow\bar\theta for every domain ii▷ trainer i owns domain i and its optimizer state
  3. repeat
  4. for each trainer ii in parallel do
  5. sample rollouts ξ∼Pθii\xi\sim P^i_{\theta_i}; θi←\theta_i\leftarrow GRPO step on ξ\xi▷ on-policy GRPO , no KL penalty
  6. Di←KL(Pθii ∥ Pθˉi)D_i\leftarrow\mathrm{KL}\big(P^i_{\theta_i}\,\|\,P^i_{\bar\theta}\big)▷ anchor scores a subsample; measured, never penalised
  7. if Di≥δiD_i\ge\delta_i for some ii then▷ rendezvous
  8. Δi←θi−θˉ\Delta_i\leftarrow\theta_i-\bar\theta, Fi←Eξ∼Pθii[∑tstst⊤]F_i\leftarrow\mathbb{E}_{\xi\sim P^i_{\theta_i}}\big[\textstyle\sum_ts_ts_t^\top\big]▷ from each trainer’s cached rollouts
  9. θˉ←θˉ+(∑iFi)+∑iFiΔi\bar\theta\leftarrow\bar\theta+\big(\sum_iF_i\big)^{+}\sum_iF_i\Delta_i▷ the merge (2), solved as in Making the merge cheap
  10. θi←θˉ\theta_i\leftarrow\bar\theta for every ii▷ optimizer states kept; in-flight rollouts reweighted per token
  11. until the training budget is spent
  12. return θˉ\bar\theta

The objective. We want one model whose behaviour on each domain is as close as possible to that domain’s trainer. A rollout ξ=(h1,a1,…,hL,aL)\xi=(h_1,a_1,\dots,h_L,a_L) of domain ii has probability Pθi(ξ)=∏tπθ(at∣ht)P^i_\theta(\xi)=\prod_t\pi_\theta(a_t\mid h_t) times environment terms that do not depend on θ\theta. Closeness of behaviour over whole rollouts is the KL between rollout distributions, in which the environment terms cancel:

θ+∈arg⁡min⁡θ  ∑iKL(Pθii ∥ Pθi).(1)\theta^{+}\in\arg\min_\theta\;\sum_i\mathrm{KL}\big(P^i_{\theta_i}\,\|\,P^i_\theta\big).\tag{1}

Up to a constant, this is the negative log-likelihood of every trainer’s own rollouts under the merged model, so it needs nothing the trainers do not already hold. The barycenter objective is not ours: Distral and Divide-and-Conquer RL use it for their shared policy and reach it by distillation ,. What is new is solving it in closed form, in one shared network, during training.

The closed form. In a round, trainer ii moves from the anchor to θi=θˉ+Δi\theta_i=\bar\theta+\Delta_i. Expanding each KL around its trainer, KL≈12(θ−θi)⊤Fi(θ−θi)\mathrm{KL}\approx\tfrac12(\theta-\theta_i)^\top F_i(\theta-\theta_i), and setting the gradient of the sum to zero gives the OPM merge:

  θ+=θˉ+(∑iFi)+∑iFi Δi  ,Fi=Eξ∼Pθii[∑tstst⊤],st=∇log⁡πθi(at∣ht).(2)\boxed{\;\theta^{+}=\bar\theta+\Big(\sum_iF_i\Big)^{+}\sum_iF_i\,\Delta_i\;},\qquad F_i=\mathbb{E}_{\xi\sim P^i_{\theta_i}}\Big[\sum_t s_ts_t^\top\Big],\quad s_t=\nabla\log\pi_{\theta_i}(a_t\mid h_t).\tag{2}

Each direction in weight space goes to the domains that are sensitive to it. Where only domain ii has curvature, the anchor takes all of Δi\Delta_i. Where several do, it takes their curvature-weighted mean. Where none does, nothing changes that any domain can see. For tabular policies the exact minimiser of (1) is, at every history, the mixture of the trainers’ action distributions weighted by how often each one visits that history; (2) is its second-order version.

Formula (2) is Fisher-weighted averaging . The difference is which Fisher. Here it is the Fisher of each trainer’s current policy, on its own domain, estimated from rollouts that policy has just sampled, summed over tokens. That is where the name comes from. The data are on-policy: every gradient step of every trainer uses its own rollouts. The metric is on-policy: the Fisher is an expectation over the policy’s own actions, and on stale or foreign rollouts the same formula becomes the empirical Fisher, a different and biased object . And the continuation is on-policy: the merge moves each trainer by a small per-token KL, so the rollouts it holds remain usable and training resumes without a hand-over stage.

#What the merge guarantees, and what it costs

The theory rests on one property of the on-policy Fisher. Write S(ξ)=∑tstS(\xi)=\sum_ts_t for the score of a rollout. If Fiv=0F_iv=0, then E[(v⊤S)2]=0\mathbb{E}[(v^\top S)^2]=0, so v⊤S=0v^\top S=0 on every rollout of domain ii, and therefore v⊤∇Ji=E[(Ri−b) v⊤S]=0v^\top\nabla J_i=\mathbb{E}[(R_i-b)\,v^\top S]=0. The policy gradient lies in the range of the Fisher. A displacement the Fisher cannot see is a displacement the return cannot feel. This holds because the gradient and the Fisher are expectations under the same distribution. It fails for supervised gradients against a model Fisher, and for Fishers computed on stale or foreign states.

Call a round lossless if some displacement dd leaves every trainer’s policy unchanged to second order, Fj(d−Δj)=0F_j(d-\Delta_j)=0 for all jj. This holds for every possible set of updates exactly when the Fisher ranges are linearly independent: independent, not orthogonal.

Theorem 1 · decoupling

In a lossless round, (2) returns a lossless point and, for every domain jj, to first order in the displacements,

Jj(θ+)−Jj(θj)=0,Jj(θˉ+∑iΔi)−Jj(θ+)=Tj,Tj=∇Jj(θj)⊤∑i≠jΔi.J_j(\theta^{+})-J_j(\theta_j)=0,\qquad J_j\Big(\bar\theta+\sum_i\Delta_i\Big)-J_j(\theta^{+})=T_j,\qquad T_j=\nabla J_j(\theta_j)^\top\sum_{i\ne j}\Delta_i .

The second identity is the important one. TjT_j is the cross-domain term: what the other domains’ updates do to domain jj’s return. Summing keeps it; OPM removes it. When Tj<0T_j<0 that term is interference, and removing it is the point. When Tj>0T_j>0 it is positive transfer, and OPM removes that too. Coupling between domains needs overlapping Fishers, not aligned gradients. In the released 4B specialists merged below, parameter cosines are at most 0.022, yet the math specialist’s update moves the code specialist’s policy by 1.78 times the code specialist’s own drift.

For log-linear policies whose domains have linearly independent feature spans, the theorem is exact at every interval, not only to first order: after rr rounds the anchor’s policy on domain jj equals that of a run that trained domain jj alone for rKrK steps. This holds for any trainer optimizer whose update depends only on its own history, from gradient descent and momentum to natural gradient and Adam with persistent state.

anchortrainer 1trainer 2averagetask arithmeticOPM merge
Figure 2. One round in a two-dimensional slice of weight space. Height is the second-order KL residual summed over both domains. Each domain is a flat-bottomed trough, the set of weights with the same policy on that domain, to second order the affine subspace θi+null⁡(Fi)\theta_i+\operatorname{null}(F_i). The two trainers slide from the anchor to the bottom of their own troughs. Averaging their updates and summing them (task arithmetic) both land on the walls; the OPM merge (2) solves for the point where the troughs cross, which exists when the round is lossless. The quadratic picture holds only near the trainers, which is why the rendezvous happens before they drift far.

Where lossless rounds exist, decoupling gives guarantees that multi-task RL does not have.

Proposition 2 · per-domain guarantees, given lossless rounds
  1. Monotone per domain. JjJ_j decreases only if domain jj’s solo run does.
  2. Invariant to shares. Domain jj’s trajectory is independent of the other domains’ batch shares, gradient scales and speeds.
  3. Idle protection. If trainer jj makes no progress, the anchor moves inside null⁡Fj\operatorname{null}F_j.
  4. The trigger is the trust region. For any optimizer, Di≤δ+ωD_i\le\delta+\omega, with ω\omega one step, and the exact merge has residual ∑iεi≤∑iDi≤n(δ+ω)\sum_i\varepsilon_i\le\sum_iD_i\le n(\delta+\omega).

Items 1–3 are exact for log-linear policies and hold per round to first order otherwise; item 4 needs no lossless point. Multi-task RL has none of these. One step changes JjJ_j by η(cj∇Jj⊤P∇Jj+∑i≠jci∇Jj⊤P∇Ji)\eta\big(c_j\nabla J_j^\top P\nabla J_j+\sum_{i\ne j}c_i\nabla J_j^\top P\nabla J_i\big), with cic_i the batch shares and PP the optimizer’s preconditioner, so under conflict any domain with a small enough share or gradient regresses. Two parameters are enough to see it. Take J1=a σ(s)J_1=a\,\sigma(s) and J2=b (1−σ(s+p))J_2=b\,(1-\sigma(s+p)), with domain 2’s share at 0.2. Multi-task SGD drives J2J_2 from 0.50 down to 0.24, Adam in a rotated parameterisation drives it to 3×10−43\times10^{-4} and leaves it there, and periodic summing dips by 0.20. OPM’s J2J_2 trajectory is identical, to 2×10−162\times10^{-16}, for every share and reward scale of domain 1. A cascade on the same problem takes J2J_2 from 0.9987 to 0.0735 while domain 1 trains; keeping the idle trainer in an OPM merge keeps it at 0.9987.

No reference-KL penalty is needed for a trust region, because the trigger is one. That matches frontier practice, where RL runs without a KL term, and no arm of our experiments uses one. The ceiling is best set per token. The cubic remainder of the quadratic model accumulates token by token, so δi\delta_i should scale with domain ii’s response length (Appendix B.6).

Proposition 3 · the price

Let nn domains share a direction with common curvature and agreeing objectives, and let each round solve a fraction xx of it. Along that direction OPM moves xx, the sum moves nxnx, and multi-task RL’s convergence exponent is about nn times OPM’s. With an interior optimum the sum is better per round iff x<2/(n+1)x<2/(n+1) and diverges iff x>2/nx>2/n; with a boundary optimum it is better at every xx.

A boundary optimum is the common case: pushing up an action that every domain rewards. With nonlinear policies OPM also carries an error floor of order ηK\eta K, set by how far the lossless point lies in parameter space. So OPM has no general rate advantage over a well-tuned multi-task RL run, and we do not claim one. In KL geometry, positive transfer is extra reward-aligned drift on a domain, in effect a larger step, and a larger trainer learning rate recovers part of it. Meeting more often does not change this for an exact merge: in lossless directions OPM equals solo training at every interval. What OPM offers instead of speed are the guarantees above, and when domains conflict, those are worth more than speed.

A controlled toy shows both sides (Appendix E): four chain tasks share a 704-parameter policy, with exact gradients, ten seeds, and five variants ranging from strong positive transfer to conflict. With learning rates tuned for both methods, OPM and multi-task RL end within a few hundredths of each other on the original problem: 0.955 to 0.993 for OPM against 0.949 at 50 steps per task. An earlier version of this comparison used an under-tuned multi-task baseline (0.688) and overstated the gap as 0.27. Under positive transfer, multi-task RL leads at small budgets: by 0.20–0.23 at 10 steps per task and 0.03–0.05 at 50. Under conflict OPM leads at every budget once both learning rates are tuned. At 50 steps it reaches 0.97 at its default rate, against 0.77–0.80 for the best multi-task rate. No OPM run ever ended with a task below its starting point, while multi-task RL, task arithmetic and summing left 20–37% of tasks below it.

Proposition 4 · MOPD

Let Li(θ)=KL(Pθi∥Pθii)L_i(\theta)=\mathrm{KL}(P^i_\theta\|P^i_{\theta_i}) be MOPD’s loss for teacher ii. Then Ji(θ)≤Ji(θi)+Li(θ)/2J_i(\theta)\le J_i(\theta_i)+\sqrt{L_i(\theta)/2}. The OPM point is a stationary point of ∑iLi\sum_iL_i to second order, with gradient O(∥⋅∥2)O(\|\cdot\|^2) against O(∥⋅∥)O(\|\cdot\|) for task arithmetic and the sum, and in a lossless round it is the global minimiser to third order.

The bound is Pinsker’s inequality. It means that at its optimum the MOPD student equals its teachers, and can exceed them only by not converging or through an added outcome reward. OPM reaches that optimum without any student rollouts. Liu et al. showed that MOPD is Fisher merging to second order; what we add is the lossless statement and which Fisher to use. OPM has no fixed teacher: each round’s “teacher” θi\theta_i is retrained on reward from the last merge. And a rendezvous needs no environment interaction, while a round of MOPD needs fresh rollouts in every domain, which for agentic domains means sandboxes running for minutes to tens of minutes per episode.

Every statement here carries a status in Appendix B: theorem, proposition under stated assumptions, or conjecture with numerical support. The guarantees of Proposition 2 assume lossless rounds, which we have not shown to exist at LLM scale (see the Discussion). Appendix B also has the proofs, the first-order coupling of every merge, and a list of what is not claimed.

#Making the merge cheap

Formula (2) cannot be formed for a model with billions of parameters, but it can be solved. Conjugate gradient needs only Fisher-vector products . Each trainer computes its own FivF_iv on its own GPUs, at its own weights, over its own cached rollouts, and one all-reduce per iteration sums them. We profiled this on MiMo-V2.6-Distill-Qwen-9B and found three things.

The Fisher-vector product is the cost. One exact product costs 6.9 GPU-milliseconds per generated token in fp32, about sixteen bf16 gradient passes. Roughly 40% goes to the finite-difference forward passes, 50% to the backward pass of the vector-Jacobian product and 7% to the vocabulary head. Communication and the n×nn\times n algebra are negligible. Keeping the CG vectors in CPU memory, a natural choice for vectors with nine billion entries, made each product about three times slower: a float64 reduction over 2.35 billion entries alone cost 31 seconds per product. At the Fisher budget we first planned, ten iterations made one merge take one to three hours, the time of tens of training steps.

Conjugate gradient overshoots. In a first test, CG started from the summed update made the true held-out KL residual worse at each of its first two iterations (56, then 112, then 198 nats). The quadratic model was not the problem: along the cross-domain directions the locality ratio κ\kappa, true KL over quadratic prediction, stayed between 0.87 and 1.0. The problem is where CG looks. Its first search directions lie in the Fisher’s stiffest eigendirections, about 10710^7 times stiffer than typical ones, where the KL saturates within roughly 0.3% of the update norm. A finite-difference Fisher-vector product measures a secant over its step, and over a saturating stretch the secant is flatter than the local curvature. It underestimated the curvature sixteen-fold at a logit step of 0.3 and by about 40% at 0.03, so CG takes too long a step.

The fix is a guarded merge. Use a finite-difference logit step of 0.01, run three CG iterations from the summed update dΣd_\Sigma, and choose γ∈{0,14,12,1}\gamma\in\{0,\tfrac14,\tfrac12,1\} along d(γ)=dΣ+γ(xCG−dΣ)d(\gamma)=d_\Sigma+\gamma(x_{\rm CG}-d_\Sigma) by the true KL on held-out rollouts of the trainers. Since γ=0\gamma=0 is the sum, the chosen point is never worse than the sum up to estimation error, and no quadratic assumption is needed for that (Proposition 5, Appendix C).

Mergeγ=1\gamma=1γ=12\gamma=\tfrac12γ=14\gamma=\tfrac14
No merge (stay at the anchor)125.3
Average of the updates30.7
Sum of the updates13.35
CG, 1 iteration10.7110.0511.23
CG, 2 iterations10.448.7410.11
CG, 3 iterations11.108.579.91
CG, 5 iterations14.079.3410.08
CG, 10 iterations21.8911.4610.55
CG, 3 iterations, finite-difference step 0.3214.6652.7317.40

Table 2. Held-out KL residual ∑iKL(Pθii∥Pmergei)\sum_i\mathrm{KL}(P^i_{\theta_i}\|P^i_{\rm merge}) in nats, for two trainers branched from the 9B model (math, 40 steps; instruction following, 30 steps) with drifts of 76.4 and 49.0 nats. Lower is better. Finite-difference logit step 0.01 except in the last row. Values for γ=0\gamma=0 are the sum.

Three iterations plus the line search cut the residual by 36% relative to the sum (13.35 to 8.57 nats) and by 72% relative to averaging, in about ten minutes: four Fisher-vector products and four forward passes. The full CG step never beats a damped one, five or more iterations make things worse, and a large finite-difference step blows the residual up to 113–215 nats. This guarded merge is the default in all our runs. When rendezvous are frequent, the CG step can be dropped, leaving the sum plus the line search. This is one test case, two domains on one model. The main runs log the residual for every γ\gamma at every rendezvous.

1015202530held-out KL residual (nats)123510CG iterationsaverage of the updates 30.7sum of the updates 13.35finite-difference step 0.3, γ = ¼, 3 CG iterations: 17.4 natsFD step 0.3: 214.7 · 52.7 · 17.4γ = 1 (full CG step), 1 CG iteration: 10.71 natsγ = 1 (full CG step), 2 CG iterations: 10.44 natsγ = 1 (full CG step), 3 CG iterations: 11.1 natsγ = 1 (full CG step), 5 CG iterations: 14.07 natsγ = 1 (full CG step), 10 CG iterations: 21.89 natsγ = 1γ = ½ (half step), 1 CG iteration: 10.05 natsγ = ½ (half step), 2 CG iterations: 8.74 natsγ = ½ (half step), 3 CG iterations: 8.57 natsγ = ½ (half step), 5 CG iterations: 9.34 natsγ = ½ (half step), 10 CG iterations: 11.46 natsγ = ½γ = ¼ (quarter step), 1 CG iteration: 11.23 natsγ = ¼ (quarter step), 2 CG iterations: 10.11 natsγ = ¼ (quarter step), 3 CG iterations: 9.91 natsγ = ¼ (quarter step), 5 CG iterations: 10.08 natsγ = ¼ (quarter step), 10 CG iterations: 10.55 natsγ = ¼8.57, the default
Figure 3. Held-out KL residual of the guarded merge after one to ten CG iterations, for three damping factors γ\gamma along the path from the summed update (data of Table 2). Dashed lines mark the sum (13.35 nats) and the average (30.7); not merging at all leaves 125.3. Every damped step stays below the sum, and the residual is lowest at three iterations with γ=12\gamma=\tfrac12, the default. The full step degrades after two iterations, and a finite-difference step of 0.3 pushes two of the three settings off the chart.

#A first test on released specialists

Before training anything, we can test the merge operator on RL specialists that someone else trained. Wu et al. released five 300-step GRPO specialists of Qwen3-4B-Instruct-2507 in fp32 (math, science, code, instruction following and a tool-using agent), together with their task-arithmetic merge (λ=0.6\lambda=0.6), a mixed-RL model, an MOPD model, and their data and evaluation. We merged the five specialists once with the OPM rule: ten CG iterations from the summed update, with exact per-token Fisher-vector products on 256 of each specialist’s own fp32 rollouts. There is no training and there are no student rollouts. This tests the operator in its least favourable setting, one merge at the end of 75 to 248 nats of drift. It does not test the loop.

The quadratic model holds even at that drift. The locality ratio κ\kappa toward the merged points is 0.96–1.17, where a quadratic would give exactly 1, and the directions of the specialists’ updates carry 5,800 times the curvature of random directions of the same norm. The merge brings each specialist’s KL residual far below task arithmetic’s. On held-out prompts the residual averages 0.31 of the drift, against 0.52 for task arithmetic, 1.12 for the sum and 0.34 for the released MOPD model, and it falls to 0.16 after 30 iterations. For the instruction-following and agent specialists it is 0.025 and 0.037 of their drift, against 0.26 for task arithmetic. Held-out and in-sample residuals agree, so 256 rollouts per domain do not overfit.

ModelAIME25AIME26GPQALCB v5LCB v6IFEvalIFBenchBFCLMean
Base45.952.957.458.659.983.730.763.856.62
Specialist, own domain50.358.661.763.363.789.648.972.463.56
Task arithmetic, λ = 0.653.960.263.964.063.888.240.471.763.26
Mixed RL (released)52.760.061.664.164.286.340.271.762.60
MOPD (released)52.059.762.963.963.989.146.772.663.84
Sum, λ = 155.762.764.568.064.489.649.372.465.84
OPM merge, 10 CG iterations51.059.363.067.365.689.550.573.364.94
OPM merge, 30 CG iterations53.959.862.866.766.689.050.873.065.32
OPM merge, 10 iterations, deployed in fp3253.362.364.168.966.890.052.072.566.24

Table 3. One-shot merges of the five released Qwen3-4B specialists of Wu et al., mean over 32 samples per prompt, in percent. Shading runs within each column, from its lowest to its highest score; the base model is left unshaded as the reference. Bold marks the best bf16 merge in each column. All merged models are deployed in bf16 except the last row. The specialist row takes each benchmark from its own domain’s specialist. Run-to-run noise, measured by evaluating a bit-identical model twice, is about 0.3 points on the mean.

On benchmarks (Table 3) the one-shot OPM merge beats task arithmetic by 1.68 ± 0.42 points on the eight-benchmark mean, paired over prompts, and the released MOPD model by 1.11 ± 0.40. It uses no rollouts beyond the 256 per domain behind the Fisher. On IFBench it scores 50.5, against 40.4 for task arithmetic and 48.9 for the instruction-following specialist itself, so it recovers 119% of the gap that task arithmetic leaves. Deployed in fp32 instead of bf16, the same merge gains another 1.30 ± 0.34 points.

Plain summation is the best one-shot merge. The sum (λ=1\lambda=1) beats the OPM merge by 0.89 ± 0.36 points, mostly on AIME, even though its KL residual is almost four times larger. This is Theorem 1 at work, not a failure of the metric. Math, science and code help each other, and the merged reasoning scores exceed every individual specialist. The OPM merge pulls each specialist back toward its own behaviour and gives that transfer up. The theory’s diagnostic agrees: the sum’s estimated first-order advantage on the reasoning domains is positive, 0.17 to 0.26 of each domain’s own gain, while the instruction-following and agent domains barely see the others’ updates, and there every merge ties. The KL residual bounds how much of each specialist’s own behaviour is lost. It does not predict reward.

Two more observations temper the result. A metric computed on stale rollouts (the base model’s, instead of each specialist’s own) is worse in KL, 0.38 against 0.31 of drift, but ties on benchmarks (−0.22 ± 0.31). And starting CG from the task-arithmetic point instead of the sum does not beat task arithmetic (−0.25 ± 0.35). One-shot at 4B, then, the metric earns its keep against task arithmetic and MOPD, but not against the sum. The case for OPM rests on merging repeatedly while trainers are still close. That is what the main experiment tests. Appendix D has the full residual tables, the interference matrix and the evaluation protocol.

#Main results

Setting. The main comparison trains MiMo-V2.6-Distill-Qwen-9B on eight domains that public frontier recipes train with RL: software engineering, terminal, cyber, knowledge work, search, tool use, hard math and instruction following. Verifiers are executable tests, rule checkers, state checks and, for knowledge work, a rubric judge. Agentic episodes run up to 150 turns and 128k tokens. All arms share one RL framework (slime), one GRPO recipe, one chat template with thinking on, the same verifiers, harnesses and evaluation, and no reference-KL penalty. The budget is matched in environment rollouts per domain, counting MOPD’s student rollouts. GPU-hours and wall-clock are reported separately. Appendix F gives every detail.

  • A1 · Multi-task RL. One trainer on all eight domains, made strong: per-domain rollout quotas matched to the budget, ledger-normalised gradient shares, deficit scheduling with replay for slow domains, per-domain length and turn budgets, and a tuned learning rate.
  • A2 · Merge-after. Eight specialists trained to the full budget and merged once. We report the best of task arithmetic with λ∈{0.4,0.6,0.8,1.0}\lambda\in\{0.4,0.6,0.8,1.0\} (chosen on held-out reward), DARE, TIES, the closed-form merge of Liu et al., and a one-shot OPM merge.
  • A3 · MOPD. The same eight specialists as teachers, with and without an outcome-reward term.
  • A4 · OPM. Eight trainers, KL-triggered rendezvous, and the guarded merge described above.
Placeholder · Headline paragraph · pending main runs

Mean and worst-domain normalised score of A4 against A1–A3 at matched rollouts, three seeds, with confidence intervals; one sentence per baseline on where the gap comes from.

ArmSWE-VTB 2.1CyberKWBC-Plusτ²AIME26IFBenchMeanWorst
Base––––––––––
Specialists, own domain––––––––––
A1 · Multi-task RL––––––––––
A2 · Merge-after, best––––––––––
A3 · MOPD––––––––––
A3 · MOPD + outcome reward––––––––––
A4 · OPM––––––––––
Placeholder · Table 4 · pending main runs (A1–A4, 3 seeds)

Per-domain scores: SWE-bench Verified, Terminal-Bench 2.1, MiMo Cyber (mini), MiMo General (mini), BrowseComp-Plus, τ²-Bench, AIME26, IFBench. Mean and worst columns are normalised from base (0) to each domain’s specialist (1). Mean ± 95% CI over 3 seeds. Secondary benchmarks per domain (SWE-bench Pro, HMMT Feb 26, IFEval, BFCL v4, τ³-Bench) and the held-out suite (MMLU-Pro, GPQA, HLE) go in Appendix F.

Cost. The arms spend their budgets in different places. Multi-task RL spends nothing on combining. MOPD’s combining cost is a second stage: student rollouts in every domain, including agentic sandboxes, and teacher scoring of every student token. OPM’s is its merges, which need no environment interaction: a few Fisher-vector products per trainer on cached rollouts and four forward passes for the line search.

ArmEnv. rolloutsStudent rolloutsTeacher scoring (GPU-h)Merges (GPU-h)Total (GPU-h)Wall-clock
A1 · Multi-task RL––––––
A2 · Merge-after––––––
A3 · MOPD––––––
A4 · OPM––––––
Placeholder · Table 5 · pending main runs (ledger)

Cost per arm from the shared ledger, seed 1: environment rollouts per domain (matched by design), student rollouts, teacher-scoring and merge GPU-hours, total GPU-hours and wall-clock. Also the number of OPM rendezvous and the mean wall-clock per merge.

Transfer. Theorem 1 makes a sharp prediction about where OPM should lose. Per domain and rendezvous, the sign of Jj(sum)−Jj(OPM)J_j(\text{sum})-J_j(\text{OPM}) should be the sign of the transfer term TjT_j, which each trainer can estimate from its own batch with one forward-mode pass per rollout. OPM should match or beat multi-task RL on the worst domain where domains conflict, and trail it on domains that help each other (knowledge work and tool use, or math and held-out reasoning) unless trainer learning rates are raised. In an exact-gradient check the correlation between TjT_j and the realised gap was 0.84–0.87 for short rounds and fell to 0.27 for long ones, where second-order terms dominate. Whether it predicts at LLM scale is open.

Placeholder · Table 6 and Figure 4 · pending main runs

Table 6: per-domain score of each arm normalised to that domain’s specialist (above 1 means the domain benefits from the others), A1–A4, 3 seeds. Figure 4: at every rendezvous of A4, the estimated TjT_j against the realised held-out reward gap between the summed update and the OPM merge, one point per domain and rendezvous; sign agreement reported in the caption.

#How often to merge

The rendezvous frequency is the one knob that moves OPM along the frequency axis, from merge-after (a single rendezvous at the end) toward multi-task RL. The theory says what frequency does and does not do. It keeps every round local, so the quadratic model holds and rollouts in flight stay usable. It keeps the Fisher weights calibrated along shared directions: as a trainer grows confident, its endpoint Fisher vanishes on the logits it pushed, and a merge made too late undervalues it. In a two-action example the locality ratio is 1.00, 1.35, 1.92 and 8.9 after 1, 10, 20 and 200 steps. In lossless directions, frequency changes nothing.

In the toy the interval matters sharply. At 50 steps per task, merging every 1, 5, 10, 25 or 50 steps gave 0.97, 0.96, 0.93, 0.34 and 0.08. A KL trigger beat a fixed interval with a similar number of rendezvous, 0.987 against 0.88. Frequent rendezvous also passed more transfer into an idle task: under the exact merge, +0.48 at an interval of 1 against −0.21 at 50. The theory attributes this to calibration along shared directions, not to transfer through lossless ones, and part of it is unexplained (Appendix E). Experiment E measures the same curve on the 9B model.

Rendezvous per runMeanWorstIn-domain benefit (T1)Untrained domain (T2)Merge cost
1 (one-shot OPM merge, end of run)–––––
≈ 4–––––
≈ 16–––––
≈ 32–––––
A1 · Multi-task RL (reference)–––––
A2 · Merge-after, best (reference)–––––
Placeholder · Figure 5 and Table 7 · pending experiment E

The merge-frequency curve E. A4 with the trigger ceiling set for about 1, 4, 16 and 32 rendezvous at a fixed budget; the one-rendezvous end coincides with A2’s one-shot OPM merge. Horizontal axis: rendezvous per run (log scale). Vertical axes: mean and worst normalised score, with A1 and A2 as horizontal bands. Table 7 adds in-domain benefit from other domains (T1), the score on competitive code, which no arm trains on (T2), and merge cost.

#Adding a skill, containing a fault

Because each domain has its own trainer, OPM can change its set of domains mid-run, and a fault in one domain reaches the others only through the merge. Two demonstrations test whether that matters in practice. Both branch from the seed-1 runs and apply the same intervention to multi-task RL.

Demo B: adding a skill. At half the budget, competitive code, a domain no arm has trained on, is added. OPM starts a code trainer from the current anchor at the next rendezvous; multi-task RL resumes with a re-mixed nine-domain quota. We measure how many rollouts and how much wall-clock code needs to approach its specialist, the largest drop on the eight existing domains, and the extra cost. Idle protection predicts that the existing domains hold wherever the new trainer’s Fisher range is independent of theirs.

Placeholder · Figure 6 and Table 8 · pending demo B

Code score against rollouts after the branch point for A1 and A4, with the code specialist as a reference line; below it, the per-domain change on the eight existing domains over the same window. Table 8: rollouts and wall-clock to reach 90% of the code specialist’s gain, maximum drop on an existing domain, extra GPU-hours.

Demo C: containing a broken domain. At 30% of the budget the tool-use reward is broken for a fixed window: any syntactically valid tool call is rewarded. Then it is restored. Both arms get the same fault. We measure the other domains’ scores and behaviour during and after the window, how fast they recover, whether OPM’s per-rendezvous diagnostics flag the faulty trainer, and what rolling back that one trainer does. Theorem 1 predicts that the fault reaches other domains only through Fisher overlap, and the trigger bounds how far it moves them per round.

Placeholder · Figure 7 and Table 9 · pending demo C

Per-domain scores over training for A1 and A4 with the fault window shaded, plus tool-call validity and response length. Table 9: worst drop on another domain during the window and after recovery; the rendezvous at which the guard flagged the tool-use trainer; the effect of rolling back that trainer alone.

#Side findings

Measuring merges at the level of policies, in exact fp32, turned up five findings that matter beyond OPM. Appendix G has the numbers.

Orthogonal is not independent. The released specialists’ task vectors are nearly orthogonal, with cosines of at most 0.022, yet they interfere strongly. The math specialist’s update moves the code specialist’s policy by 1.78 times the code specialist’s own drift. Interference lives in the Fisher, which weights directions by how much the policy cares about them, and the directions RL moves are thousands of times stiffer than random ones. Cosine similarity of task vectors is a poor guide to whether two specialists can be merged.

“Sparse” RL updates are a bf16 artefact. In fp32 the specialists’ updates are dense: 99.996% of entries are nonzero, with a median size of about 2−112^{-11} relative to the weights. Rounding the specialists to bf16 leaves only 18% of the update entries nonzero, and the rounded update has cosine 0.57 with the true one and an 84% relative error. Claims that RL makes sparse updates should be checked against fp32 weights.

There is a bf16 floor. Casting an fp32 specialist to bf16 moves its policy by 25–40% of its whole RL drift, measured in KL on its own rollouts. The same merged model scores 1.30 ± 0.34 points higher deployed in fp32 than in bf16. Any merge, trigger or residual threshold has to sit above this floor, and scoring has to run in true fp32.

KL residual is not reward loss. The sum has the largest KL residual of any merge and the best benchmark mean. The exact identity behind this is that a return change equals the reward-weighted log-likelihood ratio plus at most the KL. To first order it is ρ σ2ε\rho\,\sigma\sqrt{2\varepsilon}, with ρ\rho the correlation between reward and the log-ratio. A large residual costs little reward when ρ≈0\rho\approx0 and helps when ρ>0\rho>0. Pinsker-type bounds are vacuous above about 2 nats, while specialists drift 75–248 nats.

When the sum beats OPM. The sum wins exactly when domains help each other, and that can be measured per domain: how much of the others’ updates domain jj’s Fisher sees, and the sign of the transfer term TjT_j. The 4B reasoning domains are in this regime. The instruction-following and agent domains barely see each other, so every merge ties. The toy’s conflict variants are the opposite regime, where OPM wins. Logging TjT_j at each rendezvous tells a practitioner which regime a run is in.

#Discussion

When to use it. OPM is a decoupling device. It pays where coupling hurts: domains that conflict, many domains with small shares each, rollout latencies that differ by orders of magnitude, and teams that want to add, drop or fix a domain without retraining the rest. It gives ground where coupling helps: when domains share skills, multi-task RL or the plain sum moves faster along the shared directions, and OPM recovers that only through larger trainer steps. The transfer term and visibility, both logged at every rendezvous, tell which situation a run is in.

Limitations. The guarantees assume lossless rounds, and we have not shown that they exist at LLM scale. In the 4B merge the residual after 30 CG iterations is still 15% of drift, and the reasoning domains overlap. The informative statements at realistic drift are second-order or about KL, not reward. The trajectory Fisher grows with tokens, so long-response domains get more weight in shared directions. OPM keeps one model copy per domain in memory. The Fisher is estimated from a subsample, and the evidence for the guarded merge comes from one two-domain test. At one shot, a stale metric tied the on-policy one on benchmarks. All results are at 4B and 9B on dense models; mixture-of-experts models and larger scales are untested.

Related work. The barycenter objective comes from Distral and Divide-and-Conquer RL ,, and policy distillation into one network from Rusu et al.. The merge operator comes from Fisher merging, RegMean and MaTS ,,, all applied once and in supervised settings. Liu et al. showed that MOPD is Fisher merging to second order. WARP iterates merging and re-initialisation across parallel RLHF runs on one task . Co-Evolving Policy Distillation couples per-domain GRPO branches during training through distillation , and M2A merges a math vector into an agent only along the null space of agent-critical activations , a close cousin of Theorem 1. Federated learning studies periodic averaging under heterogeneity ,,,, including function-space aggregation , federated GRPO and federated natural policy gradient for multi-task RL . As far as we know, OPM is the first to use the on-policy trajectory Fisher inside RL, where Theorem 1 holds, and the first to trigger a weight merge on trajectory KL. Appendix H discusses each in detail.

Conclusion. Combining RL domains is a choice of how often and in which geometry. Choosing often and in the on-policy KL geometry gives a merge with a closed form that removes cross-domain terms of both signs. It guarantees per-domain progress where domains can be separated, costs speed where they agree, and takes about ten minutes on a 9B model once the solver is guarded. Merged once at 4B, it beats task arithmetic and MOPD and loses to the plain sum, for reasons the theory predicts. Whether merging often closes that gap, and what OPM buys against multi-task RL and MOPD at scale, is what the pending results above settle.

#Citation

@article{wang2026onpolicymerge,
  author  = {Bowen Wang},
  title   = {On-Policy Merge Stabilizes Multi-Task Reinforcement Learning},
  journal = {bowenwang.me},
  year    = {2026},
  month   = nov,
  note    = {https://bowenwang.me/blog/on-policy-merge}
}

#Appendices

The appendices hold everything the main text points to. A gives the notation, the derivation and the exact relations between methods. B states every theoretical result with its status and proof. C covers the merge solver and its cost. D has the one-shot 4B study, E the toy, F the main experimental setup, G the secondary findings and H the extended related work. Status labels: [T] theorem, [P] proposition proved under the stated assumptions, [C] conjecture with numerical support.

A. Notation, derivation and the map in formulas

Notation. Domains i=1,…,ni=1,\dots,n share parameters θ∈RP\theta\in\mathbb{R}^P. A rollout ξ\xi of domain ii has law PθiP^i_\theta and return Ji(θ)=EPθi[Ri]J_i(\theta)=\mathbb{E}_{P^i_\theta}[R_i], Ri∈[0,1]R_i\in[0,1]; Φ=∑iJi\Phi=\sum_iJ_i. The score is S(ξ)=∑tstS(\xi)=\sum_ts_t, st=∇log⁡πθ(at∣ht)s_t=\nabla\log\pi_\theta(a_t\mid h_t). The trajectory Fisher of domain ii at θ\theta is Fi(θ)=EPθi[SS⊤]F_i(\theta)=\mathbb{E}_{P^i_\theta}[SS^\top]; without an argument, Fi=Fi(θi)F_i=F_i(\theta_i) is the on-policy Fisher at trainer ii. We write gi=∇Jig_i=\nabla J_i. From the anchor θˉ\bar\theta, trainer ii takes KK local steps to θi=θˉ+Δi\theta_i=\bar\theta+\Delta_i; the merge returns θˉ+=θˉ+d\bar\theta^{+}=\bar\theta+d. Drift is Di=KL(Pθii∥Pθˉi)D_i=\mathrm{KL}(P^i_{\theta_i}\|P^i_{\bar\theta}), residual is εi(d)=KL(Pθii∥Pθˉ+di)\varepsilon_i(d)=\mathrm{KL}(P^i_{\theta_i}\|P^i_{\bar\theta+d}), with second-order part εi(2)(d)=12∥d−Δi∥Fi2\varepsilon_i^{(2)}(d)=\tfrac12\|d-\Delta_i\|^2_{F_i}. The OPM merge is d⋆=(∑iFi)+∑iFiΔid^\star=(\sum_iF_i)^{+}\sum_iF_i\Delta_i, the sum is dΣ=∑iΔid_\Sigma=\sum_i\Delta_i, and the others’ update seen from domain jj is sj=∑i≠jΔis_j=\sum_{i\ne j}\Delta_i. The transfer term is Tj=gj(θj)⊤sjT_j=g_j(\theta_j)^\top s_j. Throughout, A=∑iFiA=\sum_iF_i, b=∑iFiΔib=\sum_iF_i\Delta_i. There is no reference-KL penalty anywhere.

Derivation of (1) and (2)

For t<t′t<t', E[stst′⊤]=E[st E[st′∣ht′]⊤]=0\mathbb{E}[s_ts_{t'}^\top]=\mathbb{E}\big[s_t\,\mathbb{E}[s_{t'}\mid h_{t'}]^\top\big]=0 because the score has zero mean given its prefix, so Fi=E[∑tstst⊤]F_i=\mathbb{E}[\sum_ts_ts_t^\top]: a sum over the tokens of a rollout, not an average. The log-ratio of two rollout probabilities in the same environment is ∑tlog⁡πθi(at∣ht)−log⁡πθ(at∣ht)\sum_t\log\pi_{\theta_i}(a_t\mid h_t)-\log\pi_\theta(a_t\mid h_t), so KL(Pθii∥Pθi)=EPθii[∑tKL(πθi(⋅∣ht)∥πθ(⋅∣ht))]\mathrm{KL}(P^i_{\theta_i}\|P^i_\theta)=\mathbb{E}_{P^i_{\theta_i}}\big[\sum_t\mathrm{KL}(\pi_{\theta_i}(\cdot\mid h_t)\|\pi_\theta(\cdot\mid h_t))\big] and the environment terms cancel. The Hessian of a KL at its minimum is the Fisher of its first argument, which gives the quadratic form, and the normal equations ∑iFi(θ−θi)=0\sum_iF_i(\theta-\theta_i)=0 give (2). The pseudo-inverse picks the minimum-norm solution; directions in null⁡A\operatorname{null}A are invisible to every domain.

The exact barycenter is a visitation-weighted mixture [T]

For tabular policies over histories, the minimiser of (1) is π+(⋅∣h)=∑idi(h) πθi(⋅∣h)/∑idi(h)\pi^{+}(\cdot\mid h)=\sum_id_i(h)\,\pi_{\theta_i}(\cdot\mid h)\big/\sum_id_i(h), with di(h)d_i(h) the expected number of visits to hh under PθiiP^i_{\theta_i}. Proof. By the chain rule, (1) equals ∑h∑idi(h) KL(πθi(⋅∣h)∥π(⋅∣h))\sum_h\sum_id_i(h)\,\mathrm{KL}(\pi_{\theta_i}(\cdot\mid h)\|\pi(\cdot\mid h)). Each history separates; minimising −∑a(∑idi(h)πθi(a∣h))log⁡π(a∣h)-\sum_a\big(\sum_id_i(h)\pi_{\theta_i}(a\mid h)\big)\log\pi(a\mid h) over the simplex gives the mixture by Gibbs’ inequality. The minimum is zero iff all πθi(⋅∣h)\pi_{\theta_i}(\cdot\mid h) with di(h)>0d_i(h)>0 coincide. So a round is lossless iff the trainers agree on every history they share, and (2), the second-order version in logit space with Fishers at the trainers’ endpoints, inherits the mixture only while per-context logit moves stay of order one (B.6). ∎

Table A1. The displacement of the shared model under each method.

MethodDisplacement of the shared modelSignal
Multi-task RLηP∑ici∇Ji(θˉ)\eta P\sum_ic_i\nabla J_i(\bar\theta), every stepReward; cic_i are token shares of the batch; PP is the shared optimizer’s preconditioner
Cascade RLηP∇Jσ(b)(θˉ)\eta P\nabla J_{\sigma(b)}(\bar\theta), one domain per stage bbReward of the active domain only
Periodic sum or average∑iΔi\sum_i\Delta_i or 1n∑iΔi\tfrac1n\sum_i\Delta_i, every KK stepsReward, through each Δi\Delta_i
Merge-after (TA)λ∑iτi\lambda\sum_i\tau_i once, with τi\tau_i the whole specialist runNone at merge time
MOPD−η∑iFistudent(θ−θi)-\eta\sum_iF_i^{\rm student}(\theta-\theta_i) plus higher orders, iterated to its fixed pointTeacher log-probabilities on student states
OPM(∑iFi)+∑iFiΔi(\sum_iF_i)^{+}\sum_iF_i\Delta_i, when a trainer’s drift reaches its ceilingReward, through each Δi\Delta_i

Table A2. Exact limits [T]. Each row is a setting of OPM and what it becomes.

SettingBecomesWhat is lost
Sum every step; SGD trainers at rates ηci\eta c_i; cic_i token sharesMulti-task RL with SGD, exactly. With any other optimizer: one shared optimizer state, which is not the same as summing per-domain Adam stepsDecoupling: cross terms return at first order, in both signs, and batch shares set the weights
Merge every step, natural-gradient trainersMulti-task natural policy gradient, d=η(∑iFi)+∑igid=\eta(\sum_iF_i)^{+}\sum_ig_iNothing; exact only for natural-gradient trainers
Fishers mutually orthogonalThe sum—
All Fishers equalThe average—
One trainer active per stage, others idle, summedCascade RL: multi-task RL with every share but one at 0Protection of idle domains; keeping them in an OPM merge gives a protected cascade (B.5)
One round, interval = whole run, Euclidean metricTask arithmeticThe metric and locality
Merge once, Fisher on student states, solved by gradient descentMOPD, to second orderReward, and reuse of rollouts: a second stage, and a teacher ceiling
Objective (1) solved in function spaceDistral, Divide-and-Conquer RLThe closed form: distillation replaces a linear solve
Proofs of the limits

Multi-task SGD gives η∑icigi(θˉ)\eta\sum_ic_ig_i(\bar\theta); periodic sum at K=1K=1 with rates ηci\eta c_i gives the same. With a token-mean loss over the mixed batch the gradient is ∑i(toki/tok)g^i\sum_i(\mathrm{tok}_i/\mathrm{tok})\hat g_i, so cic_i is the token share. For natural-gradient trainers FiΔi=ηFiFi+gi=ηgiF_i\Delta_i=\eta F_iF_i^{+}g_i=\eta g_i by the range lemma (B.1), so d⋆=ηA+∑igid^\star=\eta A^{+}\sum_ig_i. Orthogonal ranges give A+=∑iFi+A^{+}=\sum_iF_i^{+}, so d⋆=∑jFj+FjΔj=∑jΔjd^\star=\sum_jF_j^{+}F_j\Delta_j=\sum_j\Delta_j for Δj∈range⁡Fj\Delta_j\in\operatorname{range}F_j. Equal Fishers give d⋆=(nF)+F∑iΔi=1n∑iΔid^\star=(nF)^{+}F\sum_i\Delta_i=\tfrac1n\sum_i\Delta_i. The rest hold by definition. Numerically: the natural-gradient identity to 3×10−143\times10^{-14}, the sum and average limits to 2×10−92\times10^{-9} and 4×10−124\times10^{-12}. ∎

B. Theory: statements and proofs

B.1 Two lemmas

Lemma 1 (range) [T]. gi(θ)∈range⁡Fi(θ)g_i(\theta)\in\operatorname{range}F_i(\theta). For every vv, v⊤gi=Cov⁡Pθi(Ri,v⊤S)v^\top g_i=\operatorname{Cov}_{P^i_\theta}(R_i,v^\top S), so ∣v⊤gi∣≤σiv⊤Fiv|v^\top g_i|\le\sigma_i\sqrt{v^\top F_iv} with σi2=Var⁡(Ri)\sigma_i^2=\operatorname{Var}(R_i); and ∥gi∥Fi+2≤σi2\|g_i\|^2_{F_i^{+}}\le\sigma_i^2. The lemma needs the gradient and the Fisher to be expectations under the same law, which is the on-policy property. It fails for supervised gradients against a model Fisher, and for stale or foreign-state Fishers.

Lemma 2 (exact surrogate) [T]. For any θ,θ′\theta,\theta' and baseline bb,

J(θ′)−J(θ)=EPθ[(R−b)log⁡Pθ′Pθ]+r,∣r∣≤sup⁡∣R−b∣  KL(Pθ∥Pθ′),J(\theta')-J(\theta)=\mathbb{E}_{P_\theta}\Big[(R-b)\log\frac{P_{\theta'}}{P_\theta}\Big]+r,\qquad|r|\le\sup|R-b|\;\mathrm{KL}(P_\theta\|P_{\theta'}),

and by Pinsker J(θ′)≥J(θ)−KL/2J(\theta')\ge J(\theta)-\sqrt{\mathrm{KL}/2}. To first order the change is ρ σ2ε(2)\rho\,\sigma\sqrt{2\varepsilon^{(2)}}, with ρ∈[−1,1]\rho\in[-1,1] the correlation between reward and the log-ratio. KL is a worst case over rewards. In 2,000 random checks neither bound was violated.

Proofs

Lemma 1. If Fiv=0F_iv=0 then E[(v⊤S)2]=0\mathbb{E}[(v^\top S)^2]=0, so v⊤S=0v^\top S=0 almost surely and v⊤gi=E[(Ri−b)v⊤S]=0v^\top g_i=\mathbb{E}[(R_i-b)v^\top S]=0. Hence gi⊥null⁡Fi=(range⁡Fi)⊥g_i\perp\operatorname{null}F_i=(\operatorname{range}F_i)^\perp. Cauchy–Schwarz on the covariance gives the bound; with v=Fi+giv=F_i^{+}g_i it gives the last claim.

Lemma 2. Let w=Pθ′/Pθw=P_{\theta'}/P_\theta; softmax gives full support and environment terms cancel. Since Eθw=1\mathbb{E}_\theta w=1, J(θ′)−J(θ)=Eθ[(w−1)(R−b)]J(\theta')-J(\theta)=\mathbb{E}_\theta[(w-1)(R-b)]. Write w−1=log⁡w+(w−1−log⁡w)w-1=\log w+(w-1-\log w); the second term is nonnegative with mean Eθ[−log⁡w]=KL(Pθ∥Pθ′)\mathbb{E}_\theta[-\log w]=\mathrm{KL}(P_\theta\|P_{\theta'}), which bounds ∣r∣|r|. Pinsker: ∣J(θ′)−J(θ)∣≤TV≤KL/2|J(\theta')-J(\theta)|\le\mathrm{TV}\le\sqrt{\mathrm{KL}/2}. ∎

B.2 Decoupling

Pythagoras [T]. For any PSD FiF_i, any Δi\Delta_i and any dd, ∑i12∥d−Δi∥Fi2=∑i12∥d⋆−Δi∥Fi2+12∥d−d⋆∥A2\sum_i\tfrac12\|d-\Delta_i\|^2_{F_i}=\sum_i\tfrac12\|d^\star-\Delta_i\|^2_{F_i}+\tfrac12\|d-d^\star\|^2_A. Also ∑iFjA+Fi=Fj\sum_iF_jA^{+}F_i=F_j for every jj.

Coupling [T]. To first order, with gradients and Fishers at θj\theta_j, the merged model’s return minus trainer jj’s is:

MergeJj(θˉ+d)−Jj(θj)J_j(\bar\theta+d)-J_j(\theta_j), first order
SumTjT_j, the cross terms, both signs
Average1nTj−(1−1n) gj⊤Δj\tfrac1nT_j-(1-\tfrac1n)\,g_j^\top\Delta_j
Task arithmetic, coefficient λ\lambdaλTj−(1−λ) gj⊤Δj\lambda T_j-(1-\lambda)\,g_j^\top\Delta_j
OPMuj⊤(∑iΩjiΔi−FjΔj)u_j^\top\big(\sum_i\Omega_{ji}\Delta_i-F_j\Delta_j\big), at most σj2εj(2)(d⋆)\sigma_j\sqrt{2\varepsilon_j^{(2)}(d^\star)} in size; zero in a lossless round

Here uj=Fj+gju_j=F_j^{+}g_j and Ωji=FjA+Fi\Omega_{ji}=F_jA^{+}F_i, with ∑iΩji=Fj\sum_i\Omega_{ji}=F_j. If FiFj=0F_iF_j=0 no merge couples the two domains.

Theorem 1 (decoupling) [T]. A round is lossless for every choice of updates iff rank⁡A=∑irank⁡Fi\operatorname{rank}A=\sum_i\operatorname{rank}F_i. In a lossless round, for every jj, Jj(θˉ+d⋆)−Jj(θj)=O(∥d⋆−Δj∥2)J_j(\bar\theta+d^\star)-J_j(\theta_j)=O(\|d^\star-\Delta_j\|^2) and Jj(θˉ+dΣ)−Jj(θˉ+d⋆)=Tj+O(∥⋅∥2)J_j(\bar\theta+d_\Sigma)-J_j(\bar\theta+d^\star)=T_j+O(\|\cdot\|^2). For log-linear policies π(a∣h)∝eϕ(h,a)⊤θ\pi(a\mid h)\propto e^{\phi(h,a)^\top\theta}, with linearly independent centred feature spans VjV_j of the domains’ reachable histories, exact merges with any full-support Fishers (endpoint, anchor or stale), and trainers whose update depends only on their own domain’s gradient and Fisher history, the anchor’s policy on domain jj after rr rounds equals that of a solo run of rKrK steps. Checks: in log-linear problems the OPM error is at most 3×10−163\times10^{-16}; in a small MLP it scales as t2t^2 and the sum’s change matches TjT_j to about 1% at small tt; the deviation from solo is at most 2×10−122\times10^{-12} for GD, natural gradient and persistent Adam at intervals 1–60. The constant in O(⋅)O(\cdot) grows with the null-space excursion ∥d⋆−Δj∥\|d^\star-\Delta_j\|; in the MLP, ∥d⋆∥\|d^\star\| was 8–12 times ∥dΣ∥\|d_\Sigma\|.

Proofs

Pythagoras. Q(d)=12d⊤Ad−d⊤b+cQ(d)=\tfrac12d^\top Ad-d^\top b+c with b∈range⁡Ab\in\operatorname{range}A, so Ad⋆=bAd^\star=b and Q(d)−Q(d⋆)=12∥d−d⋆∥A2Q(d)-Q(d^\star)=\tfrac12\|d-d^\star\|_A^2. Partition of unity: ∑iFjA+Fi=FjA+A=Fj\sum_iF_jA^{+}F_i=F_jA^{+}A=F_j because range⁡Fj⊆range⁡A\operatorname{range}F_j\subseteq\operatorname{range}A for PSD summands.

Coupling. By Taylor, Jj(θˉ+d)−Jj(θj)=gj⊤(d−Δj)+O(∥d−Δj∥2)J_j(\bar\theta+d)-J_j(\theta_j)=g_j^\top(d-\Delta_j)+O(\|d-\Delta_j\|^2), where d−Δjd-\Delta_j is sjs_j, 1nsj−(1−1n)Δj\tfrac1ns_j-(1-\tfrac1n)\Delta_j, λsj−(1−λ)Δj\lambda s_j-(1-\lambda)\Delta_j and d⋆−Δjd^\star-\Delta_j for the four rows. For OPM, gj=Fjujg_j=F_ju_j by Lemma 1 and Fjd⋆=∑iΩjiΔiF_jd^\star=\sum_i\Omega_{ji}\Delta_i. The bound is Lemma 1 with v=d⋆−Δjv=d^\star-\Delta_j.

Lossless iff independent ranges. The map d↦(F1d,…,Fnd)d\mapsto(F_1d,\dots,F_nd) takes values in ⊕irange⁡Fi\oplus_i\operatorname{range}F_i. Its kernel is ⋂inull⁡Fi=null⁡A\bigcap_i\operatorname{null}F_i=\operatorname{null}A, so its rank is rank⁡A\operatorname{rank}A, and it is onto iff rank⁡A=∑irank⁡Fi\operatorname{rank}A=\sum_i\operatorname{rank}F_i.

Main claim. If some dd has Q(d)=0Q(d)=0, Pythagoras gives Q(d⋆)=0Q(d^\star)=0, so Fj(d⋆−Δj)=0F_j(d^\star-\Delta_j)=0 for every jj. Lemma 1 at θj\theta_j gives gj(θj)⊤(d⋆−Δj)=0g_j(\theta_j)^\top(d^\star-\Delta_j)=0. Subtracting from the sum row gives the second display.

Log-linear policies. With full-support softmax, null⁡Fj(θ)=Nj\operatorname{null}F_j(\theta)=N_j, the set of vv with ϕ(h,a)⊤v\phi(h,a)^\top v independent of aa at every history reachable in domain jj, for every θ\theta. Domain jj’s policy, hence gjg_j, FjF_j and JjJ_j, depends on θ\theta only modulo NjN_j. Independence makes every round lossless, so θˉ+≡θj(modNj)\bar\theta^{+}\equiv\theta_j\pmod{N_j}. Induct over steps and rounds: if the anchor is congruent to the solo iterate and the trainer’s optimizer state equals the solo state, the trainer sees the same gradients and applies the same updates. Any full-support Fisher has range Nj⊥N_j^\perp, so endpoint, anchor and stale metrics all work. ∎

B.3 KL-efficiency and the natural-gradient limit

KL-efficiency [T]. (a) For any update dd, gj⊤d≤∥gj∥Fj+d⊤Fjdg_j^\top d\le\|g_j\|_{F_j^{+}}\sqrt{d^\top F_jd}, with equality iff FjdF_jd is a positive multiple of gjg_j. (b) If every trainer takes one natural-gradient step and the round is lossless, each domain gets exactly its solo gain: xi=A+gix_i=A^{+}g_i satisfies Fixi=giF_ix_i=g_i and Fjxi=0F_jx_i=0 for j≠ij\ne i, so every cross term gj⊤A+gig_j^\top A^{+}g_i vanishes. Without a lossless point, ∥gj∥A+2≤∥gj∥Fj+2\|g_j\|^2_{A^{+}}\le\|g_j\|^2_{F_j^{+}}: the own term shrinks and nonzero cross terms redistribute gain. (c) With natural-gradient trainers in a lossless round, the merged model attains (a) on every domain at once with Dj(2)(d⋆)=Dj(2)(Δj)D^{(2)}_j(d^\star)=D^{(2)}_j(\Delta_j). (d) Multi-task RL gives gj⊤d=η(cj∥gj∥2+∑i≠jcigi⊤gj)g_j^\top d=\eta(c_j\|g_j\|^2+\sum_{i\ne j}c_ig_i^\top g_j); it beats OPM on jj only by spending more drift on jj that happens to align with reward. Check: own-term share 1.000, identity to 10−1410^{-14}.

So multi-task natural gradient does pass every gradient through A+A^{+}, but in a lossless round each piece xix_i is invisible to every other domain. Genuine cross-domain effects appear only in shared directions, and there it averages rather than amplifies.

Tabular reduction [T]. With tabular policies over histories, one step per round, natural-gradient trainers and Fishers at the anchor, OPM is exactly natural policy gradient on the domain-mixture process, with advantage ∑iP(i∣h)Ai(h,⋅)\sum_iP(i\mid h)A_i(h,\cdot) and P(i∣h)∝di(h)P(i\mid h)\propto d_i(h). It inherits the dimension-free global rate of natural policy gradient (discounted setting; we do not re-derive episodic constants), whereas Euclidean softmax policy gradient can take time exponential in the number of states . The separation belongs to natural against Euclidean geometry, not to OPM, and holds for exact gradients.

Proofs

(a) gj⊤d=uj⊤Fjd≤uj⊤Fjujd⊤Fjdg_j^\top d=u_j^\top F_jd\le\sqrt{u_j^\top F_ju_j}\sqrt{d^\top F_jd} with uj⊤Fjuj=gj⊤Fj+gju_j^\top F_ju_j=g_j^\top F_j^{+}g_j. (b) Apply a lossless merge to Δi=Fi+gi\Delta_i=F_i^{+}g_i and Δk=0\Delta_k=0 for k≠ik\ne i: then b=gib=g_i, so Fixi=giF_ix_i=g_i and Fkxi=0F_kx_i=0. Hence gj⊤xi=uj⊤Fjxi=0g_j^\top x_i=u_j^\top F_jx_i=0 for j≠ij\ne i and gi⊤xi=∥gi∥Fi+2g_i^\top x_i=\|g_i\|^2_{F_i^{+}}. For A⪰FjA\succeq F_j and gj∈range⁡Fjg_j\in\operatorname{range}F_j, gj⊤A+gj=sup⁡v(2gj⊤v−v⊤Av)≤sup⁡v(2gj⊤v−v⊤Fjv)=gj⊤Fj+gjg_j^\top A^{+}g_j=\sup_v(2g_j^\top v-v^\top Av)\le\sup_v(2g_j^\top v-v^\top F_jv)=g_j^\top F_j^{+}g_j. (c) Fjd⋆=FjΔj=ηgjF_jd^\star=F_j\Delta_j=\eta g_j gives equality in (a). Tabular: Fisher and gradient are block-diagonal over histories, Fi=⊕hdi(h)(diag⁡πh−πhπh⊤)F_i=\oplus_hd_i(h)(\operatorname{diag}\pi_h-\pi_h\pi_h^\top) and gi=⊕hdi(h) πh⊙Ai(h,⋅)g_i=\oplus_hd_i(h)\,\pi_h\odot A_i(h,\cdot), so the merged natural step on block hh is η∑idi(h)Ai(h,⋅)/∑idi(h)\eta\sum_id_i(h)A_i(h,\cdot)/\sum_id_i(h) up to a constant. ∎

B.4 The price

Proposition 3 [P], in the stated models. (A) Amplification against averaging. With nn domains sharing a direction uu with common curvature ff and agreeing objectives, each trainer’s KK steps solve x=1−(1−ηf)Kx=1-(1-\eta f)^K of it. OPM moves xx, the sum nxnx, multi-task RL 1−(1−nηf)K1-(1-n\eta f)^K. With an interior optimum the per-round error factors are 1−x1-x and ∣1−nx∣|1-nx|; the sum is better iff x<2/(n+1)x<2/(n+1) and diverges iff x>2/nx>2/n. With a boundary optimum the sum is at least as good at every xx. (B) Multi-task natural gradient redistributes. Along uu, it moves ηaˉ/f\eta\bar a/f, with aˉ\bar a the mean gradient component; domain jj’s gain relative to solo is aˉ/aj\bar a/a_j, and the total is at most the sum of solo gains. Euclidean multi-task RL’s ratio is ∑iai/aj\sum_ia_i/a_j, which amplifies. (C) Fixed points under conflict. Multi-task RL: ∑icigi=0\sum_ic_ig_i=0. OPM at one step per round with Δi=ηiPigi\Delta_i=\eta_iP_ig_i: ∑iηiFiPigi=0\sum_i\eta_iF_iP_ig_i=0, which for natural-gradient trainers is ∑iηigi=0\sum_i\eta_ig_i=0: the trainer step sizes act as batch shares. Trainers that solve their round (quadratic model, Fi=HiF_i=H_i) reach the optimum of the unweighted ∑iJi\sum_iJ_i, whatever the speeds. Checks: all predictions to 10−1110^{-11}; OPM with natural-gradient rates (0.09, 0.01) lands exactly where multi-task RL with shares (0.9, 0.1) does.

Rate floor [P]. Assume relative smoothness along natural-gradient steps, Jj(θ+u)≥Jj(θ)+gj⊤u−L2u⊤FjuJ_j(\theta+u)\ge J_j(\theta)+g_j^\top u-\tfrac L2u^\top F_ju; null-space curvature Jj(θ+v)≥Jj(θ)−M2∥v∥2J_j(\theta+v)\ge J_j(\theta)-\tfrac M2\|v\|^2 for v∈null⁡Fjv\in\operatorname{null}F_j; lossless exact merges; ∥Fj+gj∥≤G\|F_j^{+}g_j\|\le G and κc=max⁡i∥Fi∥/λmin⁡+(A)\kappa_c=\max_i\|F_i\|/\lambda^{+}_{\min}(A) along the run. With KK natural-gradient steps per round, η≤1/L\eta\le1/L, and T=KRT=KR:

1nT∑r,j,k∥gj(θjr,k)∥Fj+2≤2(Φ∗−Φ(θˉ0))nηT+M(1+nκc)2 ηK G2.\frac1{nT}\sum_{r,j,k}\|g_j(\theta^{r,k}_j)\|^2_{F_j^{+}}\le\frac{2(\Phi^\ast-\Phi(\bar\theta_0))}{n\eta T}+M(1+n\kappa_c)^2\,\eta K\,G^2 .

With M=0M=0 (log-linear) this is the solo rate for every domain, independent of KK. With M>0M>0 there is an O(ηK)O(\eta K) floor, the analogue of client drift in local SGD ,. In a check the deviation from solo grew from 0.03 to 0.14 as KK went from 1 to 30.

Proofs

(A) KK gradient steps contract the error by (1−ηf)K(1-\eta f)^K. Identical trainers make OPM the average (Table A2), error 1−x1-x; the sum gives ∣1−nx∣|1-nx|. For nx≤1nx\le1 the sum is better since (n−1)x>0(n-1)x>0; for nx>1nx>1 it is better iff nx−1<1−xnx-1<1-x. Divergence is ∣1−nx∣>1|1-nx|>1. With a boundary optimum JjJ_j is non-decreasing along uu and the sum lies further along it. (B) Along uu, A+∑g=∑iai/(nf)A^{+}\sum g=\sum_ia_i/(nf); gains are ajηaˉ/fa_j\eta\bar a/f against solo ηaj2/f\eta a_j^2/f, and ∑jajaˉ=naˉ2≤∑jaj2\sum_ja_j\bar a=n\bar a^2\le\sum_ja_j^2. (C) At one step, d=A+∑iηiFiPigid=A^{+}\sum_i\eta_iF_iP_ig_i vanishes iff the sum does; substitute Pi=Fi+P_i=F_i^{+} or Pi=IP_i=I. Round-solving trainers reach θ+Πi(ai−θ)\theta+\Pi_i(a_i-\theta), and the merge returns (∑iHi)−1∑iHiai(\sum_iH_i)^{-1}\sum_iH_ia_i.

Rate floor. Each natural-gradient step gains at least η2∥gj∥Fj+2\tfrac\eta2\|g_j\|^2_{F_j^{+}}. ∥Δj∥≤ηKG\|\Delta_j\|\le\eta KG and ∥d⋆∥≤nκcηKG\|d^\star\|\le n\kappa_c\eta KG, so ∥d⋆−Δj∥≤(1+nκc)ηKG\|d^\star-\Delta_j\|\le(1+n\kappa_c)\eta KG inside null⁡Fj\operatorname{null}F_j, costing at most M2(1+nκc)2η2K2G2\tfrac M2(1+n\kappa_c)^2\eta^2K^2G^2. Sum over jj, telescope over RR rounds, and divide by nηT/2n\eta T/2. ∎

B.5 Regression under multi-task RL; cascade forgetting

Multi-task RL can regress a domain; OPM cannot [T]. An MTRL step changes JjJ_j by η(cjgj⊤Pgj+∑i≠jcigj⊤Pgi)+O(η2)\eta(c_jg_j^\top Pg_j+\sum_{i\ne j}c_ig_j^\top Pg_i)+O(\eta^2). Whenever the cross sum is negative, a small enough cj∥gj∥c_j\|g_j\| makes domain jj regress. In the example J1=aσ(s)J_1=a\sigma(s), J2=b(1−σ(s+p))J_2=b(1-\sigma(s+p)), J2J_2 falls at the origin iff c1a>2c2bc_1a>2c_2b. The Fisher ranges span⁡(1,0)\operatorname{span}(1,0) and span⁡(1,1)\operatorname{span}(1,1) are independent, so Theorem 1 applies to OPM. Numbers are in the main text.

Cascade forgetting [T]. During the stage that trains domain ii, exactly, Jj(θend)−Jj(θstart)=∑kgj(θk)⊤Δθk+12∑kΔθk⊤∇2Jj(ξk)ΔθkJ_j(\theta_{\rm end})-J_j(\theta_{\rm start})=\sum_kg_j(\theta_k)^\top\Delta\theta_k+\tfrac12\sum_k\Delta\theta_k^\top\nabla^2J_j(\xi_k)\Delta\theta_k, bounded only by KL(Pstartj∥Pendj)/2\sqrt{\mathrm{KL}(P^j_{\rm start}\|P^j_{\rm end})/2}, domain jj’s drift during the stage, which cascade does not control. A protected cascade keeps idle trainers in the merge with Δj=0\Delta_j=0; in a lossless round the anchor moves inside null⁡Fj\operatorname{null}F_j, exactly for log-linear policies. In the example, cascade takes J2J_2 from 0.9987 to 0.0735 (accumulated first-order term −0.891, remainder −0.034); protected cascade keeps 0.9987 with the same J1J_1 of 0.997. The OPM merge is symmetric in the domains; cascade depends on the order and passes positive transfer forward along it, which protection blocks.

B.6 The trigger, locality and the choice of δ

The trigger is a trust region [T] for (a) and (c), [P] second order for (b). Let rounds end when max⁡iDi≥δ\max_iD_i\ge\delta, with one-step overshoot ω\omega. Then Di≤δ+ωD_i\le\delta+\omega for every trainer, whatever its optimizer. (a) The exact minimiser of (1) has ∑iεi≤∑iDi≤n(δ+ω)\sum_i\varepsilon_i\le\sum_iD_i\le n(\delta+\omega), because the old anchor is feasible. An inexact merge inherits this only if its candidate set contains a point with residual at most ∑iDi\sum_iD_i; the production merge guarantees only “no worse than the sum” (Proposition 5). (b) On domain jj the new anchor is within ∥d∥Fj≤2εj(2)+2Dj(2)\|d\|_{F_j}\le\sqrt{2\varepsilon^{(2)}_j}+\sqrt{2D^{(2)}_j} of the old one. (c) Jj(θˉ+)≥Jj(θˉ)+Gj−εj/2J_j(\bar\theta^{+})\ge J_j(\bar\theta)+G_j-\sqrt{\varepsilon_j/2}, with Gj=Jj(θj)−Jj(θˉ)G_j=J_j(\theta_j)-J_j(\bar\theta). Without a reference KL, the trainers’ local solution is not a natural-gradient step, and nothing makes drift level off: the trigger is a ceiling plus a cap. KL penalties toward a shared model appear in federated policy gradient ; OPM replaces the penalty by the trigger and moves the anchor at every rendezvous.

Locality [P], lossless rounds, per-round displacements of order h=ℓ/Rh=\ell/R, with ℓ\ell the Fisher path length. The residual of a round is O(h3)O(h^3), and O(h4)O(h^4) along null directions, so over RR rounds it totals O(ℓ3/R2)O(\ell^3/R^2). One-shot merging is R=1R=1. Real conflicts have a second-order residual per round whose first-order reward cost frequency does not reduce.

Endpoint-Fisher collapse [T] for the logistic case, [C] for many rounds. One trainer pushes a logit to z1z_1; an idle trainer stays at 0. Formula (2) returns z+=f(z1)z1/(f(z1)+14)→0z^{+}=f(z_1)z_1/(f(z_1)+\tfrac14)\to0, with f=σ(1−σ)f=\sigma(1-\sigma), while the exact barycenter keeps σ(z+)=(σ(z1)+12)/2→34\sigma(z^{+})=(\sigma(z_1)+\tfrac12)/2\to\tfrac34. The locality ratio κ=KL(1)/(16 KL(14))\kappa=\mathrm{KL}(1)/(16\,\mathrm{KL}(\tfrac14)) is 1.00, 1.35, 1.92 and 8.9 after 1, 10, 20 and 200 steps, so the threshold κ≤1.5\kappa\le1.5 detects the collapse. In a multi-round tabular check at fixed budget, as the interval grows from 1 to 120, the quadratic merge’s idle-domain gain falls from 0.43 to 0.11 and the exact barycenter’s from 0.43 to 0.29.

Repair and data validity [T]. After a rendezvous trainers resume on-policy RL, so a round’s residual costs steps rather than final quality: the final bound involves only the last round. Whole-trajectory importance weights have χ2≥eKL−1\chi^2\ge e^{\mathrm{KL}}-1, so at residuals of tens of nats no trajectory-level reuse is possible, for OPM or anyone else. What makes in-flight reuse possible is per-token importance sampling, as in truncated importance sampling and IcePop , where χtok2≈2εtok\chi^2_{\rm tok}\approx2\varepsilon_{\rm tok}. One-shot per-token residuals at 4B are 0.01–0.44 nats (sum: 0.03–0.07).

Choosing δ [P], partly [C]. (1) Locality is a per-token condition, so set δi=Liτtok\delta_i=L_i\tau_{\rm tok} proportional to response length, or trigger on per-token KL. At 4B, κ≈1\kappa\approx1 toward the merged points at 0.02–0.04 nats per token. (2) The Fisher sample needs many more tokens than the effective rank of the subspace in which domains see each other, so it grows with the drift budget. (3) Rounds per unit of Fisher path scale as δ−1/2\delta^{-1/2}; take the largest δ\delta meeting (1) and (2) within the merge-cost budget. (4) Log κ\kappa and the in-sample against held-out KL gap at each rendezvous to tell estimation failures from locality failures.

B.7 MOPD

Teacher ceiling [T]. With Li(θ)=KL(Pθi∥Pθii)L_i(\theta)=\mathrm{KL}(P^i_\theta\|P^i_{\theta_i}), Ji(θ)≤Ji(θi)+Li(θ)/2J_i(\theta)\le J_i(\theta_i)+\sqrt{L_i(\theta)/2} by Pinsker on the reverse KL; at L=0L=0 the student equals the teacher. OPM has Ji(θˉ+)≥Ji(θi)−εi/2J_i(\bar\theta^{+})\ge J_i(\theta_i)-\sqrt{\varepsilon_i/2}, with θi\theta_i retrained on reward each round.

Stationarity [P], KL three times differentiable. ∇Li(θ)=Fi(θi)(θ−θi)+O(∥θ−θi∥2)\nabla L_i(\theta)=F_i(\theta_i)(\theta-\theta_i)+O(\|\theta-\theta_i\|^2). At d⋆d^\star the leading terms cancel by the normal equations, so ∥∇∑iLi(θˉ+)∥=O(∑i∥θˉ+−θi∥2)\|\nabla\sum_iL_i(\bar\theta^{+})\|=O(\sum_i\|\bar\theta^{+}-\theta_i\|^2); for task arithmetic and the sum it is O(∥⋅∥)O(\|\cdot\|). In a lossless round ∑iLi(θˉ+)=O(∥⋅∥3)\sum_iL_i(\bar\theta^{+})=O(\|\cdot\|^3). Student-state and trainer-state Fishers give quadratic forms that differ by a relative O(εi)O(\sqrt{\varepsilon_i}). In a check the OPM gradient fell from 0.60 to 0.0034 as t2t^2; the sum’s fell as tt. The second-order equivalence of MOPD and Fisher merging is due to Liu et al..

Cost identity [T]. MOPD’s second stage costs ∑iNis(cigen+cienv+citeach)\sum_iN^s_i(c^{\rm gen}_i+c^{\rm env}_i+c^{\rm teach}_i), including fresh environment interaction. An OPM rendezvous needs none: a few Fisher-vector products per trainer on cached rollouts plus forward passes for the line search.

B.8 The transfer predictor and a regime map

Transfer predictor [T] as an estimator, [C] as a predictor. Tj=Cov⁡Pθjj(Rj, Sj⊤sj)T_j=\operatorname{Cov}_{P^j_{\theta_j}}(R_j,\,S_j^\top s_j) is estimable without bias from trainer jj’s own batch: rewards plus one forward-mode JVP per rollout along sjs_j. For natural-gradient-like trainers, Tj=Gj(1)⟨Δj,sj⟩Fj/∥Δj∥Fj2T_j=G^{(1)}_j\langle\Delta_j,s_j\rangle_{F_j}/\|\Delta_j\|^2_{F_j}, computable by polarisation from three KLs. In an exact check with Adam trainers and a CG merge, its correlation with Jj(sum)−Jj(merge)J_j(\text{sum})-J_j(\text{merge}) was 0.84–0.87 for K≤10K\le10, falling to 0.27 at K=30K=30. The sign was always right for agreeing shared contexts.

Per domain and round, define visibility Vj=∥sj∥Fj2/∥Δj∥Fj2≈εj(sum)/DjV_j=\|s_j\|^2_{F_j}/\|\Delta_j\|^2_{F_j}\approx\varepsilon_j(\text{sum})/D_j, alignment ρj=⟨Δj,sj⟩Fj/(∥Δj∥Fj∥sj∥Fj)\rho_j=\langle\Delta_j,s_j\rangle_{F_j}/(\|\Delta_j\|_{F_j}\|s_j\|_{F_j}), and the sum’s first-order advantage relative to own gain Cj=ρjVj=Tj/Gj(1)C_j=\rho_j\sqrt{V_j}=T_j/G^{(1)}_j.

RegimeDiagnosticPredictionEvidence so far
Low visibilityVj≪1V_j\ll1Every merge ties4B IF and agent: VV 0.03–0.18; sum ≈ OPM on IFEval (89.6 vs 89.5) and BFCL (72.4 vs 73.3)
Aligned overlapVj≳0.3V_j\gtrsim0.3, Cj>0C_j>0Sum and MTRL beat OPM on the mean4B reasoning: CC +0.17 to +0.26 (estimated), VV 0.5–3.1; sum wins on AIME. Toy positive variants at small budgets
Conflicting overlap or small shareCj<0C_j<0OPM wins, most on the worst domainToy conflict variants; literature pairs IF and tool use, search and math
Non-localκ>1.5\kappa>1.5; saturated trainersExact closed form collapses; line search neededToy at interval 50: exact OPM 0.07–0.37, CG-10 0.61–0.91, sum 0.54–0.76

Falsifiable predictions for the main runs: (P1) per domain, the sign of Jj(sum)−Jj(merge)J_j(\text{sum})-J_j(\text{merge}) is the sign of TjT_j; (P2) OPM is at least multi-task RL on the worst domain exactly where Cj<0C_j<0, and below it on the mean for reasoning pairs unless trainer learning rates are raised; (P3) beyond κ>1.5\kappa>1.5, the line search picks γ→0\gamma\to0 and CG gains vanish; (P4) the best task-arithmetic coefficient for RL specialists stays near 1.

B.9 What is not claimed

  • No general rate advantage over well-tuned multi-task RL. OPM is slower along agreeing shared directions, up to nn times in the exponent, and has an O(ηK)O(\eta K) floor with nonlinear networks.
  • No positive transfer within a round. Frequency does not change this for exact lossless merges (refuted: “frequent rendezvous captures transfer”).
  • No reward guarantee at realistic trajectory KL. Pinsker-type bounds are vacuous above about 2 nats; 4B specialists drift 75–248.
  • Lossless points are assumed, not shown, at LLM scale. At 4B the residual after 30 CG iterations is still 15% of drift.
  • Speed invariance holds only in lossless directions or with round-solving trainers; under conflict, step sizes act as weights.
  • No invariance to rollout length: long-response domains weigh more in shared directions.
  • Exact gradients and Fishers are assumed except in Proposition 5 and the CG step criterion; the tabular rate is inherited from the discounted literature.

C. The merge solver and its cost

Proposition 5 (the production merge) [T]. Candidates are d(γ)=dΣ+γ(xk−dΣ)d(\gamma)=d_\Sigma+\gamma(x_k-d_\Sigma) with γ∈Γ={1,12,14,0}\gamma\in\Gamma=\{1,\tfrac12,\tfrac14,0\}. Let E^(γ)\hat E(\gamma) estimate E(γ)=∑iεi(d(γ))E(\gamma)=\sum_i\varepsilon_i(d(\gamma)) on fresh trainer rollouts. If max⁡γ∣E^(γ)−E(γ)∣≤e\max_\gamma|\hat E(\gamma)-E(\gamma)|\le e, then E(γ^)≤E(sum)+2eE(\hat\gamma)\le E(\text{sum})+2e. With per-rollout log-ratio spread ss and MM rollouts, e=O(slog⁡∣Γ∣/M)e=O(s\sqrt{\log|\Gamma|/M}). Proof. E(γ^)≤E^(γ^)+e≤E^(0)+e≤E(0)+2eE(\hat\gamma)\le\hat E(\hat\gamma)+e\le\hat E(0)+e\le E(0)+2e by a union bound over sub-Gaussian means. ∎ The guarantee is about total KL, not each domain’s, measured on the trainers’ distributions, and silent about reward. Estimating E^(γ)−E^(0)\hat E(\gamma)-\hat E(0) on the same rollouts shrinks ss. Adding a candidate on the anchor side would restore the nδn\delta bound of B.6.

When CG raises the true KL [T] for the criterion, [C] for the attribution. With A=∑FiA=\sum F_i, A^=∑F^i+μI\hat A=\sum\hat F_i+\mu I, rr the true and r^\hat r the sampled residual of (2) at the sum, the first CG step raises the true quadratic iff r^⊤r/∥r^∥2<12 r^⊤Ar^/r^⊤A^r^\hat r^\top r/\|\hat r\|^2<\tfrac12\,\hat r^\top A\hat r/\hat r^\top\hat A\hat r; likewise for later directions. Two failure sources: estimation noise, with relative size growing as reff/Ntok\sqrt{r_{\rm eff}/N_{\rm tok}}, and curvature underestimated along the step. Proof: for q(x)=12x⊤Ax−b⊤xq(x)=\tfrac12x^\top Ax-b^\top x and r=b−Ax0r=b-Ax_0, q(x0+αp)−q(x0)=−αp⊤r+12α2p⊤Apq(x_0+\alpha p)-q(x_0)=-\alpha p^\top r+\tfrac12\alpha^2p^\top Ap; substitute p=r^p=\hat r and α=∥r^∥2/r^⊤A^r^\alpha=\|\hat r\|^2/\hat r^\top\hat A\hat r. In synthetic checks with κ≈1\kappa\approx1 three CG iterations cut the true residual by 40–50%; with saturated trainers (κ≈16\kappa\approx16 at the sum) five iterations cut the quadratic residual by 93% but the true one by only 5–24%. The 9B diagnosis in the main text (secant underestimation along the stiffest directions) is consistent with the second source.

Cost model. Per merge, Tmerge=Trhs+Nitmax⁡iTFVP(Ti)+NevalTKL+NitTarT_{\rm merge}=T_{\rm rhs}+N_{\rm it}\max_iT_{\rm FVP}(T_i)+N_{\rm eval}T_{\rm KL}+N_{\rm it}T_{\rm ar}, with TFVP(T)=(T/G)(nfdcfwdfp32+ccal+cvjp+chead)T_{\rm FVP}(T)=(T/G)(n_{\rm fd}c^{\rm fp32}_{\rm fwd}+c_{\rm cal}+c_{\rm vjp}+c_{\rm head}) for TT cached action tokens on GG GPUs and central differences (nfd=2n_{\rm fd}=2). The all-reduce costs about 10 ms per billion parameters. Measured on B200 for the 9.4B hybrid model, in GPU-milliseconds per generated token:

Componentfp32TF32bf16
Forward, reference linear-attention kernel1.390.60–
Forward, fused linear-attention kernel1.080.210.17
Forward + backward (checkpointed)≈ 3.6–0.44
Vocabulary-softmax head0.510.06–
Full FVP, CG vectors in CPU memory20–24––
Full FVP, CG vectors on GPU6.9––

Moving the vectors to the GPU took one product on 11k tokens from 66 s to 19 s. One product over about 36k tokens takes 100–120 s. Further levers we have not yet adopted: running the vector-Jacobian product in bf16 (it is linear in the cotangent, and bf16 forward+backward costs 0.44 against 3.6 in fp32), TF32 finite-difference forwards with fp32 refinement, subsampling the curvature batch as Hessian-free optimisation and TRPO do ,, warm starts across rendezvous, and running the solve on spare GPUs while training continues.

The 9B test case. Two trainers branched from MiMo-V2.6-Distill-Qwen-9B: math for 40 steps and instruction following for 30. Fisher-vector products on 32 rollouts per trainer, residuals on held-out rollouts, damping μ=10−3\mu=10^{-3} of the mean curvature. Held-out drifts 76.4 and 49.0 nats. Along the sum-minus-own direction the quadratic model holds, κ\kappa 0.87–1.0. With a finite-difference step of 0.3, CG with the full step gave 112.7 nats after two iterations and 214.7 after three; with step 0.01 the numbers are those of Table 2. The per-trainer residuals at the chosen point (three iterations, γ=12\gamma=\tfrac12) are 3.0 and 5.5 nats, against 5.2 and 8.1 for the sum.

System. Each trainer is an ordinary GRPO trainer with a hook: after each optimizer step it runs the frozen anchor on about one rollout in eight to update its drift. At a rendezvous, the merge runs as a collective across trainers: every trainer computes its own Fisher-vector products on its own GPUs and cached tokens (a forward-mode pass for the change in logits, multiplication by diag⁡(π)−ππ⊤\operatorname{diag}(\pi)-\pi\pi^\top, and a backward pass), one all-reduce per iteration sums them, and the line search scores the candidates on held-out rollouts. Rendezvous are non-blocking: a slow trainer contributes its current weights and its latest batch’s Fisher. Each trainer keeps its optimizer moments, learning rate, batch size, rollout servers and sandboxes; the trainers share only a weight format. Logged at every rendezvous: drift, residual per γ\gamma, the chosen γ\gamma, CG iterations, tokens, wall-clock, κ\kappa, and TjT_j.

D. One-shot merge of released 4B specialists

Artifacts. Base Qwen3-4B-Instruct-2507 and the five fp32 specialists of Wu et al., with their training and test data and their verl-based code. Their recipe: GRPO with 128 prompts × 16 rollouts per step, learning rate 10−610^{-6}, 300 steps, responses up to 16,384 tokens, temperature 1, and a KL loss to the base with coefficient 0.001, which is effectively no penalty. The released task-arithmetic merge uses λ=0.6\lambda=0.6 over the fp32 specialists and is stored in bf16. Casting each fp32 specialist to bf16 reproduces the released bf16 checkpoint bit for bit.

Measurement. For each specialist, 1,024 rollouts on its own training prompts plus 256 held-out prompts. Every KL is the exact per-token KL over the full vocabulary, summed over the model’s tokens, scored in true fp32. Ratios use a paired bootstrap.

Drift. Trajectory KL to the base (nats) and per-token KL: math 248 / 0.040, agent 177 / 0.225, code 155 / 0.024, IF 100 / 0.67, science 75 / 0.029. The rankings by trajectory and by per-token KL differ.

Interference. The index Iij=KL(θj∥θj+τi)/KL(θj∥base)I_{ij}=\mathrm{KL}(\theta_j\|\theta_j+\tau_i)/\mathrm{KL}(\theta_j\|\text{base}) (rows: added update ii; columns: scored specialist jj):

i\ji\backslash jmathsciencecodeIFagent
math1.181.231.780.0110.040
science0.2880.9890.4290.0100.017
code0.1560.1740.9480.0040.027
IF0.0810.2780.1730.9700.078
agent0.0880.1650.2860.0120.952

The off-diagonal median is 0.122. Global task-vector cosines are at most 0.022 (math–science); the Spearman correlation between IijI_{ij} and cosine is 0.64. The IF and agent specialists are barely moved by other updates, but their own updates move the reasoning specialists.

Locality. Along every direction, KL(12)/(4 KL(14))\mathrm{KL}(\tfrac12)/(4\,\mathrm{KL}(\tfrac14)) lies in [0.80, 1.25]. The locality ratio is 0.87–1.10 across cross directions, at most 0.997 toward task arithmetic, and 0.96–1.17 toward the merged points.

The merge. CG on (∑iFi+μI)d=∑iFiΔi+μd0(\sum_iF_i+\mu I)d=\sum_iF_i\Delta_i+\mu d_0 from d0d_0, μ=10−3tr⁡(∑iFi)/P\mu=10^{-3}\operatorname{tr}(\sum_iF_i)/P by a Hutchinson estimate. FiF_i is the exact per-token Gauss–Newton Fisher J⊤(diag⁡π−ππ⊤)JJ^\top(\operatorname{diag}\pi-\pi\pi^\top)J, applied as a forward-mode then backward product, on 256 fp32 rollouts per domain (about 5.7M tokens in total). Against finite differences of 2 KL(θ∥θ+ϵv)/ϵ22\,\mathrm{KL}(\theta\|\theta+\epsilon v)/\epsilon^2 the quadratic form agrees within 1–3% along the math update, 0.05% along the code update and 0.1% along a random direction, which carries 5,800 times less curvature. Points: (a) from the sum, 10 iterations; (c) continued to 30; (b) from task arithmetic, 10; (d) stale metric, Fisher on base-model rollouts, 10.

Residual over drift, mean (max) over the five domains:

PointIn-sampleSame distribution, unusedHeld-out prompts
Task arithmetic, λ = 0.60.550 (1.194)0.557 (1.180)0.521 (1.121)
Sum, λ = 11.191 (3.085)1.190 (3.057)1.116 (2.899)
Released mixed RL1.051 (1.275)1.066 (1.452)1.073 (1.409)
Released MOPD0.348 (0.421)0.345 (0.416)0.340 (0.413)
OPM (a), 10 iterations0.322 (0.683)0.350 (0.776)0.311 (0.631)
OPM, 20 iterations0.202 (0.476)0.235 (0.576)0.205 (0.457)
OPM (c), 30 iterations0.148 (0.363)0.183 (0.468)0.158 (0.363)
OPM (b), from task arithmetic0.271 (0.537)0.294 (0.602)0.266 (0.498)
OPM (d), stale metric0.405 (0.836)0.416 (0.852)0.380 (0.799)

Per domain, OPM (a) against task arithmetic: math 0.29 / 0.33, science 0.68 / 0.71, code 0.58 / 1.19, IF 0.025 / 0.26, agent 0.037 / 0.26. The OPM point moves only 3.9% of ∥∑iτi∥\|\sum_i\tau_i\| from the sum after 10 iterations (6.4% after 20, 8.3% after 30), and 99.4–99.6% of that move lies outside the span of the five updates; even so it cuts the residual relative to the sum 3.7-fold at 10 iterations and 8-fold at 30. The relative CG residual is 0.13, 0.078 and 0.068 at 10, 20 and 30 iterations.

Evaluation. The protocol of Wu et al.: mean over 32 samples, temperature 0.6, top-p 0.95, responses up to 16,384 tokens, non-thinking mode, their scorers for each benchmark, and a local sandbox for LiveCodeBench. Our base, MOPD and IF-specialist IFBench scores (30.7 / 46.7 / 48.9) reproduce the published 30.3 / 47.3 / 49.0. Our task-arithmetic point is bit-identical to the released merge after the bf16 cast; evaluating both gives −0.32 ± 0.22, the run-to-run noise floor. The sum writes the longest code answers (about 6,000 tokens against 3,600 for task arithmetic; 5–7% hit the cap), the OPM points about 4,400–4,900.

Paired differences, mean over eight benchmarks ± standard error:

ComparisonMean of 8Detail
OPM (a) − task arithmetic+1.68 ± 0.42IFBench +10.1 ± 1.9; LCB v5 +3.3; AIME25 −2.8 ± 1.7
OPM (c) − task arithmetic+2.06 ± 0.40
OPM (d) − task arithmetic+1.90 ± 0.46
OPM (b) − task arithmetic−0.25 ± 0.35
Sum − task arithmetic+2.57 ± 0.47
OPM (a) − sum−0.89 ± 0.36AIME25 −4.7, AIME26 −3.4
OPM (c) − sum−0.51 ± 0.36
OPM (a) − OPM (d), on-policy vs stale metric−0.22 ± 0.31
OPM (a) − released MOPD+1.11 ± 0.40
OPM (a) fp32 − OPM (a) bf16+1.30 ± 0.34AIME +2.3 / +3.0, GPQA +1.1, LCB +1.6 / +1.2
OPM (a) fp32 − sum bf16+0.41 ± 0.37The sum was not evaluated in fp32

About 330 B200 GPU-hours were logged for the merges, residual scoring and evaluation. Only bf16 checkpoints of the 8B specialists were released, and bf16 rounding destroys their task vectors (Appendix G), so we did not merge at 8B.

E. The toy, with the corrected baseline

Setting. A 704-parameter policy shared by four chain tasks with horizons 2, 6, 16 and 16, trained with exact policy gradients and Adam; returns, gradients, Fishers and KLs are computed in closed form. Ten random problems per cell. Five variants set the cross-task effect: the original (gradient cosine −0.16; training one task changes another by −0.16), pos-soft (+0.21; +0.10), pos-strong (+0.50; +0.19), neg (−0.24; −0.19) and mixed (−0.19; −0.13). OPM uses an interval of 5 steps and 10 CG iterations unless noted. Script of the first version: /research/opm-toy/opm_toy.py.

Correction. The previous version of this post compared OPM with multi-task RL at its default learning rate, 0.688 against 0.963 on the original variant. Tuning the multi-task learning rate over 1–16× its default raises it to 0.949. The comparison below tunes both.

Mean final return at 50 steps per task (± about 0.01 or less):

Methodorigpos-softpos-strongnegmixed
Multi-task RL, default rate0.6880.8200.9100.6190.648
Multi-task RL, 4× rate0.9490.9900.9930.7620.804
Multi-task RL, best of 1–16×0.9491.0001.0000.7730.804
Specialists + task arithmetic, oracle coefficient0.5500.8150.8720.4100.459
Specialists + MOPD, best split0.6410.8750.9030.4980.552
Periodic sum, interval 50.7500.9410.9730.5580.634
OPM, exact merge0.9630.9570.9180.9690.965
OPM, CG-10, default rate0.9550.9370.9630.9740.967
OPM, CG-10, 2× rate0.9930.9900.9940.9960.995

At smaller budgets the positive variants favour multi-task RL more strongly. Against multi-task RL at 4× its rate, OPM (CG-10, default rate) trails by 0.23 / 0.20 at 10 steps per task, 0.18 / 0.10 at 25 and 0.05 / 0.03 at 50 (pos-soft / pos-strong). Under conflict, OPM leads at every budget once both rates are tuned (at 25 steps, 0.980 / 0.986 for OPM at 4× against 0.762 / 0.776 for the best multi-task rate, neg / mixed).

Transfer into an idle task (the task’s trainer is in the merge but does not train), 10 steps: periodic sum −0.13 / +0.13 / +0.33 / −0.19 / −0.15 over the five variants; exact OPM −0.07 / −0.08 / −0.11 / −0.03 / −0.05; OPM CG-10 −0.01 / +0.04 / +0.19 / −0.03 / −0.01. OPM blocks positive transfer into idle tasks, as Theorem 1 predicts. A task absent from the run receives about as much transfer under OPM as under multi-task RL or the sum (pos-strong: +0.30, +0.24, +0.33). In a sweep of the interval at 50 steps (pos-strong), idle-task transfer under exact OPM is +0.48 at an interval of 1, more than the sum’s +0.38, and −0.21 at 50; CG-10 goes from +0.49 to +0.13. A tabular check of the same quantity gives 0.43 against 0.45, so the excess at interval 1 is probably an artefact of summing Adam steps; we flag it as unexplained.

A reward-aware merge. Adding the trainers’ gradients to the merge, θˉ+A+∑i(FiΔi+gi/βm)\bar\theta+A^{+}\sum_i(F_i\Delta_i+g_i/\beta_m), looks like a way to recover transfer. Because gi∈range⁡Fig_i\in\operatorname{range}F_i, it is exactly OPM applied to trainers that each took one extra natural-gradient step (identity error at most 2.6×10−32.6\times10^{-3}). Its gain is flat across variants (+0.21 to +0.26 at 10 steps), a step-size effect that OPM with a 4× trainer rate reproduces within about 0.02. It remains an ablation.

Guarantees. Pinsker’s bound held in all 79,012 trainer–round pairs. OPM CG-10 never lost more than a trainer’s gain in a round, and no CG-merged run ended with a task below its starting point; multi-task RL, task arithmetic and the sum left 20–37% of tasks below it in the conflict variants.

Results from the first toy version (same problem, original variant, 50 steps per task). These do not involve the multi-task baseline and stand.

  • Metric: Fisher from the trainer’s current rollouts 0.96; from the anchor’s, one round old, 0.95; from the base model’s, never refreshed, 0.13.
  • Operator: average 0.48, diagonal Fisher 0.47, layer Grams 0.62, K-FAC 0.71, sum 0.75, ten likelihood steps 0.84, CG with 3 iterations 0.82, with 10 iterations 0.955, exact 0.963. Cheap layer-wise approximations did worse than the plain sum.
  • Interval: every 1, 5, 10, 25, 50 steps gave 0.97, 0.96, 0.93, 0.34, 0.08. With 100 steps per task, a KL trigger of 0.3 nats reached 0.987 with 6.6 rendezvous; a fixed interval with 5 rendezvous reached 0.88.
  • Staleness after a rendezvous at interval 5: median residual 0.0003 nats and effective sample size 99.9% for OPM, against 0.059 nats and 86% for the sum.
  • A slow task given a fifth of the steps: final success 0.03 under multi-task RL, 0.01 under the periodic sum, 0.70 under OPM, from a start of 0.25. If it never trains: 0.01, 0.00 and 0.14, so protection is per round and partial.

F. Main experimental setup

Model. MiMo-V2.6-Distill-Qwen-9B , a 9.4B dense model with the Qwen3.5-9B architecture, already supervised-fine-tuned. Thinking is on in every domain with one chat template shared by all specialists and arms, so weights stay mergeable; earlier turns’ thinking is kept in agentic domains. The language-model head runs in fp32. The vision tower and multi-token-prediction head are excluded from merges and keep the anchor’s values.

DomainTraining tasksVerifierEvaluation
Software engineeringMiMo-V2.6-RL-oss code split (2,602) and SWE-smithExecuted tests in containersSWE-bench Verified, SWE-bench Pro
TerminalFACET-Terminal-Tasks-6k and MiMo terminal tasks (64)Executed test scriptsTerminal-Bench 2.1
CyberMiMo-V2.6-RL-oss cyber split (1,000)Crash reproductionMiMo Cyber (mini), held out by project
Knowledge workMiMo-V2.6-RL-oss general split (925 environments)Per-environment checks plus an LLM rubric judgeMiMo General (mini), held out by industry
SearchASearcher training data over a local Wikipedia corpusAnswer match with aliasesBrowseComp-Plus
Tool useWorkBench, EnterpriseOps-Gym, EnvScaler (4,743 tasks)Executable state checks, binary rewardτ²-Bench, τ³-Bench, BFCL v4
MathPolaris, DAPO and Skywork-OR1, filteredAnswer matchAIME26, HMMT Feb 26
Instruction followingThe IF split of Wu et al. (2026)Rule checkersIFBench, IFEval
Competitive code (demo B only)The code split of Wu et al. (2026)Unit tests in a sandboxLiveCodeBench v6

Training repositories are decontaminated against the SWE-bench evaluation sets, and one AIME26 duplicate is removed from DAPO. A held-out suite no arm trains on: MMLU-Pro, GPQA and HLE (the older publicly available release, not comparable to published HLE numbers). Base scores are re-measured in our harnesses, with one fixed open harness per agentic domain for every arm. The knowledge-work judge is a closed-API model, a reproducibility caveat.

Shared RL recipe. slime, run asynchronously with bounded staleness (at most 4 for reasoning; for agentic domains the rollout-to-step time ratio, with the same rule in every arm). GRPO with groups of 16 : advantage is reward minus the group mean with no standard-deviation normalisation, a global token-mean loss, clipping 0.2 / 0.28, train–inference mismatch masking with IcePop , zero-variance groups dropped, no entropy bonus, no KL penalty. Environment-observation tokens are masked out of the loss, the drift KL and the Fisher.

DomainMax response / turnMax turnsMax context
Math32k1–
Instruction following16k1–
Competitive code32k1–
Tool use8k4064k
Search8k2464k
Software engineering8k100128k
Terminal8k150128k
Cyber8k100128k
Knowledge work8k100128k

Budget. Agentic domains: 32 prompts × 8 rollouts per step for 200 steps. Math and instruction following: 32 × 16 for 300 steps. These define the per-domain rollout budget every arm matches. Learning rates are chosen per trainer type (single-domain, multi-task, MOPD) from {1,2,4}×10−6\{1,2,4\}\times10^{-6} on held-out reward, using 15% of the budget on four domains, and frozen before the comparison. Three seeds per arm.

Arms. A1 multi-task RL: per-domain rollout quotas matched to the budget, uniform gradient weights per domain through a ledger rule, deficit scheduling with slow-source replay; a data-proportional mixture as a secondary variant. A2 merge-after: the best of task arithmetic at λ∈{0.4,0.6,0.8,1.0}\lambda\in\{0.4,0.6,0.8,1.0\} chosen on held-out reward, DARE-TA, TIES, the closed form of Liu et al. and a one-shot OPM merge. A3 MOPD: per-token advantage teacher minus student log-probability, clipped to ±5, 30–60 steps, with an outcome-reward variant and a two-round variant; student rollouts count toward the budget. A4 OPM: eight trainers, drift measured on about one rollout in eight, a per-token ceiling, non-blocking rendezvous, the guarded merge (finite-difference logit step 0.01, three CG iterations from the sum, γ\gamma on held-out true KL), rollouts in flight kept with per-token importance weights.

Metrics. Per benchmark, the score normalised from base (0) to that domain’s specialist (1); the mean and the worst domain are primary, with 95% confidence intervals over three seeds and several samples per task. Cost from one shared ledger: GPU-hours including teacher scoring and merges, wall-clock, rollouts per domain. Secondary: the held-out suite, a cross-domain matrix (each arm and specialist on every domain), response length, entropy, thinking leakage, tool-call validity, and OPM’s per-rendezvous diagnostics.

Placeholder · Table F1 · pending main runs

Secondary benchmarks per domain (SWE-bench Pro, HMMT Feb 26, IFEval, BFCL v4, τ³-Bench) and the held-out suite (MMLU-Pro, GPQA, HLE) for base, specialists and A1–A4, 3 seeds.

Placeholder · Table F2 · pending main runs

Ablations at reduced length: sum and average in place of the OPM merge; fixed interval in place of the KL trigger; stale against fresh metric; Fisher sample size; OPM with a reference-KL penalty; the reward-aware merge.

G. Secondary findings in detail

Orthogonal is not independent. At 4B, global cosines between task vectors are at most 0.022 and mostly below 0.002, yet the off-diagonal interference index has median 0.122 and maximum 1.78 (Appendix D). Directions along the specialists’ updates carry 5,800 times the Fisher curvature of random directions of the same norm. Whether two domains interfere is a property of their Fishers’ overlap, which cosines do not measure; this is the coupling table of B.2.

Sparsity is a bf16 artefact. On layer-16 tensors of the 4B specialists, task vectors have ∥τ∥/∥θ∥≈7×10−4\|\tau\|/\|\theta\|\approx7\times10^{-4} per layer and a median entry of about 1.1×10−51.1\times10^{-5}, about 2−112^{-11} relative to the weights, exactly the rounding step of TF32. In fp32, 99.996% of entries are nonzero. After bf16 rounding only 18% are, the rounded task vector has cosine 0.57 with the true one, and its relative L2 error is 84%. The layer-16 cosines of Wu et al. reproduce exactly (math–science 0.0217, math–code 0.0103), so near-orthogonality is real; apparent sparsity is not. Building the merge from bf16 checkpoints instead of fp32 raises the median off-diagonal interference from 0.122 to 0.14 and the maximum from 1.78 to 2.51.

The bf16 floor. The KL between an fp32 specialist and its own bf16 cast, as a fraction of its drift from the base: math 0.276, science 0.248, code 0.314, IF 0.247, agent 0.403. Deploying the OPM merge in fp32 adds 1.30 ± 0.34 points, mostly on AIME, GPQA and LiveCodeBench. Whether fp32 deployment favours the OPM merge specifically is open, because the sum was not evaluated in fp32. Merges, residuals and triggers must be computed in true fp32; perturbations at the 2−112^{-11} level are invisible to TF32.

KL residual is not reward loss. The sum has the largest residual (1.19 of drift in-sample, up to 3.06 on code) and the best one-shot mean. By Lemma 2 the return change is the reward-weighted log-ratio plus at most the KL; to first order ρσ2ε\rho\sigma\sqrt{2\varepsilon}, with the reward–log-ratio correlation ρ\rho estimated at 0.1–0.3 here. Residual bounds how much of a specialist’s own behaviour is lost, which is the right objective when domains conflict and the wrong one when they help each other.

When the sum beats OPM. In the regime map of B.8, aligned overlap with Cj>0C_j>0 favours the sum. One-shot at 4B the reasoning domains are there, IF and agent are in the low-visibility regime where every merge ties, and the sum is the best one-shot merge. The same release also shows the conflicting regime outside the merge: trained alone, the IF specialist lowers BFCL from 63.9 to 54.1, and the agent and code specialists lose about 2 points on AIME25 . Which regime each pair of the eight main domains is in is what Table 6 and Figure 4 measure.

The stale metric ties at one shot. A Fisher from base-model rollouts gives a worse residual (0.38 vs 0.31 held-out) but the same benchmark mean (−0.22 ± 0.31). In the toy a metric that was never refreshed failed badly (0.13 against 0.96) because it was used across many rounds. One shot cannot tell the two apart; the main runs log both.

H. Extended related work

The objective and the loop. Distral shares a distilled policy across tasks and regularises each task policy toward it by KL . Divide-and-Conquer RL trains one policy per context, periodically distils them into a central policy on their own trajectories, and resets them to it . That is objective (1) solved by gradient descent at a fixed interval; the barycenter objective is theirs. Policy distillation compresses several task policies into one network . OPM replaces distillation by its closed form in one shared network, adds the KL trigger and the coupling theory. Co-Evolving Policy Distillation runs per-domain GRPO branches that distil into each other during training and merges them at the end . WARP runs parallel RLHF policies with a moving EMA anchor, merges them by spherical interpolation and re-initialises, for one task and on a fixed schedule .

The merge rule. Fisher-weighted averaging , RegMean’s layer-wise Gram solve and MaTS, which solves the same system with conjugate gradient , are applied once to finished supervised models with diagonal, layer or empirical Fishers, for which the gradient need not lie in the Fisher’s range. FedFish derives a function-space average and applies it each round in supervised federated learning . Liu et al. show that MOPD is Fisher merging to second order; B.7 adds the lossless statement and which Fisher. M2A merges a math task vector into a software agent only along the null space of agent-critical activations , which is the intuition of Theorem 1 applied once.

Combining RL domains. Controlled comparisons of mixing, merging and distillation at small scale find them within a few points of each other, with the ordering depending on budget and benchmark ,. Cascade RL trains domains in sequence ,. Frontier reports train specialists and consolidate by on-policy distillation ,,, building on on-policy distillation of language models ,; MiMo-V2.6 runs mixed RL followed by multi-teacher distillation .

Federated and local optimisation. Periodic averaging of local trainers is federated averaging and DiLoCo ,; its bias under heterogeneous clients is client drift, addressed by proximal terms, control variates and normalised averaging ,,. In lossless rounds OPM has no drift bias, and otherwise a floor proportional to Mκc2ηKM\kappa_c^2\eta K (B.4). Federated natural policy gradient for multi-task RL aggregates natural gradients for a common objective ; the tabular reduction in B.3 shows why the multi-task Fisher, not an average of directions, is needed when domains visit different states. Federated GRPO with plain averaging fell below local-only training under non-IID data . FedKL penalises KL to the global policy . Synchronising when models drift apart rather than on a clock has been used with parameter distance in supervised decentralised training ; we know of no use of trajectory KL to trigger a weight merge.

Policy optimisation. The natural policy gradient and trust-region policy optimisation supply the geometry; the return bound of B.6 is the standard trust-region argument applied to a merge residual. The limitations of the empirical Fisher are why the metric has to be computed on the trainer’s own samples.