ℒ_RL[1]hatched band: not updated in this step · solid band: what this stage trainsKimi K3’s report, of this step: partial rollouts, extended · over its token budget a trajectory’s reward is −1 · “forward-only non-policy models” in loss computation

RL step

Subtract the mean

no parameters

A response is measured against the mean reward of the $K$ responses to its problem. There is no division by their deviation.

{\htmlData{light=advantage advantage-op}{A_j}}={\htmlData{light=reward rewards}{r}}(x,y_j)-\bar r(x),\qquad \bar r(x)=\frac1{\htmlClass{vl-stream-k}{K}}\sum_{j=1}^{{\htmlClass{vl-stream-k}{K}}}{\htmlData{light=reward rewards}{r}}(x,y_j)

in: $r$ [B batch, K]
out: $A$ [B batch, K]

$A_j$ is this plate’s name; the report writes $r(x,y_j)-\bar r(x)$ .

Sources1

Direct evidence

Kimi K2.5 report §4.4.2, Policy Optimization: for each problem x, K responses are generated using the previous policy π_old; the objective (eq. 1) averages over the N generated tokens of the batch the token-level ratio π_θ/π_old clipped to [α, β], times the reward less the mean reward of the K responses, less τ times the squared log-ratio. The clipping is “a simple gradient masking scheme”: gradients are computed normally for tokens with log-ratios within [α, β] and zeroed outside it, “regardless of the sign of the advantages”. The MuonClip optimizer minimizes this objective.

k25 · arXiv 2602.02276 / KimiK2.5-2602.02276v2.pdf
Pinned revision v2 ↗ · p. 8