ℒ_RL[1]hatched band: not updated in this step · solid band: what this stage trainsKimi K3’s report, of this step: partial rollouts, extended · over its token budget a trajectory’s reward is −1 · “forward-only non-policy models” in loss computation

RL step

Objective

no parameters

For each token, the clipped ratio times its response’s advantage, less $\tau$ times the squared log-ratio; the mean is over the $N$ generated tokens, “in a batch” in the report’s words.

\mathcal L_{\mathrm{RL}}(\theta)=\mathbb E_{x\sim\mathcal D}\Big[\frac1N\sum_{j=1}^{{\htmlClass{vl-stream-k}{K}}}\sum_{i=1}^{|y_j|}\Big(c_j^i\,{\htmlData{light=advantage advantage-op}{A_j}}-\tau\,D_j^i\Big)\Big],\qquad N=\sum_{j=1}^{{\htmlClass{vl-stream-k}{K}}}|y_j|

in: $c$ [B batch, K, T positions] $A$ [B batch, K] $D$ [B batch, K, T positions]
out: $\mathcal L_{\mathrm{RL}}$ [1]

The parentheses round the summand are this plate’s. The report says the optimizer is employed “to minimize this objective”; as printed, the expression rewards responses above the mean when it is maximized. The report does not resolve this.

Sources1

Direct evidence

Kimi K2.5 report §4.4.2, Policy Optimization: for each problem x, K responses are generated using the previous policy π_old; the objective (eq. 1) averages over the N generated tokens of the batch the token-level ratio π_θ/π_old clipped to [α, β], times the reward less the mean reward of the K responses, less τ times the squared log-ratio. The clipping is “a simple gradient masking scheme”: gradients are computed normally for tokens with log-ratios within [α, β] and zeroed outside it, “regardless of the sign of the advantages”. The MuonClip optimizer minimizes this objective.

k25 · arXiv 2602.02276 / KimiK2.5-2602.02276v2.pdf
Pinned revision v2 ↗ · p. 8