ℒ_RL[1]hatched band: not updated in this step · solid band: what this stage trainsKimi K3’s report, of this step: partial rollouts, extended · over its token budget a trajectory’s reward is −1 · “forward-only non-policy models” in loss computation

RL step

Policy

learned · not counted here

The model being trained gives each token of each response its probability, against the previous policy’s.

w_j^i=\frac{\pi_\theta(y_j^i\mid {\htmlData{light=prompt}{x}},y_j^{0:i})}{\pi_{\mathrm{old}}(y_j^i\mid {\htmlData{light=prompt}{x}},y_j^{0:i})}

in: $x$ [B batch] $y$ [B batch, K, T positions]
out: $w$ [B batch, K, T positions]

$w_j^i$ is this plate’s name for the ratio at the $i$ -th token of the $j$ -th response. The previous policy’s probability of each token is needed here; the Kimi K2.5 report says its framework records log probabilities for the inference engine’s outputs.

Sources2

Direct evidence

Kimi K2.5 report §4.4.2, Policy Optimization: for each problem x, K responses are generated using the previous policy π_old; the objective (eq. 1) averages over the N generated tokens of the batch the token-level ratio π_θ/π_old clipped to [α, β], times the reward less the mean reward of the K responses, less τ times the squared log-ratio. The clipping is “a simple gradient masking scheme”: gradients are computed normally for tokens with log-ratios within [α, β] and zeroed outside it, “regardless of the sign of the advantages”. The MuonClip optimizer minimizes this objective.

k25 · arXiv 2602.02276 / KimiK2.5-2602.02276v2.pdf
Pinned revision v2 ↗ · p. 8

Direct evidence

Kimi K2.5 report, Appendix D: the RL framework has “fine-grained control to enable features like partial rollout”, and records log probabilities for the inference engine’s outputs.

k25-rollout · arXiv 2602.02276 / KimiK2.5-2602.02276v2.pdf
Pinned revision v2 ↗ · p. 24