RL step
Previous policy
×frozen · not trained
The policy before this update writes responses to the problem.
- in
- [B batch]
- out
- [B batch, K, T positions]
A previous iterate of the same model: not updated in this step. In Kimi K3 generation pauses once a fraction of the rollouts completes, and a long trajectory can span several iterations: its tokens may come from more than one earlier policy.
Sources2
Direct evidence
Kimi K2.5 report §4.4.2, Policy Optimization: for each problem x, K responses are generated using the previous policy π_old; the objective (eq. 1) averages over the N generated tokens of the batch the token-level ratio π_θ/π_old clipped to [α, β], times the reward less the mean reward of the K responses, less τ times the squared log-ratio. The clipping is “a simple gradient masking scheme”: gradients are computed normally for tokens with log-ratios within [α, β] and zeroed outside it, “regardless of the sign of the advantages”. The MuonClip optimizer minimizes this objective.
k25 · arXiv 2602.02276 / KimiK2.5-2602.02276v2.pdf
Pinned revision v2 ↗ · p. 8
Direct evidence
§4.1.2: K completions are sampled for each of N prompts, and generation pauses once a fraction λ of the trajectories completes (partial rollouts); policy optimization “follows the algorithm in Kimi K2.5”, with a per-token regularization that tolerates stale data. A trajectory whose tokens exceed τ·b₀(x) has its task reward overridden with −1. Tasks that cannot be verified take an agentic generative reward model with a tournament of binary comparisons.
k3 · MoonshotAI/Kimi-K3 / k3_tech_report.pdf
Pinned revision 3cb39dfd32e5 ↗ · p. 13