RL step
Squared log-ratio
×no parameters
The squared logarithm of the ratio, for each token. Subtracted with the coefficient , it holds the policy near the previous one when the expression is maximized.
- in
- [B batch, K, T positions]
- out
- [B batch, K, T positions]
The Kimi K3 report credits “a per-token regularization” that constrains updates “within a localized neighborhood”; it does not say whether that is this term, the token-level clip, or both. The Kimi K2.5 report gives the clip the role of bounding off-policy drift and says nothing in words about this term. is this plate’s name.
Sources2
Direct evidence
Kimi K2.5 report §4.4.2, Policy Optimization: for each problem x, K responses are generated using the previous policy π_old; the objective (eq. 1) averages over the N generated tokens of the batch the token-level ratio π_θ/π_old clipped to [α, β], times the reward less the mean reward of the K responses, less τ times the squared log-ratio. The clipping is “a simple gradient masking scheme”: gradients are computed normally for tokens with log-ratios within [α, β] and zeroed outside it, “regardless of the sign of the advantages”. The MuonClip optimizer minimizes this objective.
k25 · arXiv 2602.02276 / KimiK2.5-2602.02276v2.pdf
Pinned revision v2 ↗ · p. 8
Direct evidence
§4.1.2: K completions are sampled for each of N prompts, and generation pauses once a fraction λ of the trajectories completes (partial rollouts); policy optimization “follows the algorithm in Kimi K2.5”, with a per-token regularization that tolerates stale data. A trajectory whose tokens exceed τ·b₀(x) has its task reward overridden with −1. Tasks that cannot be verified take an agentic generative reward model with a tournament of binary comparisons.
k3 · MoonshotAI/Kimi-K3 / k3_tech_report.pdf
Pinned revision 3cb39dfd32e5 ↗ · p. 13