RL step

Contextual evidence from Rewards

Kimi K2.5 report §4.4.2, Reward Function: a rule-based outcome reward for tasks with verifiable solutions, a budget-control reward for token efficiency, and generative reward models for general-purpose tasks.

rewards · arXiv 2602.02276 / KimiK2.5-2602.02276v2.pdf
Pinned revision v2 ↗ · p. 8

Contextual evidence from Rewards

§4.1.2: K completions are sampled for each of N prompts, and generation pauses once a fraction λ of the trajectories completes (partial rollouts); policy optimization “follows the algorithm in Kimi K2.5”, with a per-token regularization that tolerates stale data. A trajectory whose tokens exceed τ·b₀(x) has its task reward overridden with −1. Tasks that cannot be verified take an agentic generative reward model with a tournament of binary comparisons.

k3 · MoonshotAI/Kimi-K3 / k3_tech_report.pdf
Pinned revision 3cb39dfd32e5 ↗ · p. 13

Contextual evidence from Subtract the mean

Kimi K2.5 report §4.4.2, Policy Optimization: for each problem x, K responses are generated using the previous policy π_old; the objective (eq. 1) averages over the N generated tokens of the batch the token-level ratio π_θ/π_old clipped to [α, β], times the reward less the mean reward of the K responses, less τ times the squared log-ratio. The clipping is “a simple gradient masking scheme”: gradients are computed normally for tokens with log-ratios within [α, β] and zeroed outside it, “regardless of the sign of the advantages”. The MuonClip optimizer minimizes this objective.

k25 · arXiv 2602.02276 / KimiK2.5-2602.02276v2.pdf
Pinned revision v2 ↗ · p. 8