ℒ_RL[1]hatched band: not updated in this step · solid band: what this stage trainsKimi K3’s report, of this step: partial rollouts, extended · over its token budget a trajectory’s reward is −1 · “forward-only non-policy models” in loss computation

RL step

Rewards

no parameters

Each response is scored. Kimi K2.5: a rule-based outcome reward for tasks with verifiable solutions, generative reward models for general-purpose tasks, a budget-control reward, and rewards of their own for visual tasks. Kimi K3: the task reward is overridden with −1 for a trajectory over $\tau\cdot b_0(x)$ tokens, and non-verifiable general tasks take an agentic generative reward model.

\text{rules}\ \cdot\ \text{generative RM}\ \cdot\ \text{budget}

in: $y$ [B batch, K, T positions]
out: $r$ [B batch, K]

Sources2

Direct evidence

Kimi K2.5 report §4.4.2, Reward Function: a rule-based outcome reward for tasks with verifiable solutions, a budget-control reward for token efficiency, and generative reward models for general-purpose tasks.

rewards · arXiv 2602.02276 / KimiK2.5-2602.02276v2.pdf
Pinned revision v2 ↗ · p. 8

Direct evidence

§4.1.2: K completions are sampled for each of N prompts, and generation pauses once a fraction λ of the trajectories completes (partial rollouts); policy optimization “follows the algorithm in Kimi K2.5”, with a per-token regularization that tolerates stale data. A trajectory whose tokens exceed τ·b₀(x) has its task reward overridden with −1. Tasks that cannot be verified take an agentic generative reward model with a tournament of binary comparisons.

k3 · MoonshotAI/Kimi-K3 / k3_tech_report.pdf
Pinned revision 3cb39dfd32e5 ↗ · p. 13