Kimi K3 · training steps · RL step

out: $\mathcal L_{\mathrm{RL}}$ [1]

Explanation

The Kimi K3 report says its policy optimization “follows the algorithm in Kimi K2.5” and does not write it. The equation here is the Kimi K2.5 report’s (eq. 1). The comparison with GRPO, as drawn for DeepSeek-V3.2-Exp, is this atlas’s: the Kimi K2.5 report compares its loss with the algorithm of Kimi K1.5 and with PPO’s clipping, and neither Kimi report names GRPO.

	GRPO (DeepSeek-R1, eq. 1 to 3)	Kimi K2.5, eq. 1
Advantage	the reward less the group’s mean, over the group’s standard deviation	the reward less the mean; no division
Clip	the smaller of the ratio times the advantage and its clipped form	the ratio clipped to $[\alpha,\beta]$ token by token: a mask on the gradient by the log-ratio, whatever the advantage’s sign
Holding term	$\beta$ times a KL term to a reference model	$\tau$ times the squared log-ratio to the old policy
Average	over the group’s outputs	over the generated tokens of the batch
Ratio	of a whole output, $\pi_\theta(o_i\mid q)$	of each token after its prefix
Direction	“maximizing the following objective”	“to minimize this objective”; see below

The Kimi K3 report says three things of this step. It extends the partial rollouts of the earlier framework, which Kimi K2.5 already had. It overrides the task reward with −1 for a trajectory over $\tau\cdot b_0(x)$ tokens, $\tau$ there a budget multiplier; Kimi K2.5 also had a budget, with another rule (Toggle). And in its infrastructure section it says “RL loss computation often requires forward-only non-policy models, such as reference models”; it does not say which models, or which term they enter. The report does not say whether the previous policy of eq. 1 is one of them.

Eq. 1 of the Kimi K2.5 report is the expression drawn here; the names $A$ , $w$ , $c$ , $D$ and the parentheses round the summand are this plate’s. The report calls it an objective and says the MuonClip optimizer is employed “to minimize this objective”. As printed, it rewards a higher probability for responses above the mean, which is what a maximized objective does. The report does not resolve this; the cards describe the terms as they act when the expression is maximized.

$B$ , $K$ and $T$ are this plate’s: $B$ problems, $K$ responses to each, $T$ tokens of a response (the report’s $|y_j|$ , which varies). $K$ , $\alpha$ , $\beta$ , the coefficient $\tau$ of eq. 1 and the learning rate are not disclosed for Kimi K3.

Sources7

Direct evidence

Kimi K2.5 report §4.4.2, Policy Optimization: for each problem x, K responses are generated using the previous policy π_old; the objective (eq. 1) averages over the N generated tokens of the batch the token-level ratio π_θ/π_old clipped to [α, β], times the reward less the mean reward of the K responses, less τ times the squared log-ratio. The clipping is “a simple gradient masking scheme”: gradients are computed normally for tokens with log-ratios within [α, β] and zeroed outside it, “regardless of the sign of the advantages”. The MuonClip optimizer minimizes this objective.

k25 · arXiv 2602.02276 / KimiK2.5-2602.02276v2.pdf
Pinned revision v2 ↗ · p. 8

Direct evidence

§4.1.2: K completions are sampled for each of N prompts, and generation pauses once a fraction λ of the trajectories completes (partial rollouts); policy optimization “follows the algorithm in Kimi K2.5”, with a per-token regularization that tolerates stale data. A trajectory whose tokens exceed τ·b₀(x) has its task reward overridden with −1. Tasks that cannot be verified take an agentic generative reward model with a tournament of binary comparisons.

k3 · MoonshotAI/Kimi-K3 / k3_tech_report.pdf
Pinned revision 3cb39dfd32e5 ↗ · p. 13

Direct evidence

§5.3.1: “RL loss computation often requires forward-only non-policy models, such as reference models”.

loss · MoonshotAI/Kimi-K3 / k3_tech_report.pdf
Pinned revision 3cb39dfd32e5 ↗ · p. 22

Direct evidence

Kimi K2.5 report §4.4.2, Reward Function: a rule-based outcome reward for tasks with verifiable solutions, a budget-control reward for token efficiency, and generative reward models for general-purpose tasks.

rewards · arXiv 2602.02276 / KimiK2.5-2602.02276v2.pdf
Pinned revision v2 ↗ · p. 8

Direct evidence

Kimi K2.5 report §4.4.2, Token Efficient Reinforcement Learning: Toggle alternates phases; in the budget-limited phase a response longer than a problem-dependent budget has its reward zeroed, when the problem’s mean reward is above a threshold (eq. 2).

toggle · arXiv 2602.02276 / KimiK2.5-2602.02276v2.pdf
Pinned revision v2 ↗ · p. 9

Direct evidence

Kimi K2.5 report, Appendix D: the RL framework has “fine-grained control to enable features like partial rollout”, and records log probabilities for the inference engine’s outputs.

k25-rollout · arXiv 2602.02276 / KimiK2.5-2602.02276v2.pdf
Pinned revision v2 ↗ · p. 24

Direct evidence

DeepSeek-R1 paper §2.2.1, eq. 1 to 3: GRPO, as drawn on DeepSeek-V3.2-Exp’s plate. The advantage is the reward less the group’s mean over its standard deviation; the objective takes the smaller of the ratio times the advantage and its clipped form, less β times a KL term to a reference, averaged over the group.

grpo · arXiv 2501.12948 / DeepSeek-R1-2501.12948v1.pdf
Pinned revision v1 ↗ · p. 5