RL: nine experts
×Stage · Kimi K3
Reinforcement learning trains one expert for each of three domains at each of three reasoning efforts.
- Experts
- 3 domains × 3 efforts: 9 experts
- Order over efforts
- max first, then high and low
- Budget rule
- over budget: reward −1
- Rewards
- environment checks; a generative reward model
- Context
- 1M
Not disclosed K, N, λ, τ, σ · Steps · Learning rate · Tokens
Why
One expert is trained for each broad domain at every reasoning effort, in place of a specialized model for each task. Kimi K3 report §4.1.2, p. 12
Pausing generation once a fraction of the rollouts completes lets optimization proceed without waiting for the slowest trajectories. Kimi K3 report §4.1.2, p. 13
The training step →What one step of this stage computes, as a plate.
Architecture at this stageNine experts, one for each domain and effort; rollout and training share one quantization.
All settings13
- Experts
- 3 domains (general tasks, general agents, coding agents) × 3 reasoning efforts (low, high, max): 9 experts
- Order over efforts
- A max-budget expert first, with a large budget multiplier τ; then τ annealed to smaller values for the high- and low-effort experts
- Budget rule
- A trajectory whose tokens exceed τ·b₀(x), b₀ the budget estimated from the cold-start model, has its task reward replaced by −1
- Rollouts
- K completions for each of N prompts; generation pauses once a fraction λ of the trajectories completes, and paused rollouts resume first in the next iteration
- Algorithm
- Policy optimization “follows the algorithm in Kimi K2.5”, with a per-token regularization; this report does not write it. The Kimi K2.5 report does (eq. 1): the clipped token-level ratio to the old policy times the reward less the group’s mean, less τ times the squared log-ratio
- Against GRPO
- The advantage is the reward less the group’s mean, with no division by its deviation; the ratio to the old policy is clipped to [α, β] token by token, a mask on the gradient whatever the advantage’s sign; the holding term is τ times the squared log-ratio to the old policy, where GRPO has a KL term to a reference; the average is over the batch’s tokens
- Context
- 1M: “each 1M-context Kimi K3 RL experiment”
- Effort in the prompt
- A thinking-effort option message, turned into a short instruction in context; the schema has four levels, of which the model supports a subset
- Harnesses
- Different harness configurations are constructed for different task groups during RL
- Speculative decoding
- Used in rollouts: it “further accelerates request turnover within relatively fixed tool-call intervals”
- Compute
- “Within a few hundred GPUs” for each RL experiment, rollout and training on the same devices
- Rewards
- Verifiable tasks: the environments’ checks, such as a verifier’s evaluation of the final state. Other tasks: an agentic generative reward model, a tournament of binary comparisons scored against a rubric the judge writes
- Number format
- Quantization-aware training: MoE expert weights in MXFP4, activations in MXFP8; the other parts in higher precision
Sources9
- k3_tech_report.pdf, p. 12Experts, Number format
- k3_tech_report.pdf, p. 13Order over efforts, Budget rule, Rollouts, Algorithm, Rewards
- KimiK2.5-2602.02276v2.pdf, p. 8Algorithm, Against GRPO
- k3_tech_report.pdf, p. 21Context, Speculative decoding, Compute
- k3_tech_report.pdf, p. 17Context
- k3_tech_report.pdf, p. 47Effort in the prompt
- k3_tech_report.pdf, pp. 14–15Harnesses
- k3_tech_report.pdf, pp. 14–16Rewards
- k3_tech_report.pdf, p. 14Number format