Training

  1. Start of training
  2. Pre-training, 8KKimi K3not stated8KNTPtext and visionMTPallMTP layer
  3. Pre-training, 64KKimi K3not stated64KNTPtext and visionMTPall
  4. Cooldown, 256KKimi K3not stated256KNTPall
  5. Cooldown, 1MKimi K3not stated1MNTPall
  6. Off this path: Draft model1 stage
  7. The frozen target and the time: not stated. Speculative decoding is used in RL rollouts
  8. Draft-model fine-tuningKimi K3not statednot statedLKdraftMTP → draft
  9. Draft model
  10. SFTKimi K3not statednot statedSFTnot statedMXFP4 experts
  11. Cold-start model
  12. RL: nine expertsKimi K3not stated1MRLKimi K2.5’s, near GRPOnot stated
  13. Nine experts
  14. Where the student starts: not stated
  15. Multi-teacher on-policy distillationKimi K3not statednot statedMOPD9 teachersnot stated
  16. Kimi K3
  • tokens, log scale
  • derived: computed from stated numbers
  • context length
  • all parameters train
  • a part trains
  • all but a part trains
  • NTPdashed: concluded, no source names it
  • not stated by the sources
  • the architecture changes
  • checkpoint
  • checkpoint, weights published
  • a join the sources leave unexplained

Multi-teacher on-policy distillation

×

Stage · Kimi K3

The nine experts are consolidated into one model: for a domain and a sampled effort, the expert of that pair scores the student’s own tokens.

Teachers
the expert of the domain and the sampled effort
Reward
clipped log-ratio of teacher to student, per token
How it trains
The reward enters the same RL framework, partial rollouts included

Not disclosed R_max · Where the student starts · Steps · Learning rate · Tokens

Why

Distillation consolidates the experts’ capabilities at every reasoning effort into a single model. Kimi K3 report §4.1.3, p. 13

The dense per-token reward fits the same RL framework, partial rollouts included. Kimi K3 report §4.1.3, p. 14

The training step →What one step of this stage computes, as a plate.

Architecture at this stageOne student model; the nine experts are its teachers.

All settings3
Teachers
For a domain d and a sampled effort e ∈ {low, high, max}, the expert of that pair among the nine
Reward
Per token: the clipped, stop-gradient log-ratio of the teacher’s probability to the student’s, within ±R_max (eq. 15)
How it trains
The reward enters the same RL framework, partial rollouts included
Sources3

Kimi K3

All models, with their blocks →

Size
2.8T
Published
Moonshot AI · Release date not pinned
Line
New parts in the block · after Kimi K2
View
Source

Kimi K3 technical report @ 3cb39df · no training code is publishedHow to read