Training

  1. Start of training
  2. Pre-training, 8KKimi K3not stated8KNTPtext and visionMTPallMTP layer
  3. Pre-training, 64KKimi K3not stated64KNTPtext and visionMTPall
  4. Cooldown, 256KKimi K3not stated256KNTPall
  5. Cooldown, 1MKimi K3not stated1MNTPall
  6. Off this path: Draft model1 stage
  7. The frozen target and the time: not stated. Speculative decoding is used in RL rollouts
  8. Draft-model fine-tuningKimi K3not statednot statedLKdraftMTP → draft
  9. Draft model
  10. SFTKimi K3not statednot statedSFTnot statedMXFP4 experts
  11. Cold-start model
  12. RL: nine expertsKimi K3not stated1MRLKimi K2.5’s, near GRPOnot stated
  13. Nine experts
  14. Where the student starts: not stated
  15. Multi-teacher on-policy distillationKimi K3not statednot statedMOPD9 teachersnot stated
  16. Kimi K3
  • tokens, log scale
  • derived: computed from stated numbers
  • context length
  • all parameters train
  • a part trains
  • all but a part trains
  • NTPdashed: concluded, no source names it
  • not stated by the sources
  • the architecture changes
  • checkpoint
  • checkpoint, weights published
  • a join the sources leave unexplained

Where the student starts: not stated

×

Join the sources leave unexplained

The report does not say which checkpoint the distilled model starts from.

Sources1
Key
stage
tokens, log scale
a total no source states, computed from steps × batch × length
checkpoint
checkpoint, weights published
a join the sources leave unexplained
stages of a released model, folded into one row
derived
computed from reported values; the arithmetic is shown
not disclosed
no pinned source states it; nothing is filled in

Kimi K3

All models, with their blocks →

Size
2.8T
Published
Moonshot AI · Release date not pinned
Line
New parts in the block · after Kimi K2
View
Source

Kimi K3 technical report @ 3cb39df · no training code is publishedHow to read