Training

  1. Start of training
  2. Pre-training, 8KKimi K3not stated8KNTPtext and visionMTPallMTP layer
  3. Pre-training, 64KKimi K3not stated64KNTPtext and visionMTPall
  4. Cooldown, 256KKimi K3not stated256KNTPall
  5. Cooldown, 1MKimi K3not stated1MNTPall
  6. Off this path: Draft model1 stage
  7. The frozen target and the time: not stated. Speculative decoding is used in RL rollouts
  8. Draft-model fine-tuningKimi K3not statednot statedLKdraftMTP → draft
  9. Draft model
  10. SFTKimi K3not statednot statedSFTnot statedMXFP4 experts
  11. Cold-start model
  12. RL: nine expertsKimi K3not stated1MRLKimi K2.5’s, near GRPOnot stated
  13. Nine experts
  14. Where the student starts: not stated
  15. Multi-teacher on-policy distillationKimi K3not statednot statedMOPD9 teachersnot stated
  16. Kimi K3
  • tokens, log scale
  • derived: computed from stated numbers
  • context length
  • all parameters train
  • a part trains
  • all but a part trains
  • NTPdashed: concluded, no source names it
  • not stated by the sources
  • the architecture changes
  • checkpoint
  • checkpoint, weights published
  • a join the sources leave unexplained

Start of training

×

Checkpoint

Language and vision are “jointly optimized from the start of training”; the vision encoder is trained “entirely from scratch”. The report names no earlier checkpoint the model starts from.

Weights
no pinned source shows them published
Sources2
Key
stage
tokens, log scale
a total no source states, computed from steps × batch × length
checkpoint
checkpoint, weights published
a join the sources leave unexplained
stages of a released model, folded into one row
derived
computed from reported values; the arithmetic is shown
not disclosed
no pinned source states it; nothing is filled in

Kimi K3

All models, with their blocks →

Size
2.8T
Published
Moonshot AI · Release date not pinned
Line
New parts in the block · after Kimi K2
View
Source

Kimi K3 technical report @ 3cb39df · no training code is publishedHow to read