Training

  1. Start of training
  2. Pre-training, 8KKimi K3not stated8KNTPtext and visionMTPallMTP layer
  3. Pre-training, 64KKimi K3not stated64KNTPtext and visionMTPall
  4. Cooldown, 256KKimi K3not stated256KNTPall
  5. Cooldown, 1MKimi K3not stated1MNTPall
  7. SFTKimi K3not statednot statedSFTnot statedMXFP4 experts
  8. Cold-start model
  9. RL: nine expertsKimi K3not stated1MRLKimi K2.5’s, near GRPOnot stated
  10. Nine experts
  11. Where the student starts: not stated
  12. Multi-teacher on-policy distillationKimi K3not statednot statedMOPD9 teachersnot stated
  13. Kimi K3
  • tokens, log scale
  • derived: computed from stated numbers
  • context length
  • all parameters train
  • a part trains
  • all but a part trains
  • NTPdashed: concluded, no source names it
  • not stated by the sources
  • the architecture changes
  • checkpoint
  • checkpoint, weights published
  • a join the sources leave unexplained

Where the student starts: not stated

×

Join the sources leave unexplained

The report does not say which checkpoint the distilled model starts from.

Sources1
Key
stage
tokens, log scale
a total no source states, computed from steps × batch × length
checkpoint
checkpoint, weights published
a join the sources leave unexplained
stages of a released model, folded into one row
derived
computed from reported values; the arithmetic is shown
not disclosed
no pinned source states it; nothing is filled in

Kimi K3

All models, with their blocks →

Size
2.8T
Published
Moonshot AI · Release date not pinned
Line
New parts in the block · after Kimi K2
View
Source

Kimi K3 technical report @ 3cb39df · no training code is publishedHow to read