Pre-training, 64K
×Stage · Kimi K3
The same pre-training at a context of 64K, “in a subsequent training phase”.
- Context
- 64K
- Objective
- next-token prediction over text and visual tokens
- Learning rate
- Cosine decay with a 1% linear warm-up
Derived
Trained. The report says language and vision are jointly optimized from the start and that the MTP layer is pre-trained, and reports no part as held fixed: that every parameter trains is concluded from that. The report lists no trainable set.
Not disclosed Tokens · Peak and minimum learning rate · Batch · Steps · MTP weight and depth · Number format · Optimizer for non-matrix parameters, the clipping’s rule, momentum · Where the phase begins
Architecture at this stageThe model with its MTP layer beside it. Quantization-aware training starts later, at SFT.
All settings8
- Objective
- One next-token prediction over interleaved visual and text tokens; language and vision trained jointly from the start
- Multi-token prediction
- One MTP layer with the structure of a backbone block is pre-trained with the model
- Optimizer
- Per-Head Muon for matrix parameters: the momentum of the attention’s Q, K and V projections is split by head and each head’s block is orthogonalized separately; weight clipping as in Kimi K2
- Weight decay
- 0.1 throughout
- Learning rate
- Cosine decay with a 1% linear warm-up
- Load balancing
- Auxiliary-loss-free, by Quantile Balancing: each expert’s bias is set from a quantile of its routing margins
- Trainable
- Language and vision jointly: the vision encoder, MoonViT-V2, trains from scratch with the model
- Context
- Extended to 64K in a subsequent training phase
Sources6
- k3_tech_report.pdf, p. 11Objective, Multi-token prediction, Optimizer, Weight decay, Learning rate, Load balancing, Trainable
- k3_tech_report.pdf, p. 14Multi-token prediction
- k3_tech_report.pdf, p. 10Optimizer
- k3_tech_report.pdf, pp. 8–9Load balancing
- k3_tech_report.pdf, p. 9Trainable
- k3_tech_report.pdf, p. 12Context