Pre-training, 8K
×Stage · Kimi K3
Native multimodal pre-training: one next-token prediction over text and visual tokens, at a context of 8K.
- Context
- 8K
- Objective
- next-token prediction over text and visual tokens
- Optimizer
- Per-Head Muon, weight clipping
- Learning rate
- Cosine decay with a 1% linear warm-up
- Weight decay
- 0.1
Derived
Trained. The report says language and vision are jointly optimized from the start and that the MTP layer is pre-trained, and reports no part as held fixed: that every parameter trains is concluded from that. The report lists no trainable set.
Not disclosed Tokens · Peak and minimum learning rate · Batch · Steps · MTP weight and depth · Number format · Optimizer for non-matrix parameters, the clipping’s rule, momentum · Data sizes and proportions
Why
Language and vision train together from the start, so the shared backbone learns one multimodal representation; the report says a contrastively pre-trained vision encoder made joint optimization unstable. Kimi K3 report §2.4, p. 9
Architecture at this stageThe model with its MTP layer beside it. Quantization-aware training starts later, at SFT.
A multi-token prediction layer trains with the model. The pinned checkpoint holds no tensor named for it.
All settings9
- Objective
- One next-token prediction over interleaved visual and text tokens; language and vision trained jointly from the start
- Multi-token prediction
- One MTP layer with the structure of a backbone block is pre-trained with the model
- Optimizer
- Per-Head Muon for matrix parameters: the momentum of the attention’s Q, K and V projections is split by head and each head’s block is orthogonalized separately; weight clipping as in Kimi K2
- Weight decay
- 0.1 throughout
- Learning rate
- Cosine decay with a 1% linear warm-up
- Load balancing
- Auxiliary-loss-free, by Quantile Balancing: each expert’s bias is set from a quantile of its routing margins
- Trainable
- Language and vision jointly: the vision encoder, MoonViT-V2, trains from scratch with the model
- Context
- 8K at the start of pre-training
- Data
- Four text domains (Web Text, Code, Mathematics, Knowledge) and a vision corpus: captions, interleaved image–text documents, OCR, perception, video, visual coding
Sources6
- k3_tech_report.pdf, p. 11Objective, Multi-token prediction, Optimizer, Weight decay, Learning rate, Load balancing, Trainable
- k3_tech_report.pdf, p. 14Multi-token prediction
- k3_tech_report.pdf, p. 10Optimizer, Data
- k3_tech_report.pdf, pp. 8–9Load balancing
- k3_tech_report.pdf, p. 9Trainable
- k3_tech_report.pdf, p. 12Context