Cooldown, 1M
×Stage · Kimi K3
The last stage of the context curriculum: the window reaches 1M tokens.
- Context
- 1M
- Data
- long documents and videos; synthetic long-context tasks
- Share of training
- The long-sequence computation is concentrated “within a small fraction of the overall training budget”
Derived
Objective. The report describes the cooldown as the last two stages of one four-stage curriculum and names no objective for it: next-token prediction is concluded from that, and the report does not state it.
Trained. The report names no part of the model held fixed in the cooldown: that every parameter trains is concluded, and the report does not state it.
Not disclosed Tokens · Steps · Batch · Learning rate · Where each window begins · Number format
Why
Length alone does not give long-range capability: the synthetic data embeds tasks that can be solved only with information scattered across the full 1M context, so attention is trained at that scale. Kimi K3 report §3.4, p. 12
Architecture at this stageThe model has no positional embedding: nothing in it changes when the context grows.
All settings5
- Context
- 1M, the last window of the cooldown phase
- Curriculum
- Four stages: the window grows from 8K to 64K during pre-training, and from 256K to 1M during the cooldown phase
- Positions
- None to change: the model has no positional embedding, so the context is extended with no modification of a positional encoding
- Data
- Long documents and videos, cleaned and upsampled; synthetic long-context data whose tasks need information from across the full 1M context
- Share of training
- The long-sequence computation is concentrated “within a small fraction of the overall training budget”
Sources1
- k3_tech_report.pdf, p. 12Context, Curriculum, Positions, Data, Share of training