Cooldown, 256K
×Stage · Kimi K3
The third stage of the context curriculum: the window grows to 256K.
- Context
- 256K
- Data
- long documents and videos; synthetic long-context tasks
- Share of training
- The long-sequence computation is concentrated “within a small fraction of the overall training budget”
Derived
Objective. The report describes the cooldown as the last two stages of one four-stage curriculum and names no objective for it: next-token prediction is concluded from that, and the report does not state it.
Trained. The report names no part of the model held fixed in the cooldown: that every parameter trains is concluded, and the report does not state it.
Not disclosed Tokens · Steps · Batch · Learning rate · Where each window begins · Number format
Why
The long-sequence computation is concentrated in a small fraction of the training budget, which keeps the curriculum economical while the model adapts gradually to longer dependencies. Kimi K3 report §3.4, p. 12
Architecture at this stageThe model has no positional embedding: nothing in it changes when the context grows.
All settings5
- Context
- 256K, the first window of the cooldown phase
- Curriculum
- Four stages: the window grows from 8K to 64K during pre-training, and from 256K to 1M during the cooldown phase
- Positions
- None to change: the model has no positional embedding, so the context is extended with no modification of a positional encoding
- Data
- Long documents and videos, cleaned and upsampled; synthetic long-context data whose tasks need information from across the full 1M context
- Share of training
- The long-sequence computation is concentrated “within a small fraction of the overall training budget”
Sources1
- k3_tech_report.pdf, p. 12Context, Curriculum, Positions, Data, Share of training