Context 32K
×Stage · DeepSeek-V3
YaRN extends the context from 4K to 32K in 1,000 steps.
Architecture. YaRN rescales the rotary key from here on (scale 40).
- Method
- YaRN on the decoupled shared key only: s = 40, α = 1, β = 32, scaling factor √t = 0.1 ln s + 1
- Steps, sequence, batch
- 1,000 steps; 32K tokens; batch 1,920
- Learning rate
- 7.3×10⁻⁶, the pre-training’s final rate
- Tokens
- 1,000 × 1,920 × 32,768 ≈ 62.9BderivedSteps × sequences × sequence length.
- Compute
- 119K H800 GPU hours with the 128K phase
Sources3
- DeepSeek-V3 report, p. 23Method, Steps, sequence, batch, Learning rate, Tokens
- inference/model.py, lines 535–537Method
- DeepSeek-V3 report, p. 5Compute
Key
- stage
- tokens, log scale
- a total no source states, computed from steps × batch × length
- checkpoint
- checkpoint, weights published
- a join the sources leave unexplained
- stages of a released model, folded into one row
- derived
- computed from reported values; the arithmetic is shown
- not disclosed
- no pinned source states it; nothing is filled in