Sparse training
×Stage · DeepSeek-V3.2-Exp
Every parameter trains with sparse attention: each query keeps its top 2,048 positions.
Architecture. Sparse attention from here on.
- Data
- The same 128K extension distribution
- Trainable
- All parameters; the indexer’s input detached, so the indexer learns only from its KL loss and the model only from the language-modelling loss
- Objective
- Language modelling; the indexer’s KL over the selected positions (eq. 4)
- Selection
- Top 2,048 key-value positions per query
- Learning rate
- 7.3×10⁻⁶
- Steps, batch
- 15,000 steps of 480 sequences of 128K tokens: 943.7B tokens
- Tokens
- 15,000 × 480 × 131,072 ≈ 943.7BderivedSteps × sequences × sequence length.
Not disclosed Optimiser, schedule shape
Sources2
- DeepSeek-V3.2 report, p. 2Data
- DeepSeek-V3.2 report, p. 3Trainable, Objective, Selection, Learning rate, Steps, batch, Tokens
Key
- stage
- tokens, log scale
- a total no source states, computed from steps × batch × length
- checkpoint
- checkpoint, weights published
- a join the sources leave unexplained
- stages of a released model, folded into one row
- derived
- computed from reported values; the arithmetic is shown
- not disclosed
- no pinned source states it; nothing is filled in