Dense warm-up
×Stage · DeepSeek-V3.2-Exp
Only the lightning indexer trains, to imitate the dense attention it will later restrict.
Architecture. The lightning indexer exists from here on; attention stays dense.
- Data
- The distribution of the 128K extension data used for V3.1-Terminus
- Trainable
- The lightning indexer only; every other parameter frozen
- Objective
- KL divergence from the main attention, summed over heads and L1-normalised, to the indexer’s softmax (eq. 3)
- Learning rate
- 10⁻³
- Steps, batch
- 1,000 steps of 16 sequences of 128K tokens: 2.1B tokens
- Tokens
- 1,000 × 16 × 131,072 ≈ 2.1BderivedSteps × sequences × sequence length.
Not disclosed Optimiser
Sources3
- DeepSeek-V3.2 report, p. 2Data, Trainable
- DeepSeek-V3.2 report, pp. 2–3Objective
- DeepSeek-V3.2 report, p. 3Learning rate, Steps, batch, Tokens
Key
- stage
- tokens, log scale
- a total no source states, computed from steps × batch × length
- checkpoint
- checkpoint, weights published
- a join the sources leave unexplained
- stages of a released model, folded into one row
- derived
- computed from reported values; the arithmetic is shown
- not disclosed
- no pinned source states it; nothing is filled in