Training

  1. Random initialisation
  2. Pre-trainingDeepSeek-V314.8T
  3. Context 32KDeepSeek-V3≈62.9B
  4. Context 128KDeepSeek-V3≈62.9B
  5. DeepSeek-V3-Base
  6. SFT → RLDeepSeek-V3tokens not disclosed
  7. DeepSeek-V3
  8. From V3’s base: before or after its context extension, not stated
  9. Context 32KDeepSeek-V3.1630B
  10. Context 128KDeepSeek-V3.1209B
  11. DeepSeek-V3.1-Base
  12. Post-trainingDeepSeek-V3.1tokens not disclosed
  13. DeepSeek-V3.1
  14. From V3.1 or from a base checkpoint: not stated
  15. UpdateDeepSeek-V3.1-Terminustokens not disclosed
  16. DeepSeek-V3.1-Terminus
  17. V3.1-Base or a later checkpoint: not stated
  18. V3.1-Terminus base checkpoint
  19. Dense warm-upDeepSeek-V3.2-Exp2.1B
  20. Sparse trainingDeepSeek-V3.2-Exp943.7B
  21. DeepSeek-V3.2 base
  22. Specialist distillationDeepSeek-V3.2-Exptokens not disclosed
  23. Mixed RLDeepSeek-V3.2-Exptokens not disclosed
  24. DeepSeek-V3.2-Exp

Context 32K

×

Stage · DeepSeek-V3.1

The 32K extension again, ten times longer, on more long documents.

Start
“The original V3 base checkpoint”
Method
“Following the methodology outlined in the original DeepSeek-V3 report”, with additional long documents
Tokens
630B, “increased 10-fold”
Against V3
10 × 62.9B ≈ 629BderivedThe V3 report’s 32K phase, ten times over.
Number format
Weights and activations in FP8 with UE8M0 scalesThe card says DeepSeek-V3.1 is trained this way and names no stage.

Not disclosed Steps, batch, learning rate

Sources3
Key
stage
tokens, log scale
a total no source states, computed from steps × batch × length
checkpoint
checkpoint, weights published
a join the sources leave unexplained
stages of a released model, folded into one row
derived
computed from reported values; the arithmetic is shown
not disclosed
no pinned source states it; nothing is filled in

DeepSeek-V3.2-Exp

All models, with their blocks →

Size
671B
Published
DeepSeek · Sep 2025
Line
Latent attention and experts
View