Pre-training 0–28T
×Stage · DeepSeek-V4.1-Flash
Pre-training from scratch with sparse attention at a sequence length of 64K and a constant learning rate; the vision encoder’s body is frozen.
- Tokens
- 28T of 45Tderived
- Context
- 64K
- Learning rate
- 2.6×10⁻⁴ after 2,000 steps of warm-up
- Batch
- 100.6 million tokens
- Vision encoder
- frozen, but for its final normalization layer
Derived
Objective. No sentence of the report names the pre-training loss. The model “takes images and text as input and generates text autoregressively” (p. 7) and the base model is evaluated by bits per byte (p. 24): next-token prediction is concluded from these, and the report does not state it.
Not disclosed Batch in sequences · Gradient clipping · Initialization · Compute · The indexer’s training loss
Why
The report says the model is trained with sparse attention from scratch, with no dense warm-up stage. DeepSeek-V4.1 report §1, p. 6
Architecture at this stageThe backbone with sparse attention and no MTP module; the vision encoder is frozen, but for its final normalization layer and the vision–language projector, which train.
All settings15
- Objective
- Next-token prediction on text; no MTP modulederivedNo sentence of the report names the pre-training loss. The model “takes images and text as input and generates text autoregressively” (p. 7) and the base model is evaluated by bits per byte (p. 24): next-token prediction is concluded from these, and the report does not state it.
- No MTP
- “We omit the MTP module during backbone pre-training”
- Load balancing
- Auxiliary-loss-free: bias update speed 0.001 for image tokens and for text tokens, two sets of biases; a sequence-level balance loss with weight 0.0001
- Batch
- 100.6 million tokens, fixed throughout
- Attention
- Sparse from scratch, “without any dense attention warmup stages”
- Optimizer
- Muon for linear weights (momentum 0.95, weight decay 0.1, update RMS rescaled to 0.18), head-wise for Query and Key weights; a Sinkhorn-balanced momentum update for the embeddings and the prediction head (K = 11, τ = 10⁻³); AdamW for normalization weights and other non-matrix parameters (β₁ = 0.9, β₂ = 0.95; weight decay 0.1 on normalization weights, none on biases and scaling factors)
- Engram learning rate
- 5 × the learning rate
- Data
- Text and multimodal corpora joined, 7 : 1 in tokens of text-only to multimodal; multimodal data from the start of pre-training
- Packing
- Best-fit packing with a padding rate of at most 10⁻⁴; sample-level attention masking
- Context
- 64K from the start
- Learning rate
- Linear warm-up over the first 2,000 steps, then 2.6×10⁻⁴ until 28T tokens
- Tokens
- 28T of 45TderivedThe report gives the boundaries in tokens: the learning rate is constant until 28T and decays to 40T, and the sequence length changes at 34T, of 45T in all.
- Vision encoder
- Frozen until the learning-rate decay begins; its final normalization layer and the vision–language projector stay trainable
- Total
- 45T tokens of multimodal data
- Steps
- 45T ÷ 100.6M ≈ 447,000 for all of pre-training; the warm-up’s 2,000 steps are about 0.2T tokensderivedTotal tokens over the batch in tokens; steps times the batch.
Sources9
- DeepSeek_V41_Tech_Report.pdf, pp. 7–8Objective
- DeepSeek_V41_Tech_Report.pdf, p. 24Objective
- DeepSeek_V41_Tech_Report.pdf, p. 8No MTP, Load balancing
- DeepSeek_V41_Tech_Report.pdf, p. 22Load balancing, Batch, Attention, Optimizer, Engram learning rate, Packing, Context, Learning rate, Tokens, Total, Steps
- DeepSeek_V41_Tech_Report.pdf, p. 6Attention
- DeepSeek_V41_Tech_Report.pdf, pp. 14–16Optimizer
- DeepSeek_V41_Tech_Report.pdf, p. 21Data, Packing
- DeepSeek_V41_Tech_Report.pdf, p. 7Data
- DeepSeek_V41_Tech_Report.pdf, p. 15Vision encoder