Pre-training 34–45T
×Stage · DeepSeek-V4.1-Flash
The sequence length is extended to 1M tokens; the decay ends at 40T and the learning rate stays at 2.6×10⁻⁵ to 45T.
- Tokens
- 11Tderived
- Context
- 1M
- Learning rate
- down to 2.6×10⁻⁵ at 40T
Derived
Objective. No sentence of the report names the pre-training loss. The model “takes images and text as input and generates text autoregressively” (p. 7) and the base model is evaluated by bits per byte (p. 24): next-token prediction is concluded from these, and the report does not state it.
Trained. The report says the vision encoder is unfrozen at the onset of the decay and trained jointly with the model, and names no part held fixed after it: that every parameter trains is concluded. The report lists no trainable set.
Not disclosed The encoder’s learning rate · Batch in sequences · Gradient clipping · Initialization · Compute · The indexer’s training loss
Architecture at this stageThe same model at a sequence length of 1M.
All settings13
- Objective
- Next-token prediction on text; no MTP modulederivedNo sentence of the report names the pre-training loss. The model “takes images and text as input and generates text autoregressively” (p. 7) and the base model is evaluated by bits per byte (p. 24): next-token prediction is concluded from these, and the report does not state it.
- No MTP
- “We omit the MTP module during backbone pre-training”
- Load balancing
- Auxiliary-loss-free: bias update speed 0.001 for image tokens and for text tokens, two sets of biases; a sequence-level balance loss with weight 0.0001
- Batch
- 100.6 million tokens, fixed throughout
- Attention
- Sparse from scratch, “without any dense attention warmup stages”
- Optimizer
- Muon for linear weights (momentum 0.95, weight decay 0.1, update RMS rescaled to 0.18), head-wise for Query and Key weights; a Sinkhorn-balanced momentum update for the embeddings and the prediction head (K = 11, τ = 10⁻³); AdamW for normalization weights and other non-matrix parameters (β₁ = 0.9, β₂ = 0.95; weight decay 0.1 on normalization weights, none on biases and scaling factors)
- Engram learning rate
- 5 × the learning rate
- Data
- Text and multimodal corpora joined, 7 : 1 in tokens of text-only to multimodal; multimodal data from the start of pre-training
- Packing
- Best-fit packing with a padding rate of at most 10⁻⁴; sample-level attention masking
- Context
- 1M, from 34T tokens
- Learning rate
- The cosine decay continues to 2.6×10⁻⁵ at 40T tokens, then constant to 45T
- Tokens
- 45T − 34T = 11TderivedThe report gives the boundaries in tokens: the learning rate is constant until 28T and decays to 40T, and the sequence length changes at 34T, of 45T in all.
- Vision encoder
- From the onset of learning-rate decay, unfrozen and trained with the model at a smaller learning rate
Sources9
- DeepSeek_V41_Tech_Report.pdf, pp. 7–8Objective
- DeepSeek_V41_Tech_Report.pdf, p. 24Objective
- DeepSeek_V41_Tech_Report.pdf, p. 8No MTP, Load balancing
- DeepSeek_V41_Tech_Report.pdf, p. 22Load balancing, Batch, Attention, Optimizer, Engram learning rate, Packing, Context, Learning rate, Tokens
- DeepSeek_V41_Tech_Report.pdf, p. 6Attention
- DeepSeek_V41_Tech_Report.pdf, pp. 14–16Optimizer
- DeepSeek_V41_Tech_Report.pdf, p. 21Data, Packing
- DeepSeek_V41_Tech_Report.pdf, p. 7Data
- DeepSeek_V41_Tech_Report.pdf, p. 15Vision encoder