Pre-training 28–34T
×Stage · DeepSeek-V4.1-Flash
The learning rate starts its cosine decay and the vision encoder is unfrozen; the sequence length is still 64K.
- Tokens
- 6Tderived
- Context
- 64K
- Learning rate
- cosine decay from 2.6×10⁻⁴
- Vision encoder
- unfrozen
Derived
Objective. No sentence of the report names the pre-training loss. The model “takes images and text as input and generates text autoregressively” (p. 7) and the base model is evaluated by bits per byte (p. 24): next-token prediction is concluded from these, and the report does not state it.
Trained. The report says the vision encoder is unfrozen at the onset of the decay and trained jointly with the model, and names no part held fixed after it: that every parameter trains is concluded. The report lists no trainable set.
Not disclosed The encoder’s learning rate · Batch in sequences · Gradient clipping · Initialization · Compute · The indexer’s training loss
Architecture at this stageThe same model; the vision encoder now trains with it, at a smaller learning rate.
All settings13
- Objective
- Next-token prediction on text; no MTP modulederivedNo sentence of the report names the pre-training loss. The model “takes images and text as input and generates text autoregressively” (p. 7) and the base model is evaluated by bits per byte (p. 24): next-token prediction is concluded from these, and the report does not state it.
- No MTP
- “We omit the MTP module during backbone pre-training”
- Load balancing
- Auxiliary-loss-free: bias update speed 0.001 for image tokens and for text tokens, two sets of biases; a sequence-level balance loss with weight 0.0001
- Batch
- 100.6 million tokens, fixed throughout
- Attention
- Sparse from scratch, “without any dense attention warmup stages”
- Optimizer
- Muon for linear weights (momentum 0.95, weight decay 0.1, update RMS rescaled to 0.18), head-wise for Query and Key weights; a Sinkhorn-balanced momentum update for the embeddings and the prediction head (K = 11, τ = 10⁻³); AdamW for normalization weights and other non-matrix parameters (β₁ = 0.9, β₂ = 0.95; weight decay 0.1 on normalization weights, none on biases and scaling factors)
- Engram learning rate
- 5 × the learning rate
- Data
- Text and multimodal corpora joined, 7 : 1 in tokens of text-only to multimodal; multimodal data from the start of pre-training
- Packing
- Best-fit packing with a padding rate of at most 10⁻⁴; sample-level attention masking
- Context
- 64K, until 34T tokens
- Learning rate
- Cosine decay from 2.6×10⁻⁴ to 2.6×10⁻⁵ between 28T and 40T tokens
- Tokens
- 34T − 28T = 6TderivedThe report gives the boundaries in tokens: the learning rate is constant until 28T and decays to 40T, and the sequence length changes at 34T, of 45T in all.
- Vision encoder
- From the onset of learning-rate decay, unfrozen and trained with the model at a smaller learning rate
Sources9
- DeepSeek_V41_Tech_Report.pdf, pp. 7–8Objective
- DeepSeek_V41_Tech_Report.pdf, p. 24Objective
- DeepSeek_V41_Tech_Report.pdf, p. 8No MTP, Load balancing
- DeepSeek_V41_Tech_Report.pdf, p. 22Load balancing, Batch, Attention, Optimizer, Engram learning rate, Packing, Context, Learning rate, Tokens
- DeepSeek_V41_Tech_Report.pdf, p. 6Attention
- DeepSeek_V41_Tech_Report.pdf, pp. 14–16Optimizer
- DeepSeek_V41_Tech_Report.pdf, p. 21Data, Packing
- DeepSeek_V41_Tech_Report.pdf, p. 7Data
- DeepSeek_V41_Tech_Report.pdf, p. 15Vision encoder