Training

Next-token prediction on 14.8T tokens at 4K context, with multi-token prediction and auxiliary-loss-free load balancing.

Architecture. A multi-token prediction module of depth 1 trains alongside the model in pre-training; no source says whether later stages keep it. The report says inference can discard it or reuse it for speculative decoding (p. 11); the pinned inference code builds none (model.py 881–886).

Data

14.8T tokens; more mathematics and code than DeepSeek-V2; multilingual beyond English and Chinese; documents packed without cross-sample masking

Objective

Next-token prediction; Fill-in-Middle (prefix–suffix–middle) at rate 0.1; multi-token prediction of depth 1, weight 0.3 for the first 10T tokens, then 0.1

Load balancing

Auxiliary-loss-free bias, update speed 0.001 for the first 14.3T tokens, then 0; sequence-wise balance loss α = 0.0001

Optimiser

AdamW, β₁ = 0.9, β₂ = 0.95, weight decay 0.1; gradient clipping at 1.0

Learning rate

0 → 2.2×10⁻⁴ over the first 2K steps; constant until 10T tokens; cosine down to 2.2×10⁻⁵ over 4.3T; 2.2×10⁻⁵ for 333B; 7.3×10⁻⁶ for the last 167B

Context

4K

Batch

3,072 → 15,360 sequences over the first 469B tokens, then 15,360

Tokens

10T + 4.3T + 0.5T = 14.8TderivedThe learning-rate phases add up to the reported total.

Compute

2,664K H800 GPU hours

Number format

FP8 mixed-precision training

Trainable

All parameters, randomly initialised

Not disclosed Data mixture proportions