Training

  1. Start of training
  3. Initial pre-trainingat 4K, then 8K15.6T8KNTPall
  4. Off this path: domain experts (code, multilingual)2 stages
  5. Branched from the pre-training run; the point is not named
  6. Code expert1T tokens, mostly code1T16KNTPnot stated
  7. Multilingual expert90% multilingual tokensnot statednot statedNTPnot stated
  8. Long-context pre-trainingsix stages, 8K to 128K800B128KNTPallcontext to 128K
  9. Annealinglearning rate to 040M128KNTPall
  10. Pre-trained model
  12. × 6 roundsRejection sampling and SFTthen DPO, each roundnot stated128KSFTRS and synthetic datanot stated
  13. DPOafter SFT, each roundnot statedshortDPO+ NLL 0.2not stated
  14. Llama 3.1 405B Instruct
  • tokens, log scale
  • derived: computed from stated numbers
  • context length
  • all parameters train
  • a part trains
  • all but a part trains
  • NTPdashed: concluded, no source names it
  • not stated by the sources
  • the architecture changes
  • checkpoint
  • checkpoint, weights published
  • a join the sources leave unexplained
  • data from a branch into a stage
  • stages repeated together

DPO

×

Stage · Llama 3.1 405B

Direct preference optimization on the latest preference data, with the formatting tokens masked out of the loss and a negative log-likelihood term on the chosen responses.

Objective
DPO, formatting tokens masked, NLL 0.2 on chosen
Learning rate and β
10⁻⁵; β = 0.1
Rounds
six, each an SFT followed by DPO

Not disclosed Tokens

Why

PPO was explored; DPO required less compute for large models and performed better, especially on instruction following. Llama 3 paper §4.1.4, p. 16

Masking the formatting tokens stabilizes training. The paper hypothesizes that, since they are in both chosen and rejected responses, the contrastive loss would push their likelihood up and down at once. The NLL term keeps the chosen responses’ log probability from falling. Llama 3 paper §4.1.4, p. 16

The training step →What one step of this stage computes, as a plate.

Architecture at this stageThe SFT model, aligned to preferences.

All settings8
Rounds
Six rounds, each an SFT followed by DPO. Each round’s data are generated with the best models of the previous round: “Best model for next round” (Figure 7), “sampling synthetic data from the latest models”. The paper says SFT fine-tunes pre-trained checkpoints; that a round starts from the previous round’s model is not stated.
Objective
DPO, with the formatting tokens masked out of the loss in chosen and rejected responses, and an NLL term with coefficient 0.2 on the chosen sequences
Learning rate and β
10⁻⁵; β = 0.1
Data
Mostly the latest batches of preference data, collected with the best models of the previous rounds; only samples labelled significantly better or better
Context
Short-context data only, on top of the long-context SFT checkpoints
Steps
Fewer optimizer steps than SFT
Model averaging
Models from experiments with various data or hyperparameters are averaged at each RM, SFT and DPO stage
Against PPO
PPO was explored; DPO “required less compute for large-scale models and performed better”
Sources4

Llama 3.1

All models, with their blocks →

Size

405B

Published
Meta · Jul 2024
Line
Dense decoder · follows Llama 3
View
Source

The Llama 3 Herd of Models (arXiv 2407.21783v3), whose results are all for Llama 3.1 · for Llama 3, its model card (meta‑llama/llama3 @ a0940f9) and Meta’s post of April 2024 · no training code is publishedHow to read