Training

  1. Start of training
  2. Llama 3, April 2024, predecessor2 stages
  3. Whether Llama 3.1 continues these checkpoints: not stated
  4. Llama 3 pre-training15T+ tokens at 8K15T8Knot statednot stated
  5. Llama 3 post-trainingSFT, RS, PPO, DPOnot statednot statedSFTPPODPOnot stated
  6. Llama 3 (April 2024)
  7. Initial pre-trainingat 4K, then 8K15.6T8KNTPall
  8. Off this path: domain experts (code, multilingual)2 stages
  9. Branched from the pre-training run; the point is not named
  10. Code expert1T tokens, mostly code1T16KNTPnot stated
  11. Multilingual expert90% multilingual tokensnot statednot statedNTPnot stated
  12. Long-context pre-trainingsix stages, 8K to 128K800B128KNTPallcontext to 128K
  13. Annealinglearning rate to 040M128KNTPall
  14. Pre-trained model
  15. Off this path: Reward model1 stage
  16. Reward modelin each roundnot statednot statedRMLlama 2’s, no marginnot stated
  17. Reward model
  18. × 6 roundsRejection sampling and SFTthen DPO, each roundnot stated128KSFTRS and synthetic datanot stated
  19. DPOafter SFT, each roundnot statedshortDPO+ NLL 0.2not stated
  20. Llama 3.1 405B Instruct
  • tokens, log scale
  • derived: computed from stated numbers
  • context length
  • all parameters train
  • a part trains
  • all but a part trains
  • NTPdashed: concluded, no source names it
  • not stated by the sources
  • the architecture changes
  • checkpoint
  • checkpoint, weights published
  • a join the sources leave unexplained
  • data from a branch into a stage
  • stages repeated together

Branched from the pre-training run; the point is not named

×

Join the sources leave unexplained

The paper: “branching the main pre-training run” (code, p. 19) and “branching off the pre-training run” (multilingual, p. 22), naming no stage; the multilingual expert collects annotations “until pre-training was fully complete”. The branch is drawn from the initial stage, as a reading.

Sources2
Key
stage
tokens, log scale
a total no source states, computed from steps × batch × length
checkpoint
checkpoint, weights published
a join the sources leave unexplained
stages of a released model, folded into one row
derived
computed from reported values; the arithmetic is shown
not disclosed
no pinned source states it; nothing is filled in

Llama 3.1

All models, with their blocks →

Size

405B

Published
Meta · Jul 2024
Line
Dense decoder · follows Llama 3
View
Source

The Llama 3 Herd of Models (arXiv 2407.21783v3), whose results are all for Llama 3.1 · for Llama 3, its model card (meta‑llama/llama3 @ a0940f9) and Meta’s post of April 2024 · no training code is publishedHow to read