Training

  1. Start of training
  3. Initial pre-trainingat 4K, then 8K15.6T8KNTPall
  5. Long-context pre-trainingsix stages, 8K to 128K800B128KNTPallcontext to 128K
  6. Annealinglearning rate to 040M128KNTPall
  7. Pre-trained model
  8. Off this path: Reward model1 stage
  9. Reward modelin each roundnot statednot statedRMLlama 2’s, no marginnot stated
  10. Reward model
  11. × 6 roundsRejection sampling and SFTthen DPO, each roundnot stated128KSFTRS and synthetic datanot stated
  12. DPOafter SFT, each roundnot statedshortDPO+ NLL 0.2not stated
  13. Llama 3.1 405B Instruct
  • tokens, log scale
  • derived: computed from stated numbers
  • context length
  • all parameters train
  • a part trains
  • all but a part trains
  • NTPdashed: concluded, no source names it
  • not stated by the sources
  • the architecture changes
  • checkpoint
  • checkpoint, weights published
  • a join the sources leave unexplained
  • data from a branch into a stage
  • stages repeated together

Long-context pre-training

×

Stage · Llama 3.1 405B

The context grows in six stages from 8K to 128K, on about 800B tokens. A document mask, which the paper calls important for continued pre-training on very long sequences, keeps attention within a document.

Context
six stages, 8K to 128K
Tokens
approximately 800B
When a stage ends
When short-context evaluations have recovered completely and the model solves “needle in a haystack” tasks up to that length

Derived

Objective. The paper describes this stage as continued pre-training and names no other objective: next-token prediction is concluded from that, and the paper does not state it for the stage.

Trained. The paper names no part of the model held fixed in pre-training: that every parameter trains is concluded, and the paper does not state it.

Not disclosed Lengths of the intermediate stages · Learning rate

Architecture at this stageThe context grows to 128K. The paper mentions no RoPE change for this stage; the released Llama 3.1 scales its RoPE frequencies (catalogue use_scaled_rope; llama-models model.py 45–69), at a stage the paper does not name.

All settings8
Context
Six stages, from 8K to 128K
Tokens
Approximately 800B
When a stage ends
When short-context evaluations have recovered completely and the model solves “needle in a haystack” tasks up to that length
Batch
16M tokens of 131,072-token sequences in the 128K configuration; the tokens per batch are kept constant
RoPE
Llama 3.1’s catalogue entries set use_scaled_rope, Llama 3’s set it False; it scales the frequencies by apply_scaling, a factor of 8 over an original context of 8,192 (llama-models model.py 45–69). The paper mentions no RoPE change for long-context pre-training; its one RoPE setting is the base frequency 500,000 (p. 7), which Llama 3 already has. At which stage the scaling enters is not stated
Attention mask
Attention between documents of one sequence is masked, which the paper calls important in continued pre-training on very long sequences
Objective
Next-token predictionderivedThe paper describes this stage as continued pre-training and names no other objective: next-token prediction is concluded from that, and the paper does not state it for the stage.
Trainable
AllderivedThe paper names no part of the model held fixed in pre-training: that every parameter trains is concluded, and the paper does not state it.
Sources11

Llama 3.1

All models, with their blocks →

Size

405B

Published
Meta · Jul 2024
Line
Dense decoder · follows Llama 3
View
Source

The Llama 3 Herd of Models (arXiv 2407.21783v3), whose results are all for Llama 3.1 · for Llama 3, its model card (meta‑llama/llama3 @ a0940f9) and Meta’s post of April 2024 · no training code is publishedHow to read