Rejection sampling and SFT
×Stage · Llama 3.1 405B
For each human-annotation prompt the latest policy writes typically 10 to 30 responses and the reward model keeps the best; the pre-trained model is fine-tuned on these and on synthetic and other data, the loss on the prompt masked.
- Rounds
- six, each an SFT followed by DPO
- Rejection sampling
- typically 10 to 30 outputs, the best by the reward model
- Learning rate and steps
- 10⁻⁵, 8.5K to 9K steps
- Model averaging
- Models from experiments with various data or hyperparameters are averaged at each RM, SFT and DPO stage
Not disclosed Tokens · Batch
The training step →What one step of this stage computes, as a plate.
Architecture at this stageThe pre-trained checkpoint fine-tuned; the previous round’s best model produces the round’s data.
All settings6
- Rounds
- Six rounds, each an SFT followed by DPO. Each round’s data are generated with the best models of the previous round: “Best model for next round” (Figure 7), “sampling synthetic data from the latest models”. The paper says SFT fine-tunes pre-trained checkpoints; that a round starts from the previous round’s model is not stated.
- Rejection sampling
- K outputs, typically 10 to 30, from the latest chat policy for each prompt; the reward model selects the best
- Objective
- Cross-entropy on the target tokens, the loss masked on the prompt tokens
- Learning rate and steps
- 10⁻⁵ over 8.5K to 9K steps for the largest models
- Context
- Mostly short; 0.1% of synthetic long-context data at 16K, 32K, 64K and 128K
- Model averaging
- Models from experiments with various data or hyperparameters are averaged at each RM, SFT and DPO stage
Sources5
- Llama3-Herd-2407.21783v3.pdf, p. 15Rounds
- Llama3-Herd-2407.21783v3.pdf, p. 17Rounds
- Llama3-Herd-2407.21783v3.pdf, p. 18Rejection sampling
- Llama3-Herd-2407.21783v3.pdf, p. 16Objective, Learning rate and steps, Model averaging
- Llama3-Herd-2407.21783v3.pdf, p. 24Context