- out
- [1]
Explanation
Everything here is the Llama 3 paper’s (§4.1.3, §4.2.2). K is typically 10 to 30. In later rounds system prompts steer the sampled responses’ tone, style or formatting; for code prompts the code expert does the rejection sampling (p. 19).
The fine-tuning data are not only these: the rejection-sampled responses are mixed with other sources, synthetic data among them. Learning rate over 8.5K to 9K steps for the largest models.
Sources3
Direct evidence
§4.2.2, Rejection sampling: for each prompt collected during human annotation, K outputs (typically between 10 and 30) are sampled from the latest chat model policy, usually the best checkpoint of the previous post-training iteration or the best for a capability; the reward model selects the best candidate. In later rounds system prompts steer the responses’ tone, style or formatting.
rs · arXiv 2407.21783 / Llama3-Herd-2407.21783v3.pdf
Pinned revision v3 ↗ · p. 18
Direct evidence
§4.1.3: the reward model performs rejection sampling on human annotation prompts; with this data and other sources, including synthetic data, the pre-trained language model is fine-tuned with a standard cross-entropy loss on the target tokens, the loss masked on prompt tokens; learning rate 10⁻⁵ over 8.5K to 9K steps for the largest models.
sft · arXiv 2407.21783 / Llama3-Herd-2407.21783v3.pdf
Pinned revision v3 ↗ · p. 16
Direct evidence
§4.1.2: the reward model is trained on top of the pre-trained checkpoint, with Llama 2’s objective without the margin term.
rm · arXiv 2407.21783 / Llama3-Herd-2407.21783v3.pdf
Pinned revision v3 ↗ · p. 16