Llama 3.1

All models, with their blocks →

Size

405B

Published
Meta · Jul 2024
Line
Dense decoder · follows Llama 3
View
Source

The Llama 3 Herd of Models (arXiv 2407.21783v3) · DPO (arXiv 2305.18290v3) · no training code is publishedHow to read

DPO step

DPO step

NLL on the chosen

×

no parameters

The negative log-likelihood of the chosen response, which keeps its probability from falling and holds the generation format.

LNLL=−log⁡πθ(yw∣x)\mathcal L_{\mathrm{NLL}}=-\log\pi_\theta(y_w\mid x)
in
lPl_P [B batch, 2]
out
LNLL\mathcal L_{\mathrm{NLL}} [1]

This branch reads token likelihoods inside Policy, before the masked aggregation drawn as lPl_P; it does not take the NLL from that sum. Meta’s change, from the Llama 3 paper (p. 16): whether formatting tokens are masked in this term, and whether it is averaged over tokens, is not stated.

Sources1

Direct evidence

§4.1.4, Direct Preference Optimization: DPO on the SFT models with learning rate 10⁻⁵ and β = 0.1; special formatting tokens, including header and termination tokens, are masked out of the loss in chosen and rejected responses; an NLL term with scaling coefficient 0.2 is added on the chosen sequences.

meta · arXiv 2407.21783 / Llama3-Herd-2407.21783v3.pdf
Pinned revision v3 ↗ · p. 16