Llama 3.1

All models, with their blocks →

Size

405B

Published
Meta · Jul 2024
Line
Dense decoder · follows Llama 3
View
Source

The Llama 3 Herd of Models (arXiv 2407.21783v3) · DPO (arXiv 2305.18290v3) · no training code is publishedHow to read

DPO step

DPO step

DPO loss

×

no parameters

The chosen response should gain probability against the reference more than the rejected one does; the logistic loss of that margin, scaled by β=0.1\beta=0.1.

LDPO=−E(x,yw,yl)[log⁡σ(βlog⁡πθ(yw∣x)πref(yw∣x)−βlog⁡πθ(yl∣x)πref(yl∣x))]\mathcal L_{\mathrm{DPO}}=-\mathbb E_{(x,y_w,y_l)}\Big[\log\sigma\Big(\beta\log\frac{\pi_\theta(y_w\mid x)}{\pi_{\mathrm{ref}}(y_w\mid x)}-\beta\log\frac{\pi_\theta(y_l\mid x)}{\pi_{\mathrm{ref}}(y_l\mid x)}\Big)\Big]
β=0.1\beta=0.1
in
lPl_P [B batch, 2]lRl_R [B batch, 2]
out
LDPO\mathcal L_{\mathrm{DPO}} [1]

Eq. 7 of the DPO paper; β = 0.1 is the Llama 3 paper’s (p. 16).

Sources2

Direct evidence

§4, eq. 7: L_DPO(π_θ; π_ref) = −E[log σ(β log π_θ(y_w|x)/π_ref(y_w|x) − β log π_θ(y_l|x)/π_ref(y_l|x))], over preference triples (x, y_w, y_l).

dpo · arXiv 2305.18290 / DPO-2305.18290v3.pdf
Pinned revision v3 ↗ · p. 4

Direct evidence

§4.1.4, Direct Preference Optimization: DPO on the SFT models with learning rate 10⁻⁵ and β = 0.1; special formatting tokens, including header and termination tokens, are masked out of the loss in chosen and rejected responses; an NLL term with scaling coefficient 0.2 is added on the chosen sequences.

meta · arXiv 2407.21783 / Llama3-Herd-2407.21783v3.pdf
Pinned revision v3 ↗ · p. 16