Llama 3.1

All models, with their blocks →

Size

405B

Published
Meta · Jul 2024
Line
Dense decoder · follows Llama 3
View
Source

The Llama 3 Herd of Models (arXiv 2407.21783v3) · DPO (arXiv 2305.18290v3) · no training code is publishedHow to read

DPO step

DPO step

Reference

×

frozen · not trained

The reference model gives the same log-probabilities; the DPO paper takes it to be the initial SFT model.

ℓref(y)=∑t: mt=1log⁡πref(yt∣x,y<t)\ell_{\mathrm{ref}}(y)=\sum_{t:\,m_t=1}\log\pi_{\mathrm{ref}}(y_t\mid {\htmlData{light=prompt}{x}},y_{<t})
in
xx [B batch]mm [B batch, 2, T positions]
out
lRl_R [B batch, 2]

The Llama 3 paper does not name the reference for its DPO. ℓref\ell_{\mathrm{ref}}, drawn lRl_R, is this plate’s name.

Sources1

Direct evidence

§3–4: β controls the deviation from the reference policy π_ref, “namely the initial SFT model”; π_ref is initialized to π_SFT whenever available.

ref · arXiv 2305.18290 / DPO-2305.18290v3.pdf
Pinned revision v3 ↗ · pp. 4–5