ℒ[1]hatched band: not updated in this step · solid band: what this stage trainsDPO masks formatting tokens · NLL reads policy token likelihoods; its mask is not stated

DPO step

Policy log-probabilities

tensor

Of the chosen and of the rejected response.

shape: $l_P$ [B batch, 2]

Sources

Contextual evidence from Policy

§4, eq. 7: L_DPO(π_θ; π_ref) = −E[log σ(β log π_θ(y_w|x)/π_ref(y_w|x) − β log π_θ(y_l|x)/π_ref(y_l|x))], over preference triples (x, y_w, y_l).

dpo · arXiv 2305.18290 / DPO-2305.18290v3.pdf
Pinned revision v3 ↗ · p. 4

Contextual evidence from Policy

§4.1.4, Direct Preference Optimization: DPO on the SFT models with learning rate 10⁻⁵ and β = 0.1; special formatting tokens, including header and termination tokens, are masked out of the loss in chosen and rejected responses; an NLL term with scaling coefficient 0.2 is added on the chosen sequences.

meta · arXiv 2407.21783 / Llama3-Herd-2407.21783v3.pdf
Pinned revision v3 ↗ · p. 16