GPT-2

All models, with previews →

OpenAI · Feb–Nov 2019Dense decoder

Transformer block

Reference code openai/gpt-2 @ 9b63575, huggingface/transformers @ 71c6f69, configuration openai-community/gpt2 @ 607a30d · checkpoint tensors not independently counted · How to read

xn[B, T, 768]LayerNorm 1μ,  σ2\mu,\;\sigma^2gain · bias
AttentionΔattn=Attention⁡(Z1)\Delta_{\mathrm{attn}}=\operatorname{Attention}(Z_1)Open →
Dropoutpresidp_{\mathrm{resid}}+x′n[B, T, 768]LayerNorm 2μ,  σ2\mu,\;\sigma^2gain · bias
MLPΔmlp=MLP⁡(Z2)\Delta_{\mathrm{mlp}}=\operatorname{MLP}(Z_2)Open →
Dropoutpresidp_{\mathrm{resid}}+xn+1[B, T, 768]residual bypassresidual bypass

Transformer block

MLP residual merge

×

no parametersx + m

Add the mechanism’s update to the residual stream element by element.

xn+1=xn′+Dresid(Δmlp){\htmlData{light=x-out mlp-add}{x_{n+1}}}={\htmlData{light=x-attn attn-add}{x'_{n}}}+D_{\mathrm{resid}}({\htmlData{light=mlp}{\Delta_{\mathrm{mlp}}}})
in
xn′x'_{n} [B batch, T positions, 768 d]Dresid(Δmlp)D_{\mathrm{resid}}(\Delta_{\mathrm{mlp}}) [B batch, T positions, 768 d]
out
xn+1x_{n+1} [B batch, T positions, 768 d]

DresidD_{\mathrm{resid}} is dropout during training and the identity at evaluation. The maintained code applies it inside the MLP module (modeling_gpt2.py 242); the released code adds mm as it is.

Codemodel.py 128–130 · modeling_gpt2.py 303–305, 307
model.py 128–130
        m = mlp(norm(x, 'ln_2'), 'mlp', nx*4, hparams=hparams)
        x = x + m
        return x, present
modeling_gpt2.py 303–305
        residual = hidden_states
        hidden_states = self.ln_2(hidden_states)
        feed_forward_hidden_states = self.mlp(hidden_states)
modeling_gpt2.py 307
        hidden_states = residual + feed_forward_hidden_states
Sources2

Direct evidence

Second branch and block return in the reference graph.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 123–130

Read code excerpt
        m = mlp(norm(x, 'ln_2'), 'mlp', nx*4, hparams=hparams)
        x = x + m
        return x, present

Direct evidence

The same residual contract in the maintained PyTorch implementation.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 303–309

Read code excerpt
        residual = hidden_states
        hidden_states = self.ln_2(hidden_states)
        feed_forward_hidden_states = self.mlp(hidden_states)

… lines 306–306 omitted …

        hidden_states = residual + feed_forward_hidden_states

Incoming transform evidence

The maintained training path applies residual dropout after the MLP projection and before the residual write; evaluation makes dropout the identity.

MLP residual dropout · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 238–243

Read code excerpt
        hidden_states = self.c_proj(hidden_states)
        hidden_states = self.dropout(hidden_states)
        return hidden_states