GPT-2

All models, with previews →

OpenAI · Feb–Nov 2019Dense decoder

Transformer block

Reference code openai/gpt-2 @ 9b63575, huggingface/transformers @ 71c6f69, configuration openai-community/gpt2 @ 607a30d · checkpoint tensors not independently counted · How to read

xn[B, T, 768]LayerNorm 1μ,  σ2\mu,\;\sigma^2gain · bias
AttentionΔattn=Attention⁡(Z1)\Delta_{\mathrm{attn}}=\operatorname{Attention}(Z_1)Open →
Dropoutpresidp_{\mathrm{resid}}+x′n[B, T, 768]LayerNorm 2μ,  σ2\mu,\;\sigma^2gain · bias
MLPΔmlp=MLP⁡(Z2)\Delta_{\mathrm{mlp}}=\operatorname{MLP}(Z_2)Open →
Dropoutpresidp_{\mathrm{resid}}+xn+1[B, T, 768]residual bypassresidual bypass
in
xnx_{n} [B batch, T positions, 768 d]
out
xn+1x_{n+1} [B batch, T positions, 768 d]
parameters
7,087,872 · × 12 blocks = 85,054,464
Parameters

7,087,872 parameters per block, across its learned components

Exact per-instance accounting
ComponentCalculationParameters
Pre-attention LayerNorm768+768768 + 7681,536
Masked multi-head self-attention2,362,368 across its components2,362,368
Pre-MLP LayerNorm768+768768 + 7681,536
Position-wise feed-forward network4,722,432 across its components4,722,432
Per-instance total7,087,872

Across 12 independent blocks: 7,087,872×12=85,054,4647{,}087{,}872 \times 12 = 85{,}054{,}464 parameters.

Each of the 12 block addresses owns independent weights. The implementation names the first and last addresses h.0 and h.11.

Codemodel.py 126–130 · modeling_gpt2.py 272–307
model.py 126–130
        a, present = attn(norm(x, 'ln_1'), 'attn', nx, past=past, hparams=hparams)
        x = x + a
        m = mlp(norm(x, 'ln_2'), 'mlp', nx*4, hparams=hparams)
        x = x + m
        return x, present
modeling_gpt2.py 272–280
        residual = hidden_states
        hidden_states = self.ln_1(hidden_states)
        attn_output, _ = self.attn(
            hidden_states,
            past_key_values=past_key_values,
            attention_mask=attention_mask,
            use_cache=use_cache,
            **kwargs,
        )
modeling_gpt2.py 282
        hidden_states = attn_output + residual
modeling_gpt2.py 303–307
        residual = hidden_states
        hidden_states = self.ln_2(hidden_states)
        feed_forward_hidden_states = self.mlp(hidden_states)
        # residual connection
        hidden_states = residual + feed_forward_hidden_states
Sources2

Direct evidence

The released block performs both pre-normalized residual writes and returns the updated stream.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 123–130

Read code excerpt
        a, present = attn(norm(x, 'ln_1'), 'attn', nx, past=past, hparams=hparams)
        x = x + a
        m = mlp(norm(x, 'ln_2'), 'mlp', nx*4, hparams=hparams)
        x = x + m
        return x, present

Direct evidence

The maintained implementation preserves the same two residual branches.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 272–307

Read code excerpt
        residual = hidden_states
        hidden_states = self.ln_1(hidden_states)
        attn_output, _ = self.attn(
            hidden_states,
            past_key_values=past_key_values,
            attention_mask=attention_mask,
            use_cache=use_cache,
            **kwargs,
        )

… lines 281–281 omitted …

        hidden_states = attn_output + residual

… lines 283–302 omitted …

        residual = hidden_states
        hidden_states = self.ln_2(hidden_states)
        feed_forward_hidden_states = self.mlp(hidden_states)
        # residual connection
        hidden_states = residual + feed_forward_hidden_states