GPT-2

All models, with previews →

OpenAI · Feb–Nov 2019Dense decoder

Transformer block

Reference code openai/gpt-2 @ 9b63575, huggingface/transformers @ 71c6f69, configuration openai-community/gpt2 @ 607a30d · checkpoint tensors not independently counted · How to read

xn[B, T, 768]LayerNorm 1μ,  σ2\mu,\;\sigma^2gain · bias
AttentionΔattn=Attention⁡(Z1)\Delta_{\mathrm{attn}}=\operatorname{Attention}(Z_1)Open →
Dropoutpresidp_{\mathrm{resid}}+x′n[B, T, 768]LayerNorm 2μ,  σ2\mu,\;\sigma^2gain · bias
MLPΔmlp=MLP⁡(Z2)\Delta_{\mathrm{mlp}}=\operatorname{MLP}(Z_2)Open →
Dropoutpresidp_{\mathrm{resid}}+xn+1[B, T, 768]residual bypassresidual bypass

Transformer block

Block input state

×

tensorx · 123

The residual features entering this block.

shape
xnx_{n} [B batch, T positions, 768 d]
Sources

Contextual evidence from Embedding addition

For the base GPT-2 text path, embedding dropout follows token-plus-position addition and precedes the block loop; optional token-type embeddings are outside the base text path.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 576–608

Read code excerpt
        hidden_states = inputs_embeds + position_embeds.to(inputs_embeds.device)

… lines 578–603 omitted …

        hidden_states = self.drop(hidden_states)

… lines 605–607 omitted …

        for i, block in enumerate(self.h):

Contextual evidence from Embedding addition

The original graph gathers the two tables and adds them into the initial stream.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 152–157

Read code excerpt
        h = tf.gather(wte, X) + tf.gather(wpe, positions_for(X, past_length))

Contextual evidence from Pre-attention LayerNorm

Original LayerNorm implementation and ε = 10⁻⁵.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 28–38

Read code excerpt
def norm(x, scope, *, axis=-1, epsilon=1e-5):
    """Normalize to mean = 0, std = 1, then do a diagonal affine transform."""
    with tf.variable_scope(scope):
        n_state = x.shape[-1].value
        g = tf.get_variable('g', [n_state], initializer=tf.constant_initializer(1))
        b = tf.get_variable('b', [n_state], initializer=tf.constant_initializer(0))
        u = tf.reduce_mean(x, axis=axis, keepdims=True)
        s = tf.reduce_mean(tf.square(x-u), axis=axis, keepdims=True)
        x = (x - u) * tf.rsqrt(s + epsilon)
        x = x*g + b
        return x

Contextual evidence from Pre-attention LayerNorm

PyTorch delegates the operation to nn.LayerNorm with the checkpoint epsilon.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 252–273

Read code excerpt
        self.ln_1 = nn.LayerNorm(hidden_size, eps=config.layer_norm_epsilon)

… lines 253–271 omitted …

        residual = hidden_states
        hidden_states = self.ln_1(hidden_states)

Contextual evidence from Attention residual merge

First branch of the original GPT-2 block.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 123–130

Read code excerpt
        a, present = attn(norm(x, 'ln_1'), 'attn', nx, past=past, hparams=hparams)
        x = x + a

Contextual evidence from Attention residual merge

The PyTorch block saves residual before normalization and adds it after attention.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 272–282

Read code excerpt
        residual = hidden_states
        hidden_states = self.ln_1(hidden_states)
        attn_output, _ = self.attn(

… lines 275–281 omitted …

        hidden_states = attn_output + residual