GPT-2

All models, with previews →

OpenAI · Feb–Nov 2019Dense decoder

Masked multi-head self-attention

Reference code openai/gpt-2 @ 9b63575, huggingface/transformers @ 71c6f69, configuration openai-community/gpt2 @ 607a30d · checkpoint tensors not independently counted · How to read

Full sequenceTrainingPrefillDecode

Transformer block › Attention

Attention residual update

×

tensora · 126

The projected attention output, which the attention module's residual dropout acts on during training before the block's addition.

shape
Δattn\Delta_{\mathrm{attn}} [B batch, T positions, 768 d]
Sources

Contextual evidence from Attention residual merge

First branch of the original GPT-2 block.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 123–130

Read code excerpt
        a, present = attn(norm(x, 'ln_1'), 'attn', nx, past=past, hparams=hparams)
        x = x + a

Contextual evidence from Attention residual merge

The PyTorch block saves residual before normalization and adds it after attention.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 272–282

Read code excerpt
        residual = hidden_states
        hidden_states = self.ln_1(hidden_states)
        attn_output, _ = self.attn(

… lines 275–281 omitted …

        hidden_states = attn_output + residual

Contextual evidence from Attention output projection

The released projection helper owns a learned matrix and output bias and applies them to the last feature axis.

linear-operation · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 50–56

Read code excerpt
def conv1d(x, scope, nf, *, w_init_stdev=0.02):
    with tf.variable_scope(scope):
        *start, nx = shape_list(x)
        w = tf.get_variable('w', [1, nx, nf], initializer=tf.random_normal_initializer(stddev=w_init_stdev))
        b = tf.get_variable('b', [nf], initializer=tf.constant_initializer(0))
        c = tf.reshape(tf.matmul(tf.reshape(x, [-1, nx]), tf.reshape(w, [-1, nf]))+b, start+[nf])
        return c

Contextual evidence from Attention output projection

The attention output merges its head axis and passes through c_proj before returning to the residual stream.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 109–112

Read code excerpt
        a = multihead_attn(q, k, v)
        a = merge_heads(a)
        a = conv1d(a, 'c_proj', n_state)
        return a, present

Contextual evidence from Attention output projection

The maintained attention path merges heads, applies c_proj, and returns the residual update.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 222–226

Read code excerpt
        attn_output = attn_output.reshape(*attn_output.shape[:-2], -1).contiguous()
        attn_output = self.c_proj(attn_output)
        attn_output = self.resid_dropout(attn_output)

        return attn_output, attn_weights