GPT-2

All models, with previews →

OpenAI · Feb–Nov 2019Dense decoder

Masked multi-head self-attention

Reference code openai/gpt-2 @ 9b63575, huggingface/transformers @ 71c6f69, configuration openai-community/gpt2 @ 607a30d · checkpoint tensors not independently counted · How to read

Full sequenceTrainingPrefillDecode

Transformer block › Attention

Weighted value retrieval

×

no parameterstf.matmul(w, v)

Attention weights mix value vectors independently within each head.

Hi=A~iVi{\htmlData{light=head-mix attn-mix}{H_i}}={\htmlData{light=a-dropped}{\widetilde A_i}}{\htmlData{light=kv-cache}{{\htmlClass{vl-stream-v}{V_i}}}}
in
A~\widetilde A [B batch, 12 heads, 1, 1001]
out
HH [B batch, 12 heads, 1 positions, 64 d_h]

The formula uses Ã; the released code multiplies w, the softmax output, directly (model.py 97–98).

Codemodeling_gpt2.py 63–69 · model.py 96–99
modeling_gpt2.py 63–69
    attn_weights = nn.functional.softmax(attn_weights, dim=-1)

    # Downcast (if necessary) back to V's dtype (if in mixed-precision) -- No-Op otherwise
    attn_weights = attn_weights.type(value.dtype)
    attn_weights = nn.functional.dropout(attn_weights, p=dropout, training=module.training)

    attn_output = torch.matmul(attn_weights, value)
model.py 96–99
        w = mask_attn_weights(w)
        w = softmax(w)
        a = tf.matmul(w, v)
        return a
Sources2

Direct evidence

The maintained eager attention path normalizes scores, applies training-dependent dropout, and multiplies the resulting weights by values.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 63–69

Read code excerpt
    attn_weights = nn.functional.softmax(attn_weights, dim=-1)

    # Downcast (if necessary) back to V's dtype (if in mixed-precision) -- No-Op otherwise
    attn_weights = attn_weights.type(value.dtype)
    attn_weights = nn.functional.dropout(attn_weights, p=dropout, training=module.training)

    attn_output = torch.matmul(attn_weights, value)

Direct evidence

The normalized score matrix multiplies the value field inside each head.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 91–99

Read code excerpt
        w = mask_attn_weights(w)
        w = softmax(w)
        a = tf.matmul(w, v)
        return a