GPT-2

All models, with their blocks →

Size
Published
OpenAI · Feb–Nov 2019
Line
Dense decoder
View
Architecture
Source

openai/gpt‑2 @ 9b63575, huggingface/transformers @ 71c6f69, configuration openai‑community/gpt2 @ 607a30d · checkpoint tensors not independently countedHow to read

Masked multi-head self-attention

Full sequenceTrainingPrefillDecode

Transformer block › Attention

Attention weights after dropout

×

tensorattn_weights · modeling_gpt2.py 67

Training dropout acts on the probabilities before value retrieval; at evaluation this equals A.

shape
A~iÃ_{i} head i [B batch, T queries, T keys]A~\widetilde A all heads [B batch, 12 heads, T queries, T keys]

Training only, in the maintained code: dropout zeroes some weights and rescales the rest, so a row need not sum to one (modeling_gpt2.py 67). At evaluation, and in the released OpenAI code, A~i=Ai\widetilde A_i=A_i.

Codemodeling_gpt2.py 63–69
modeling_gpt2.py 63–69
    attn_weights = nn.functional.softmax(attn_weights, dim=-1)

    # Downcast (if necessary) back to V's dtype (if in mixed-precision) -- No-Op otherwise
    attn_weights = attn_weights.type(value.dtype)
    attn_weights = nn.functional.dropout(attn_weights, p=dropout, training=module.training)

    attn_output = torch.matmul(attn_weights, value)
Sources1

Incoming transform evidence

The maintained eager attention path normalizes scores, applies training-dependent dropout, and multiplies the resulting weights by values.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 63–69

Read code excerpt
    attn_weights = nn.functional.softmax(attn_weights, dim=-1)

    # Downcast (if necessary) back to V's dtype (if in mixed-precision) -- No-Op otherwise
    attn_weights = attn_weights.type(value.dtype)
    attn_weights = nn.functional.dropout(attn_weights, p=dropout, training=module.training)

    attn_output = torch.matmul(attn_weights, value)