Transformer block › Attention
Values by head
×tensorv
The feature axis is partitioned into 12 heads of width 64; positions remain separate.
- shape
- head i [B batch, T positions, 64 d_h] all heads [B batch, 12 heads, T positions, 64 d_h]
Sources
Contextual evidence from Reshape into attention heads
The released split_heads function introduces a head axis and moves it before the position axis.
openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 75–79
Read code excerpt
def split_heads(x):… lines 76–76 omitted …
return tf.transpose(split_states(x, hparams.n_head), [0, 2, 1, 3])Contextual evidence from Weighted value retrieval
The maintained eager attention path normalizes scores, applies training-dependent dropout, and multiplies the resulting weights by values.
transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 63–69
Read code excerpt
attn_weights = nn.functional.softmax(attn_weights, dim=-1)
# Downcast (if necessary) back to V's dtype (if in mixed-precision) -- No-Op otherwise
attn_weights = attn_weights.type(value.dtype)
attn_weights = nn.functional.dropout(attn_weights, p=dropout, training=module.training)
attn_output = torch.matmul(attn_weights, value)Contextual evidence from Weighted value retrieval
The normalized score matrix multiplies the value field inside each head.
openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 91–99
Read code excerpt
w = mask_attn_weights(w)
w = softmax(w)
a = tf.matmul(w, v)
return a