Transformer block › Attention
Merged attention values
×tensora · 111
Concatenating all 12 head outputs restores the 768-feature axis before output projection.
- shape
- [B batch, T positions, 768 d]
Sources
Contextual evidence from Merge attention heads
The released merge_heads function reverses the head permutation and merges the final two axes.
openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 79–81
Read code excerpt
def merge_heads(x):… lines 80–80 omitted …
return merge_states(tf.transpose(x, [0, 2, 1, 3]))Contextual evidence from Attention output projection
The released projection helper owns a learned matrix and output bias and applies them to the last feature axis.
linear-operation · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 50–56
Read code excerpt
def conv1d(x, scope, nf, *, w_init_stdev=0.02):
with tf.variable_scope(scope):
*start, nx = shape_list(x)
w = tf.get_variable('w', [1, nx, nf], initializer=tf.random_normal_initializer(stddev=w_init_stdev))
b = tf.get_variable('b', [nf], initializer=tf.constant_initializer(0))
c = tf.reshape(tf.matmul(tf.reshape(x, [-1, nx]), tf.reshape(w, [-1, nf]))+b, start+[nf])
return cContextual evidence from Attention output projection
The attention output merges its head axis and passes through c_proj before returning to the residual stream.
openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 109–112
Read code excerpt
a = multihead_attn(q, k, v)
a = merge_heads(a)
a = conv1d(a, 'c_proj', n_state)
return a, presentContextual evidence from Attention output projection
The maintained attention path merges heads, applies c_proj, and returns the residual update.
transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 222–226
Read code excerpt
attn_output = attn_output.reshape(*attn_output.shape[:-2], -1).contiguous()
attn_output = self.c_proj(attn_output)
attn_output = self.resid_dropout(attn_output)
return attn_output, attn_weights