Transformer block › Attention
Normalized attention input
×tensornorm(x, 'ln_1')
The output of LayerNorm 1 in the containing transformer block.
- shape
- [B batch, 1, 768 d]
Sources
Contextual evidence from Pre-attention LayerNorm
Original LayerNorm implementation and ε = 10⁻⁵.
openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 28–38
Read code excerpt
def norm(x, scope, *, axis=-1, epsilon=1e-5):
"""Normalize to mean = 0, std = 1, then do a diagonal affine transform."""
with tf.variable_scope(scope):
n_state = x.shape[-1].value
g = tf.get_variable('g', [n_state], initializer=tf.constant_initializer(1))
b = tf.get_variable('b', [n_state], initializer=tf.constant_initializer(0))
u = tf.reduce_mean(x, axis=axis, keepdims=True)
s = tf.reduce_mean(tf.square(x-u), axis=axis, keepdims=True)
x = (x - u) * tf.rsqrt(s + epsilon)
x = x*g + b
return xContextual evidence from Pre-attention LayerNorm
PyTorch delegates the operation to nn.LayerNorm with the checkpoint epsilon.
transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 252–273
Read code excerpt
self.ln_1 = nn.LayerNorm(hidden_size, eps=config.layer_norm_epsilon)… lines 253–271 omitted …
residual = hidden_states
hidden_states = self.ln_1(hidden_states)Contextual evidence from Fused Q, K, V projection
The released projection helper owns a learned matrix and output bias and applies them to the last feature axis.
linear-operation · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 50–56
Read code excerpt
def conv1d(x, scope, nf, *, w_init_stdev=0.02):
with tf.variable_scope(scope):
*start, nx = shape_list(x)
w = tf.get_variable('w', [1, nx, nf], initializer=tf.random_normal_initializer(stddev=w_init_stdev))
b = tf.get_variable('b', [nf], initializer=tf.constant_initializer(0))
c = tf.reshape(tf.matmul(tf.reshape(x, [-1, nx]), tf.reshape(w, [-1, nf]))+b, start+[nf])
return cContextual evidence from Fused Q, K, V projection
The released attention module creates one 3× projection and splits it into Q, K, and V head fields.
openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 101–104
Read code excerpt
c = conv1d(x, 'c_attn', n_state*3)
q, k, v = map(split_heads, tf.split(c, 3, axis=2))Contextual evidence from Fused Q, K, V projection
The maintained implementation applies c_attn once, splits the three fields, and reshapes each to heads.
transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 185–191
Read code excerpt
query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
shape_kv = (*key_states.shape[:-1], -1, self.head_dim)
key_states = key_states.view(shape_kv).transpose(1, 2)
value_states = value_states.view(shape_kv).transpose(1, 2)
shape_q = (*query_states.shape[:-1], -1, self.head_dim)
query_states = query_states.view(shape_q).transpose(1, 2)