Transformer block › MLP
Normalized MLP input
×tensornorm(x, 'ln_2')
The output of LayerNorm 2 in the containing transformer block.
- shape
- [B batch, T positions, 768 d]
Sources
Contextual evidence from Pre-MLP LayerNorm
The second normalization appears directly inside the MLP call.
openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 28–128
Read code excerpt
def norm(x, scope, *, axis=-1, epsilon=1e-5):
"""Normalize to mean = 0, std = 1, then do a diagonal affine transform."""
with tf.variable_scope(scope):
n_state = x.shape[-1].value
g = tf.get_variable('g', [n_state], initializer=tf.constant_initializer(1))
b = tf.get_variable('b', [n_state], initializer=tf.constant_initializer(0))
u = tf.reduce_mean(x, axis=axis, keepdims=True)
s = tf.reduce_mean(tf.square(x-u), axis=axis, keepdims=True)
x = (x - u) * tf.rsqrt(s + epsilon)
x = x*g + b
return x… lines 39–127 omitted …
m = mlp(norm(x, 'ln_2'), 'mlp', nx*4, hparams=hparams)Contextual evidence from Pre-MLP LayerNorm
A second nn.LayerNorm instance precedes the MLP.
transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 254–305
Read code excerpt
self.ln_2 = nn.LayerNorm(hidden_size, eps=config.layer_norm_epsilon)… lines 255–302 omitted …
residual = hidden_states
hidden_states = self.ln_2(hidden_states)
feed_forward_hidden_states = self.mlp(hidden_states)Contextual evidence from Feature expansion
The released projection helper owns a learned matrix and output bias and applies them to the last feature axis.
linear-operation · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 50–56
Read code excerpt
def conv1d(x, scope, nf, *, w_init_stdev=0.02):
with tf.variable_scope(scope):
*start, nx = shape_list(x)
w = tf.get_variable('w', [1, nx, nf], initializer=tf.random_normal_initializer(stddev=w_init_stdev))
b = tf.get_variable('b', [nf], initializer=tf.constant_initializer(0))
c = tf.reshape(tf.matmul(tf.reshape(x, [-1, nx]), tf.reshape(w, [-1, nf]))+b, start+[nf])
return cContextual evidence from Feature expansion
The MLP's c_fc projection opens the four-times-wider feature workspace before GELU.
openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 115–128
Read code excerpt
def mlp(x, scope, n_state, *, hparams):
with tf.variable_scope(scope):
nx = x.shape[-1].value
h = gelu(conv1d(x, 'c_fc', n_state))… lines 119–127 omitted …
m = mlp(norm(x, 'ln_2'), 'mlp', nx*4, hparams=hparams)Contextual evidence from Feature expansion
The maintained MLP defines c_fc at the configured intermediate size and applies it before activation.
transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 232–260
Read code excerpt
embed_dim = config.hidden_size
self.c_fc = Conv1D(intermediate_size, embed_dim)
self.c_proj = Conv1D(embed_dim, intermediate_size)
self.act = ACT2FN[config.activation_function]
self.dropout = nn.Dropout(config.resid_pdrop)
def forward(self, hidden_states: tuple[torch.FloatTensor] | None) -> torch.FloatTensor:
hidden_states = self.c_fc(hidden_states)
hidden_states = self.act(hidden_states)
hidden_states = self.c_proj(hidden_states)
hidden_states = self.dropout(hidden_states)
return hidden_states… lines 244–248 omitted …
hidden_size = config.hidden_size
inner_dim = config.n_inner if config.n_inner is not None else 4 * hidden_size… lines 251–259 omitted …
self.mlp = GPT2MLP(inner_dim, config)