Initial residual stream
×tensorhidden_states = self.drop(hidden_states)
The sum of token and position embeddings, after training-only embedding dropout, entering the first block.
- shape
- [B batch, T positions, 768 d]
Sources
Incoming transform evidence
For the base GPT-2 text path, embedding dropout follows token-plus-position addition and precedes the block loop; optional token-type embeddings are outside the base text path.
transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 576–608
Read code excerpt
hidden_states = inputs_embeds + position_embeds.to(inputs_embeds.device)… lines 578–603 omitted …
hidden_states = self.drop(hidden_states)… lines 605–607 omitted …
for i, block in enumerate(self.h):Contextual evidence from Embedding addition
The original graph gathers the two tables and adds them into the initial stream.
openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 152–157
Read code excerpt
h = tf.gather(wte, X) + tf.gather(wpe, positions_for(X, past_length))Contextual evidence from Transformer block
The released block performs both pre-normalized residual writes and returns the updated stream.
openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 123–130
Read code excerpt
a, present = attn(norm(x, 'ln_1'), 'attn', nx, past=past, hparams=hparams)
x = x + a
m = mlp(norm(x, 'ln_2'), 'mlp', nx*4, hparams=hparams)
x = x + m
return x, presentContextual evidence from Transformer block
The maintained implementation preserves the same two residual branches.
transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 272–307
Read code excerpt
residual = hidden_states
hidden_states = self.ln_1(hidden_states)
attn_output, _ = self.attn(
hidden_states,
past_key_values=past_key_values,
attention_mask=attention_mask,
use_cache=use_cache,
**kwargs,
)… lines 281–281 omitted …
hidden_states = attn_output + residual… lines 283–302 omitted …
residual = hidden_states
hidden_states = self.ln_2(hidden_states)
feed_forward_hidden_states = self.mlp(hidden_states)
# residual connection
hidden_states = residual + feed_forward_hidden_states