GPT-2

All models, with previews →

OpenAI · Feb–Nov 2019Dense decoder

Reference code openai/gpt-2 @ 9b63575, huggingface/transformers @ 71c6f69, configuration openai-community/gpt2 @ 607a30d · checkpoint tensors not independently counted · How to read

Shared WTERaw textTokenizerByte-level BPEvocab · mergesfixed assetsToken IDs[B, T]Token embeddingWTE\mathrm{WTE}50,257 × 768Et[B, T, 768]Position IDs0,…,T−10,\ldots,T-1Position embeddingWPE\mathrm{WPE}1,024 × 768+Dropoutpembdp_{\mathrm{embd}}x0[B, T, 768]
12 independent
blocksOpen →
x12[B, T, 768]Final LayerNormμ,  σ2\mu,\;\sigma^2gain · biasTied projectionhfinalWTETh_{\mathrm{final}}WTE^{\mathsf T}0 new parametersLogits[B, T, 50257]

Token embedding vectors

×

tensortf.gather(wte, X)

Rows selected from WTE, before adding the learned position embeddings.

shape
EtE_{t} [B batch, T positions, 768 d]
Sources

Contextual evidence from Token embedding

The original TensorFlow graph creates both tables and adds their gathered rows.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 152–157

Read code excerpt
        wpe = tf.get_variable('wpe', [hparams.n_ctx, hparams.n_embd],

… lines 153–153 omitted …

        wte = tf.get_variable('wte', [hparams.n_vocab, hparams.n_embd],

… lines 155–156 omitted …

        h = tf.gather(wte, X) + tf.gather(wpe, positions_for(X, past_length))

Contextual evidence from Token embedding

Transformers names the tables wte and wpe, matching the released checkpoint keys.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 492–577

Read code excerpt
        self.wte = nn.Embedding(config.vocab_size, self.embed_dim)
        self.wpe = nn.Embedding(config.max_position_embeddings, self.embed_dim)

… lines 494–568 omitted …

            inputs_embeds = self.wte(input_ids)

… lines 570–575 omitted …

        position_embeds = self.wpe(position_ids)
        hidden_states = inputs_embeds + position_embeds.to(inputs_embeds.device)

Contextual evidence from Embedding addition

For the base GPT-2 text path, embedding dropout follows token-plus-position addition and precedes the block loop; optional token-type embeddings are outside the base text path.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 576–608

Read code excerpt
        hidden_states = inputs_embeds + position_embeds.to(inputs_embeds.device)

… lines 578–603 omitted …

        hidden_states = self.drop(hidden_states)

… lines 605–607 omitted …

        for i, block in enumerate(self.h):

Contextual evidence from Embedding addition

The original graph gathers the two tables and adds them into the initial stream.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 152–157

Read code excerpt
        h = tf.gather(wte, X) + tf.gather(wpe, positions_for(X, past_length))