GPT-2

All models, with previews →

OpenAI · Feb–Nov 2019Dense decoder

Reference code openai/gpt-2 @ 9b63575, huggingface/transformers @ 71c6f69, configuration openai-community/gpt2 @ 607a30d · checkpoint tensors not independently counted · How to read

Shared WTERaw textTokenizerByte-level BPEvocab · mergesfixed assetsToken IDs[B, T]Token embeddingWTE\mathrm{WTE}50,257 × 768Et[B, T, 768]Position IDs0,…,T−10,\ldots,T-1Position embeddingWPE\mathrm{WPE}1,024 × 768+Dropoutpembdp_{\mathrm{embd}}x0[B, T, 768]
12 independent
blocksOpen →
x12[B, T, 768]Final LayerNormμ,  σ2\mu,\;\sigma^2gain · biasTied projectionhfinalWTETh_{\mathrm{final}}WTE^{\mathsf T}0 new parametersLogits[B, T, 50257]

Logits

×

tensorlogits

Unnormalized vocabulary scores for the token following each sequence position.

shape
Logits\mathrm{Logits} [B batch, T positions, 50257 vocabulary]
Sources

Contextual evidence from Tied vocabulary projection

The original graph explicitly multiplies by wteᵀ.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 169–172

Read code excerpt
        h_flat = tf.reshape(h, [batch*sequence, hparams.n_embd])
        logits = tf.matmul(h_flat, wte, transpose_b=True)
        logits = tf.reshape(logits, [batch, sequence, hparams.n_vocab])

Contextual evidence from Tied vocabulary projection

The maintained model declares lm_head.weight tied to transformer.wte.weight and applies the vocabulary projection without a bias.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 637–698

Read code excerpt
    _tied_weights_keys = {"lm_head.weight": "transformer.wte.weight"}

… lines 639–641 omitted …

        self.transformer = GPT2Model(config)
        self.lm_head = nn.Linear(config.n_embd, config.vocab_size, bias=False)

… lines 644–694 omitted …

        hidden_states = transformer_outputs.last_hidden_state

… lines 696–697 omitted …

        logits = self.lm_head(hidden_states[:, slice_indices, :])