GPT-2

All models, with previews →

OpenAI · Feb–Nov 2019Dense decoder

Reference code openai/gpt-2 @ 9b63575, huggingface/transformers @ 71c6f69, configuration openai-community/gpt2 @ 607a30d · checkpoint tensors not independently counted · How to read

Shared WTERaw textTokenizerByte-level BPEvocab · mergesfixed assetsToken IDs[B, T]Token embeddingWTE\mathrm{WTE}50,257 × 768Et[B, T, 768]Position IDs0,…,T−10,\ldots,T-1Position embeddingWPE\mathrm{WPE}1,024 × 768+Dropoutpembdp_{\mathrm{embd}}x0[B, T, 768]
12 independent
blocksOpen →
x12[B, T, 768]Final LayerNormμ,  σ2\mu,\;\sigma^2gain · biasTied projectionhfinalWTETh_{\mathrm{final}}WTE^{\mathsf T}0 new parametersLogits[B, T, 50257]

Tied vocabulary projection

×

reuses weightstf.matmul(h_flat, wte, transpose_b=True)

Project each normalized feature vector into vocabulary logits.

logits=hfinalWtokenT\mathrm{logits}={\htmlData{light=ln-f}{h_{\mathrm{final}}}}W_{\mathrm{token}}^{\mathsf T}
in
hfinalh_{\mathrm{final}} [B batch, T positions, 768 d]
out
logits\mathrm{logits} [B batch, T positions, 50257 vocabulary]
Parameters

Reuses Token embedding: weights are counted there, with no additional parameter ownership here.

Explanation

Each entry scores one vocabulary token as the next token for its position. The softmax and the choice of token happen outside the model.

Each vocabulary row of WTE serves both the input lookup and, transposed, the output scoring.

one tablein: pick a rowout: score againstevery row
Schematic of WTE’s input lookup and output scoring uses. No quantitative scale is encoded.
Codemodel.py 170–172 · modeling_gpt2.py 638–698
model.py 170–172
        h_flat = tf.reshape(h, [batch*sequence, hparams.n_embd])
        logits = tf.matmul(h_flat, wte, transpose_b=True)
        logits = tf.reshape(logits, [batch, sequence, hparams.n_vocab])
modeling_gpt2.py 638
    _tied_weights_keys = {"lm_head.weight": "transformer.wte.weight"}
modeling_gpt2.py 642–643
        self.transformer = GPT2Model(config)
        self.lm_head = nn.Linear(config.n_embd, config.vocab_size, bias=False)
modeling_gpt2.py 695
        hidden_states = transformer_outputs.last_hidden_state
modeling_gpt2.py 698
        logits = self.lm_head(hidden_states[:, slice_indices, :])
Sources2

Direct evidence

The original graph explicitly multiplies by wteᵀ.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 169–172

Read code excerpt
        h_flat = tf.reshape(h, [batch*sequence, hparams.n_embd])
        logits = tf.matmul(h_flat, wte, transpose_b=True)
        logits = tf.reshape(logits, [batch, sequence, hparams.n_vocab])

Direct evidence

The maintained model declares lm_head.weight tied to transformer.wte.weight and applies the vocabulary projection without a bias.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 637–698

Read code excerpt
    _tied_weights_keys = {"lm_head.weight": "transformer.wte.weight"}

… lines 639–641 omitted …

        self.transformer = GPT2Model(config)
        self.lm_head = nn.Linear(config.n_embd, config.vocab_size, bias=False)

… lines 644–694 omitted …

        hidden_states = transformer_outputs.last_hidden_state

… lines 696–697 omitted …

        logits = self.lm_head(hidden_states[:, slice_indices, :])