GPT-2

All models, with previews →

OpenAI · Feb–Nov 2019Dense decoder

Reference code openai/gpt-2 @ 9b63575, huggingface/transformers @ 71c6f69, configuration openai-community/gpt2 @ 607a30d · checkpoint tensors not independently counted · How to read

Shared WTERaw textTokenizerByte-level BPEvocab · mergesfixed assetsToken IDs[B, T]Token embeddingWTE\mathrm{WTE}50,257 × 768Et[B, T, 768]Position IDs0,…,T−10,\ldots,T-1Position embeddingWPE\mathrm{WPE}1,024 × 768+Dropoutpembdp_{\mathrm{embd}}x0[B, T, 768]
12 independent
blocksOpen →
x12[B, T, 768]Final LayerNormμ,  σ2\mu,\;\sigma^2gain · biasTied projectionhfinalWTETh_{\mathrm{final}}WTE^{\mathsf T}0 new parametersLogits[B, T, 50257]
in
Raw text
out
LL [B, T, 50257]
parameters
124,439,808 in all
Parameters

124,439,808 parameters in GPT-2 Small

Exact model ownership reconciliation
ComponentCalculationParameters
Token embedding50,257×76850{,}257 \times 76838,597,376
Position embedding table1,024×7681{,}024 \times 768786,432
Transformer block7,087,872×127{,}087{,}872 \times 1285,054,464
Final LayerNorm768+768768 + 7681,536
Model total124,439,808

Derived exact parameter total: 124,439,808. The sum above counts the token embedding once; the tied vocabulary projection adds no new ownership. Sources contains the pinned configuration and implementation used for this derivation; checkpoint tensors are not independently counted.

In numbers

Fact sheet

Blocks
12, all denseconfig.json
Width
768config.json
Attention
12 heads, each with its own keys and values; 64 features per headconfig.jsonmodel.py 102–108
Feed-forward
GELU MLP, 3,072 hidden features (4 × width)config.jsonmodel.py 115–128
Vocabulary
50,257config.json
Context
1,024 positions, the learned position tableconfig.json
Parameters
124,439,808, all used per tokenconfig.json
KV cache
72 KiB per position, all layers12 heads × 64 × 2 (K, V) × 12 layers × 4 bytes. Written in Prefill and Decode; the Full sequence view draws none.model.py 102–108model.py 152–155

Weights · whole model

GPT-2 small · float32474.7 MiB
Tied embedding · float32147.2 MiB
With untied readout · float32621.9 MiB

124,439,808 parameters × 4 bytes. The token embedding holds 50,257 × 768 = 38,597,376 of them (31.0%); the readout multiplies by its transpose, so it owns none.

A separate readout of the same shape would add 38,597,376 parameters, 147.2 MiB.

config.jsonmodel.py 154–155model.py 171
Sources

Direct evidence

The released forward pass provides contextual evidence for input IDs, embedding states, the independently addressed block loop, final normalization, and tied vocabulary logits.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 147–174

Read code excerpt
        h = tf.gather(wte, X) + tf.gather(wpe, positions_for(X, past_length))

… lines 158–162 omitted …

        for layer, past in enumerate(pasts):
            h, present = block(h, 'h%d' % layer, past=past, hparams=hparams)

… lines 165–166 omitted …

        h = norm(h, 'ln_f')

… lines 168–170 omitted …

        logits = tf.matmul(h_flat, wte, transpose_b=True)

Direct evidence

The Small configuration supports a derived exact total of 124,439,808 parameters, with the token embedding counted once. 124M is a rounded size label; checkpoint tensors have not been independently counted.

configuration for derived total · openai-community/gpt2 / config.json
Pinned revision 607a30d783df ↗ · $