LDLM · Frozen GPT-2 · OpenModelAtlas

GPT-2 Small

Frozen GPT-2

frozen · not trained

GPT-2 Small, frozen, with two token rows added, for [PAD] and <mask>; the hidden state at index -3 of its 13 is taken.

{\htmlData{light=raw gpt2}{h^{\mathrm{raw}}}}=\operatorname{GPT\text{-}2}({\htmlData{light=ids}{I}})_{[-3]}

in: $I$ [B batch, T positions]
out: $h^{\mathrm{raw}}$ [B batch, T positions, 768 d]
parameters: 124,441,344

The model is loaded by name, gpt2, with no revision of its weights pinned. In the transformers library’s list of GPT-2’s hidden states the first entry is the embedding’s output, the last is the last block’s output after the final LayerNorm, and the one before it the output of block 11; so index −3 is the output of block 10, and blocks 11 and 12 and the final LayerNorm are owned and loaded while what they compute is not used. The implementation pins transformers 4.48.0; no excerpt of that library is recorded here.

Parameters

124,441,344 parameters owned by this component

Exact per-instance accounting
Component	Calculation	Parameters
table	$50{,}259 \times 768$	38,598,912
table	$1{,}024 \times 768$	786,432
gain + bias	$768 + 768$	1,536
weights + bias	$768 \times 2{,}304 + 2{,}304$	1,771,776
weights + bias	$768 \times 768 + 768$	590,592
gain + bias	$768 + 768$	1,536
weights + bias	$768 \times 3{,}072 + 3{,}072$	2,362,368
weights + bias	$3{,}072 \times 768 + 768$	2,360,064
gain + bias	$768 + 768$	1,536
weights + bias	$768 \times 2{,}304 + 2{,}304$	1,771,776
weights + bias	$768 \times 768 + 768$	590,592
gain + bias	$768 + 768$	1,536
weights + bias	$768 \times 3{,}072 + 3{,}072$	2,362,368
weights + bias	$3{,}072 \times 768 + 768$	2,360,064
gain + bias	$768 + 768$	1,536
weights + bias	$768 \times 2{,}304 + 2{,}304$	1,771,776
weights + bias	$768 \times 768 + 768$	590,592
gain + bias	$768 + 768$	1,536
weights + bias	$768 \times 3{,}072 + 3{,}072$	2,362,368
weights + bias	$3{,}072 \times 768 + 768$	2,360,064
gain + bias	$768 + 768$	1,536
weights + bias	$768 \times 2{,}304 + 2{,}304$	1,771,776
weights + bias	$768 \times 768 + 768$	590,592
gain + bias	$768 + 768$	1,536
weights + bias	$768 \times 3{,}072 + 3{,}072$	2,362,368
weights + bias	$3{,}072 \times 768 + 768$	2,360,064
gain + bias	$768 + 768$	1,536
weights + bias	$768 \times 2{,}304 + 2{,}304$	1,771,776
weights + bias	$768 \times 768 + 768$	590,592
gain + bias	$768 + 768$	1,536
weights + bias	$768 \times 3{,}072 + 3{,}072$	2,362,368
weights + bias	$3{,}072 \times 768 + 768$	2,360,064
gain + bias	$768 + 768$	1,536
weights + bias	$768 \times 2{,}304 + 2{,}304$	1,771,776
weights + bias	$768 \times 768 + 768$	590,592
gain + bias	$768 + 768$	1,536
weights + bias	$768 \times 3{,}072 + 3{,}072$	2,362,368
weights + bias	$3{,}072 \times 768 + 768$	2,360,064
gain + bias	$768 + 768$	1,536
weights + bias	$768 \times 2{,}304 + 2{,}304$	1,771,776
weights + bias	$768 \times 768 + 768$	590,592
gain + bias	$768 + 768$	1,536
weights + bias	$768 \times 3{,}072 + 3{,}072$	2,362,368
weights + bias	$3{,}072 \times 768 + 768$	2,360,064
gain + bias	$768 + 768$	1,536
weights + bias	$768 \times 2{,}304 + 2{,}304$	1,771,776
weights + bias	$768 \times 768 + 768$	590,592
gain + bias	$768 + 768$	1,536
weights + bias	$768 \times 3{,}072 + 3{,}072$	2,362,368
weights + bias	$3{,}072 \times 768 + 768$	2,360,064
gain + bias	$768 + 768$	1,536
weights + bias	$768 \times 2{,}304 + 2{,}304$	1,771,776
weights + bias	$768 \times 768 + 768$	590,592
gain + bias	$768 + 768$	1,536
weights + bias	$768 \times 3{,}072 + 3{,}072$	2,362,368
weights + bias	$3{,}072 \times 768 + 768$	2,360,064
gain + bias	$768 + 768$	1,536
weights + bias	$768 \times 2{,}304 + 2{,}304$	1,771,776
weights + bias	$768 \times 768 + 768$	590,592
gain + bias	$768 + 768$	1,536
weights + bias	$768 \times 3{,}072 + 3{,}072$	2,362,368
weights + bias	$3{,}072 \times 768 + 768$	2,360,064
gain + bias	$768 + 768$	1,536
weights + bias	$768 \times 2{,}304 + 2{,}304$	1,771,776
weights + bias	$768 \times 768 + 768$	590,592
gain + bias	$768 + 768$	1,536
weights + bias	$768 \times 3{,}072 + 3{,}072$	2,362,368
weights + bias	$3{,}072 \times 768 + 768$	2,360,064
gain + bias	$768 + 768$	1,536
weights + bias	$768 \times 2{,}304 + 2{,}304$	1,771,776
weights + bias	$768 \times 768 + 768$	590,592
gain + bias	$768 + 768$	1,536
weights + bias	$768 \times 3{,}072 + 3{,}072$	2,362,368
weights + bias	$3{,}072 \times 768 + 768$	2,360,064
gain + bias	$768 + 768$	1,536
Per-instance total		124,441,344

Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.

124,441,344 parameters, none of them updated: a token table of 50,259 rows, a position table of 1,024, 12 blocks and a final LayerNorm.

Explanation

GPT-2’s own architecture is drawn in the atlas under GPT-2 Small. It is causal: the feature at a position depends on that position and the ones before it.

Sources6

Direct evidence

The feature model is wrapped to return one entry of its hidden states, chosen by index; GPT-2’s token table is resized to the configured vocabulary.