GPT-2

All models, with their blocks →

Size
Published
OpenAI · Feb–Nov 2019
Line
Dense decoder
View
Architecture
Source

openai/gpt‑2 @ 9b63575, huggingface/transformers @ 71c6f69, configuration openai‑community/gpt2 @ 607a30d · checkpoint tensors not independently countedHow to read

Prepare Q, K, V

Full sequenceTrainingPrefillDecode

Transformer block › Attention

Fused Q, K, V projection

×

learnedconv1d(x, 'c_attn', n_state*3)

One learned projection produces a fused 2,304-feature field; splitting and head reshaping follow as separate operations.

qkv=xWQKV+bQKV\mathrm{qkv}={\htmlData{light=attn-input}{x}}W_{\mathrm{QKV}}+b_{\mathrm{QKV}}
in
xx [B batch, T positions, 768 d]
out
qkv\mathrm{qkv} [B batch, T positions, 2304 q, k, v]
parameters
1,771,776 · × 12 blocks = 21,261,312
Parameters

1,771,776 parameters per instance.

Exact per-instance accounting
ComponentCalculationParameters
Weight matrix768×2,304768 \times 2{,}3041,769,472
Output bias2,3042{,}3042,304
Per-instance total1,771,776

The reference module holds the weight input by output, [in, out].

Across 12 independent blocks: 1,771,776×12=21,261,3121{,}771{,}776 \times 12 = 21{,}261{,}312 parameters.

Derived from the configured matrix dimensions; checkpoint tensors have not been independently counted.

Explanation

One matrix product computes all three projections: WQKV=[WQ  WK  WV]W_{\mathrm{QKV}}=[W_Q\;W_K\;W_V] and bQKV=[bQ  bK  bV]b_{\mathrm{QKV}}=[b_Q\;b_K\;b_V], and the split takes queries, keys and values in that order (model.py 102–103).

Codemodel.py 50–56, 102–103 · modeling_gpt2.py 185–191
model.py 50–56
def conv1d(x, scope, nf, *, w_init_stdev=0.02):
    with tf.variable_scope(scope):
        *start, nx = shape_list(x)
        w = tf.get_variable('w', [1, nx, nf], initializer=tf.random_normal_initializer(stddev=w_init_stdev))
        b = tf.get_variable('b', [nf], initializer=tf.constant_initializer(0))
        c = tf.reshape(tf.matmul(tf.reshape(x, [-1, nx]), tf.reshape(w, [-1, nf]))+b, start+[nf])
        return c
model.py 102–103
        c = conv1d(x, 'c_attn', n_state*3)
        q, k, v = map(split_heads, tf.split(c, 3, axis=2))
modeling_gpt2.py 185–191
            query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
            shape_kv = (*key_states.shape[:-1], -1, self.head_dim)
            key_states = key_states.view(shape_kv).transpose(1, 2)
            value_states = value_states.view(shape_kv).transpose(1, 2)

        shape_q = (*query_states.shape[:-1], -1, self.head_dim)
        query_states = query_states.view(shape_q).transpose(1, 2)
Sources3

Direct evidence

The released projection helper owns a learned matrix and output bias and applies them to the last feature axis.

linear-operation · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 50–56

Read code excerpt
def conv1d(x, scope, nf, *, w_init_stdev=0.02):
    with tf.variable_scope(scope):
        *start, nx = shape_list(x)
        w = tf.get_variable('w', [1, nx, nf], initializer=tf.random_normal_initializer(stddev=w_init_stdev))
        b = tf.get_variable('b', [nf], initializer=tf.constant_initializer(0))
        c = tf.reshape(tf.matmul(tf.reshape(x, [-1, nx]), tf.reshape(w, [-1, nf]))+b, start+[nf])
        return c

Direct evidence

The released attention module creates one 3× projection and splits it into Q, K, and V head fields.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 101–104

Read code excerpt
        c = conv1d(x, 'c_attn', n_state*3)
        q, k, v = map(split_heads, tf.split(c, 3, axis=2))

Direct evidence

The maintained implementation applies c_attn once, splits the three fields, and reshapes each to heads.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 185–191

Read code excerpt
            query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
            shape_kv = (*key_states.shape[:-1], -1, self.head_dim)
            key_states = key_states.view(shape_kv).transpose(1, 2)
            value_states = value_states.view(shape_kv).transpose(1, 2)

        shape_q = (*query_states.shape[:-1], -1, self.head_dim)
        query_states = query_states.view(shape_q).transpose(1, 2)