GPT-2

All models, with their blocks →

Size
Published
OpenAI · Feb–Nov 2019
Line
Dense decoder
View
Architecture
Source

openai/gpt‑2 @ 9b63575, huggingface/transformers @ 71c6f69, configuration openai‑community/gpt2 @ 607a30d · checkpoint tensors not independently countedHow to read

Masked multi-head self-attention

Full sequenceTrainingPrefillDecode

Masked multi-head self-attention

Attention-weight dropout

×

no parameters

During training, drop and rescale attention probabilities before they mix the value vectors. Evaluation leaves them unchanged.

A~=r⊙A1−pattn\tilde{A}=\frac{r\odot A}{1-p_{\mathrm{attn}}}
ri∼Bernoulli⁡(1−pattn)r_i\sim\operatorname{Bernoulli}(1-p_{\mathrm{attn}})
in
AA [B batch, 12 heads, T queries, T keys]
out
A~\tilde{A} [B batch, 12 heads, T queries, T keys]
parameters
none

Training only: in evaluation it returns its input unchanged.

Explanation

Acts between Attention probabilities and Attention weights after dropout.

Dropout is training regularization: randomly removing contributions reduces reliance on particular features or attention connections. During training, elements are independently retained with probability 1−pattn1-p_{\mathrm{attn}} and retained values are divided by 1−pattn1-p_{\mathrm{attn}}. Evaluation returns the input unchanged.

This dropout is the maintained training implementation (Transformers). The released OpenAI graph implements inference with dropout omitted.

Sources2

Direct evidence

The maintained eager attention path normalizes scores, applies training-dependent dropout, and multiplies the resulting weights by values.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 63–69

Read code excerpt
    attn_weights = nn.functional.softmax(attn_weights, dim=-1)

    # Downcast (if necessary) back to V's dtype (if in mixed-precision) -- No-Op otherwise
    attn_weights = attn_weights.type(value.dtype)
    attn_weights = nn.functional.dropout(attn_weights, p=dropout, training=module.training)

    attn_output = torch.matmul(attn_weights, value)

Direct evidence

PyTorch documents dropout as training regularization, with retained-value rescaling and identity evaluation. This is general operation evidence; GPT-2 placement is sourced separately.

operation context · PyTorch 2.8 · pytorch/pytorch / torch/nn/modules/dropout.py
Pinned revision ba56102387ef ↗ · lines 35–70

Read code excerpt
class Dropout(_DropoutNd):
    r"""During training, randomly zeroes some of the elements of the input tensor with probability :attr:`p`.

    The zeroed elements are chosen independently for each forward call and are sampled from a Bernoulli distribution.

… lines 39–41 omitted …

    This has proven to be an effective technique for regularization and
    preventing the co-adaptation of neurons as described in the paper
    `Improving neural networks by preventing co-adaptation of feature
    detectors`_ .

    Furthermore, the outputs are scaled by a factor of :math:`\frac{1}{1-p}` during
    training. This means that during evaluation the module simply computes an
    identity function.

… lines 50–68 omitted …

    def forward(self, input: Tensor) -> Tensor:
        return F.dropout(input, self.p, self.training, self.inplace)