GPT-2 Small · MLP residual dropout

Transformer block

MLP residual dropout

no parameters

During training, drop and rescale the MLP update before its residual addition. The dropout belongs to the MLP module (dropout) and acts on its output. Evaluation leaves the update unchanged.

\tilde{\Delta}_{\mathrm{mlp}}=\frac{r\odot \Delta_{\mathrm{mlp}}}{1-p_{\mathrm{resid}}}

r_i\sim\operatorname{Bernoulli}(1-p_{\mathrm{resid}})

in: $\Delta_{\mathrm{mlp}}$ [B batch, T positions, 768 d]
out: $\tilde{\Delta}_{\mathrm{mlp}}$ [B batch, T positions, 768 d]
parameters: none

Training only: in evaluation it returns its input unchanged.

Explanation

Acts between Position-wise feed-forward network and MLP residual merge.

Dropout is training regularization: randomly removing contributions reduces reliance on particular features or attention connections. During training, elements are independently retained with probability $1-p_{\mathrm{resid}}$ and retained values are divided by $1-p_{\mathrm{resid}}$ . Evaluation returns the input unchanged.

This dropout is the maintained training implementation (Transformers). The released OpenAI graph implements inference with dropout omitted.

Sources2

Direct evidence

The maintained training path applies residual dropout after the MLP projection and before the residual write; evaluation makes dropout the identity.

transformers · huggingface/transformers / src/transformers/models/gpt2/modeling_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 238–243

Read code excerpt

        hidden_states = self.c_proj(hidden_states)
        hidden_states = self.dropout(hidden_states)
        return hidden_states

Direct evidence

PyTorch documents dropout as training regularization, with retained-value rescaling and identity evaluation. This is general operation evidence; GPT-2 placement is sourced separately.

operation context · PyTorch 2.8 · pytorch/pytorch / torch/nn/modules/dropout.py
Pinned revision ba56102387ef ↗ · lines 35–70

Read code excerpt

class Dropout(_DropoutNd):
    r"""During training, randomly zeroes some of the elements of the input tensor with probability :attr:`p`.

    The zeroed elements are chosen independently for each forward call and are sampled from a Bernoulli distribution.

… lines 39–41 omitted …

    This has proven to be an effective technique for regularization and
    preventing the co-adaptation of neurons as described in the paper
    `Improving neural networks by preventing co-adaptation of feature
    detectors`_ .

    Furthermore, the outputs are scaled by a factor of :math:`\frac{1}{1-p}` during
    training. This means that during evaluation the module simply computes an
    identity function.

… lines 50–68 omitted …

    def forward(self, input: Tensor) -> Tensor:
        return F.dropout(input, self.p, self.training, self.inplace)