Transformer · Dropout on the attention output

Encoder: 6 layers

Dropout on the attention output

no parameters

Training only, on the attention output, before the addition; P_drop = 0.1 in the base model.

\tilde{\Delta}_{\mathrm{attn}}=\frac{r\odot \Delta_{\mathrm{attn}}}{1-P_{\mathrm{drop}}}

r_i\sim\operatorname{Bernoulli}(1-P_{\mathrm{drop}})

in: $\Delta_{\mathrm{attn}}$ [B batch, S source positions, 512 d]
out: $\tilde{\Delta}_{\mathrm{attn}}$ [B batch, S source positions, 512 d]
parameters: none

Training only: in evaluation it returns its input unchanged.

Explanation

Acts between Attention output and Add attention output.

Dropout is training regularization: randomly removing contributions reduces reliance on particular features or attention connections. During training, elements are independently retained with probability $1-P_{\mathrm{drop}}$ and retained values are divided by $1-P_{\mathrm{drop}}$ . Evaluation returns the input unchanged.

Sources1

Direct evidence

Post-norm: each sublayer output is dropped out, added to its input and layer-normalized.

transformers · tensorflow/tensor2tensor / tensor2tensor/models/transformer.py
Pinned revision 0c66117d4297 ↗ · lines 59–61

Read code excerpt

    def residual_fn(x, y):
      return common_layers.layer_norm(x + tf.nn.dropout(
          y, 1.0 - hparams.residual_dropout))