Transformer

All models, with their blocks →

Size
Published
Vaswani et al. · Jun 2017
Line
Encoder–decoder
View
Architecture
Source

tensorflow/tensor2tensor v1.0.14 @ 0c66117 · arXiv:1706.03762v7 · WMT 2014 English–German · vocabulary of about 37,000 · no trained weights were publishedHow to read

Decoder: 6 layers

Decoder: 6 layers

Dropout on the self-attention output

×

no parameters

Training only, on the self-attention output, before the addition; P_drop = 0.1 in the base model.

Δ~self=r⊙Δself1−Pdrop\tilde{\Delta}_{\mathrm{self}}=\frac{r\odot \Delta_{\mathrm{self}}}{1-P_{\mathrm{drop}}}
ri∼Bernoulli⁡(1−Pdrop)r_i\sim\operatorname{Bernoulli}(1-P_{\mathrm{drop}})
in
Δself\Delta_{\mathrm{self}} [B batch, T target positions, 512 d]
out
Δ~self\tilde{\Delta}_{\mathrm{self}} [B batch, T target positions, 512 d]
parameters
none

Training only: in evaluation it returns its input unchanged.

Explanation

Acts between Self-attention output and Add self-attention output.

Dropout is training regularization: randomly removing contributions reduces reliance on particular features or attention connections. During training, elements are independently retained with probability 1−Pdrop1-P_{\mathrm{drop}} and retained values are divided by 1−Pdrop1-P_{\mathrm{drop}}. Evaluation returns the input unchanged.

Sources1

Direct evidence

Post-norm: each sublayer output is dropped out, added to its input and layer-normalized.

transformers · tensorflow/tensor2tensor / tensor2tensor/models/transformer.py
Pinned revision 0c66117d4297 ↗ · lines 59–61

Read code excerpt
    def residual_fn(x, y):
      return common_layers.layer_norm(x + tf.nn.dropout(
          y, 1.0 - hparams.residual_dropout))