Transformer

All models, with their blocks →

Size
Published
Vaswani et al. · Jun 2017
Line
Encoder–decoder
View
Architecture
Source

tensorflow/tensor2tensor v1.0.14 @ 0c66117 · arXiv:1706.03762v7 · WMT 2014 English–German · vocabulary of about 37,000 · no trained weights were publishedHow to read

Transformer

Dropout on the decoder input

×

no parameters

Training only, on the decoder input; P_drop = 0.1 in the base model.

y~0=r⊙y01−Pdrop\tilde{y}_0=\frac{r\odot y_0}{1-P_{\mathrm{drop}}}
ri∼Bernoulli⁡(1−Pdrop)r_i\sim\operatorname{Bernoulli}(1-P_{\mathrm{drop}})
in
y0y_0 [B batch, T target positions, 512 d]
out
y~0\tilde{y}_0 [B batch, T target positions, 512 d]
parameters
none

Training only: in evaluation it returns its input unchanged.

Explanation

Acts between Decoder input and Decoder: 6 layers.

Dropout is training regularization: randomly removing contributions reduces reliance on particular features or attention connections. During training, elements are independently retained with probability 1−Pdrop1-P_{\mathrm{drop}} and retained values are divided by 1−Pdrop1-P_{\mathrm{drop}}. Evaluation returns the input unchanged.

Sources1

Direct evidence

Dropout with the residual rate is applied to the encoder input and the decoder input.

transformers · tensorflow/tensor2tensor / tensor2tensor/models/transformer.py
Pinned revision 0c66117d4297 ↗ · lines 65–66

Read code excerpt
    encoder_input = tf.nn.dropout(encoder_input, 1.0 - hparams.residual_dropout)
    decoder_input = tf.nn.dropout(decoder_input, 1.0 - hparams.residual_dropout)