Transformer · Add self-attention output

Decoder 6 layers

Add self-attention output

no parametersx + tf.nn.dropout(y, …) · 60

Add the mechanism’s update to the residual stream element by element.

{\htmlData{light=self-sum self-add}{u}}={\htmlData{light=block-input}{y_n}}+D_{\mathrm{P_{\mathrm{drop}}}}({\htmlData{light=self-update self-attention}{\Delta_{\mathrm{self}}}})

in: $y_n$ [B batch, T target positions, 512 d] $D_{\mathrm{P_{\mathrm{drop}}}}(\Delta_{\mathrm{self}})$ [B batch, T target positions, 512 d]
out: $u$ [B batch, T target positions, 512 d]

Codetransformer.py 59–61

    def residual_fn(x, y):
      return common_layers.layer_norm(x + tf.nn.dropout(
          y, 1.0 - hparams.residual_dropout))

Sources1

Direct evidence

Post-norm: each sublayer output is dropped out, added to its input and layer-normalized.

residual-fn · tensorflow/tensor2tensor / tensor2tensor/models/transformer.py
Pinned revision 0c66117d4297 ↗ · lines 59–61

Read code excerpt

    def residual_fn(x, y):
      return common_layers.layer_norm(x + tf.nn.dropout(
          y, 1.0 - hparams.residual_dropout))

Incoming transform evidence

Post-norm: each sublayer output is dropped out, added to its input and layer-normalized.

Dropout on the self-attention output · tensorflow/tensor2tensor / tensor2tensor/models/transformer.py
Pinned revision 0c66117d4297 ↗ · lines 59–61

Read code excerpt

    def residual_fn(x, y):
      return common_layers.layer_norm(x + tf.nn.dropout(
          y, 1.0 - hparams.residual_dropout))