Transformer · Add encoder–decoder attention output

Decoder 6 layers

Add encoder–decoder attention output

no parametersx + tf.nn.dropout(y, …) · 60

Add the mechanism’s update to the residual stream element by element.

{\htmlData{light=cross-sum cross-add}{{\htmlClass{vl-stream-v}{v}}}}={\htmlData{light=after-self self-norm}{s_n}}+D_{\mathrm{P_{\mathrm{drop}}}}({\htmlData{light=cross-update cross-attention}{\Delta_{\mathrm{cross}}}})

in: $s_n$ [B batch, T target positions, 512 d] $D_{\mathrm{P_{\mathrm{drop}}}}(\Delta_{\mathrm{cross}})$ [B batch, T target positions, 512 d]
out: $v$ [B batch, T target positions, 512 d]

Codetransformer.py 59–61

transformer.py 59–61

    def residual_fn(x, y):
      return common_layers.layer_norm(x + tf.nn.dropout(
          y, 1.0 - hparams.residual_dropout))

Sources1

Direct evidence

Post-norm: each sublayer output is dropped out, added to its input and layer-normalized.

residual-fn · tensorflow/tensor2tensor / tensor2tensor/models/transformer.py
Pinned revision 0c66117d4297 ↗ · lines 59–61

Read code excerpt

    def residual_fn(x, y):
      return common_layers.layer_norm(x + tf.nn.dropout(
          y, 1.0 - hparams.residual_dropout))

Incoming transform evidence

Post-norm: each sublayer output is dropped out, added to its input and layer-normalized.

Dropout on the encoder–decoder attention output · tensorflow/tensor2tensor / tensor2tensor/models/transformer.py
Pinned revision 0c66117d4297 ↗ · lines 59–61

Read code excerpt

    def residual_fn(x, y):
      return common_layers.layer_norm(x + tf.nn.dropout(
          y, 1.0 - hparams.residual_dropout))