Transformer · Add feed-forward output

Decoder 6 layers

Add feed-forward output

no parametersx + tf.nn.dropout(y, …) · 60

Add the mechanism’s update to the residual stream element by element.

{\htmlData{light=ffn-sum ffn-add}{w}}={\htmlData{light=residual cross-norm}{c_n}}+D_{\mathrm{P_{\mathrm{drop}}}}({\htmlData{light=ffn-update ffn}{\Delta_{\mathrm{ffn}}}})

in: $c_n$ [B batch, T target positions, 512 d] $D_{\mathrm{P_{\mathrm{drop}}}}(\Delta_{\mathrm{ffn}})$ [B batch, T target positions, 512 d]
out: $w$ [B batch, T target positions, 512 d]

Codetransformer.py 59–61

    def residual_fn(x, y):
      return common_layers.layer_norm(x + tf.nn.dropout(
          y, 1.0 - hparams.residual_dropout))

Sources1

Direct evidence

Post-norm: each sublayer output is dropped out, added to its input and layer-normalized.

residual-fn · tensorflow/tensor2tensor / tensor2tensor/models/transformer.py
Pinned revision 0c66117d4297 ↗ · lines 59–61

Read code excerpt

    def residual_fn(x, y):
      return common_layers.layer_norm(x + tf.nn.dropout(
          y, 1.0 - hparams.residual_dropout))

Incoming transform evidence

Post-norm: each sublayer output is dropped out, added to its input and layer-normalized.

Dropout on the feed-forward output · tensorflow/tensor2tensor / tensor2tensor/models/transformer.py
Pinned revision 0c66117d4297 ↗ · lines 59–61

Read code excerpt

    def residual_fn(x, y):
      return common_layers.layer_norm(x + tf.nn.dropout(
          y, 1.0 - hparams.residual_dropout))