Transformer · Encoder: 6 layers

in: $x_n$ [B batch, S source positions, 512 d] $M$ [B batch, 1, 1, S keys]
out: $x_{n+1}$ [B batch, S source positions, 512 d]
parameters: 3,152,384 · × 6 layers = 18,914,304

Parameters

3,152,384 parameters per layer, across its learned components

Exact per-instance accounting
Component	Calculation	Parameters
Multi-head self-attention	1,050,624 across its components	1,050,624
LayerNorm	$512 + 512$	1,024
Position-wise feed-forward network	2,099,712 across its components	2,099,712
LayerNorm	$512 + 512$	1,024
Per-instance total		3,152,384

Across 6 independent layers: $3{,}152{,}384 \times 6 = 18{,}914{,}304$ parameters.

Derived from the modules the pinned tensor2tensor v1.0.14 code creates, with a vocabulary of about 37,000 (the paper’s words); no trained weights were published to count.

Explanation

The output of each sublayer is LayerNorm $(x+\text{Sublayer}(x))$ ; all sublayers and embeddings produce outputs of $d_{\text{model}}=512$ (§3.1).

Codetransformer.py 59–61, 129–167

transformer.py 129–167

def transformer_encoder(encoder_input,
                        residual_fn,
                        encoder_self_attention_bias,
                        hparams,
                        name="encoder"):
  """A stack of transformer layers.

  Args:
    encoder_input: a Tensor
    residual_fn: a function from (layer_input, layer_output) -> combined_output
    encoder_self_attention_bias: bias Tensor for self-attention
       (see common_attention.attention_bias())
    hparams: hyperparameters for model
    name: a string

  Returns:
    y: a Tensors
  """
  x = encoder_input
  # Summaries don't work in multi-problem setting yet.
  summaries = "problems" not in hparams.values() or len(hparams.problems) == 1
  with tf.variable_scope(name):
    for layer in xrange(hparams.num_hidden_layers):
      with tf.variable_scope("layer_%d" % layer):
        x = residual_fn(
            x,
            common_attention.multihead_attention(
                x,
                None,
                encoder_self_attention_bias,
                hparams.attention_key_channels or hparams.hidden_size,
                hparams.attention_value_channels or hparams.hidden_size,
                hparams.hidden_size,
                hparams.num_heads,
                hparams.attention_dropout,
                summaries=summaries,
                name="encoder_self_attention"))
        x = residual_fn(x, transformer_ffn_layer(x, hparams))
  return x

transformer.py 59–61

    def residual_fn(x, y):
      return common_layers.layer_norm(x + tf.nn.dropout(
          y, 1.0 - hparams.residual_dropout))

Sources2

Direct evidence

Each of num_hidden_layers encoder layers: self-attention over the encoder input with the padding bias, then the feed-forward network, each through residual_fn.

encoder · tensorflow/tensor2tensor / tensor2tensor/models/transformer.py
Pinned revision 0c66117d4297 ↗ · lines 129–167

Read code excerpt

def transformer_encoder(encoder_input,
                        residual_fn,
                        encoder_self_attention_bias,
                        hparams,
                        name="encoder"):
  """A stack of transformer layers.

  Args:
    encoder_input: a Tensor
    residual_fn: a function from (layer_input, layer_output) -> combined_output
    encoder_self_attention_bias: bias Tensor for self-attention
       (see common_attention.attention_bias())
    hparams: hyperparameters for model
    name: a string

  Returns:
    y: a Tensors
  """
  x = encoder_input
  # Summaries don't work in multi-problem setting yet.
  summaries = "problems" not in hparams.values() or len(hparams.problems) == 1
  with tf.variable_scope(name):
    for layer in xrange(hparams.num_hidden_layers):
      with tf.variable_scope("layer_%d" % layer):
        x = residual_fn(
            x,
            common_attention.multihead_attention(
                x,
                None,
                encoder_self_attention_bias,
                hparams.attention_key_channels or hparams.hidden_size,
                hparams.attention_value_channels or hparams.hidden_size,
                hparams.hidden_size,
                hparams.num_heads,
                hparams.attention_dropout,
                summaries=summaries,
                name="encoder_self_attention"))
        x = residual_fn(x, transformer_ffn_layer(x, hparams))
  return x

Direct evidence

Post-norm: each sublayer output is dropped out, added to its input and layer-normalized.

residual-fn · tensorflow/tensor2tensor / tensor2tensor/models/transformer.py
Pinned revision 0c66117d4297 ↗ · lines 59–61

Read code excerpt

    def residual_fn(x, y):
      return common_layers.layer_norm(x + tf.nn.dropout(
          y, 1.0 - hparams.residual_dropout))