Transformer · First linear map

Decoder 6 layers › Feed forward

First linear map

learnedconv_f1(inputs, hidden_size, kernel_size, activation=tf.nn.relu, name="conv1") · 1177–1183

Map 512 features to 2,048.

{\htmlData{light=h up}{h}}={\htmlData{light=x}{x}}W_1+b_1

in: $x$ [B batch, T target positions, 512 d]
out: $h$ [B batch, T target positions, 2048 d_ff]
parameters: 1,050,624 · × 6 layers = 6,303,744

The code applies ReLU inside the same 1×1 convolution (activation=tf.nn.relu); it is drawn as its own step.

Parameters

1,050,624 parameters per instance.

Exact per-instance accounting
Component	Calculation	Parameters
Weight matrix	$512 \times 2{,}048$	1,048,576
Output bias	$2{,}048$	2,048
Per-instance total		1,050,624

Across 6 independent layers: $1{,}050{,}624 \times 6 = 6{,}303{,}744$ parameters.

Derived from the configured matrix dimensions; checkpoint tensors have not been independently counted.

Codecommon_layers.py 354–360, 1160–1194

common_layers.py 1160–1194

def conv_hidden_relu(inputs,
                     hidden_size,
                     output_size,
                     kernel_size=(1, 1),
                     second_kernel_size=(1, 1),
                     summaries=True,
                     dropout=0.0,
                     **kwargs):
  """Hidden layer with RELU activation followed by linear projection."""
  name = kwargs.pop("name") if "name" in kwargs else None
  with tf.variable_scope(name, "conv_hidden_relu", [inputs]):
    if inputs.get_shape().ndims == 3:
      is_3d = True
      inputs = tf.expand_dims(inputs, 2)
    else:
      is_3d = False
    conv_f1 = conv if kernel_size == (1, 1) else separable_conv
    h = conv_f1(
        inputs,
        hidden_size,
        kernel_size,
        activation=tf.nn.relu,
        name="conv1",
        **kwargs)
    if dropout != 0.0:
      h = tf.nn.dropout(h, 1.0 - dropout)
    if summaries and not tf.get_variable_scope().reuse:
      tf.summary.histogram("hidden_density_logit",
                           relu_density_logit(
                               h, list(range(inputs.shape.ndims - 1))))
    conv_f2 = conv if second_kernel_size == (1, 1) else separable_conv
    ret = conv_f2(h, output_size, second_kernel_size, name="conv2", **kwargs)
    if is_3d:
      ret = tf.squeeze(ret, 2)
    return ret

common_layers.py 354–360

def conv(inputs, filters, kernel_size, **kwargs):
  return conv_internal(tf.layers.conv2d, inputs, filters, kernel_size, **kwargs)


def conv1d(inputs, filters, kernel_size, **kwargs):
  return tf.squeeze(
      conv(tf.expand_dims(inputs, 2), filters, (kernel_size, 1), **kwargs), 2)

Sources2

Direct evidence

conv_hidden_relu: a 1×1 convolution to the hidden size with ReLU, optional dropout, then a 1×1 convolution to the output size.

conv-relu · tensorflow/tensor2tensor / tensor2tensor/models/common_layers.py
Pinned revision 0c66117d4297 ↗ · lines 1160–1194

Read code excerpt

def conv_hidden_relu(inputs,
                     hidden_size,
                     output_size,
                     kernel_size=(1, 1),
                     second_kernel_size=(1, 1),
                     summaries=True,
                     dropout=0.0,
                     **kwargs):
  """Hidden layer with RELU activation followed by linear projection."""
  name = kwargs.pop("name") if "name" in kwargs else None
  with tf.variable_scope(name, "conv_hidden_relu", [inputs]):
    if inputs.get_shape().ndims == 3:
      is_3d = True
      inputs = tf.expand_dims(inputs, 2)
    else:
      is_3d = False
    conv_f1 = conv if kernel_size == (1, 1) else separable_conv
    h = conv_f1(
        inputs,
        hidden_size,
        kernel_size,
        activation=tf.nn.relu,
        name="conv1",
        **kwargs)
    if dropout != 0.0:
      h = tf.nn.dropout(h, 1.0 - dropout)
    if summaries and not tf.get_variable_scope().reuse:
      tf.summary.histogram("hidden_density_logit",
                           relu_density_logit(
                               h, list(range(inputs.shape.ndims - 1))))
    conv_f2 = conv if second_kernel_size == (1, 1) else separable_conv
    ret = conv_f2(h, output_size, second_kernel_size, name="conv2", **kwargs)
    if is_3d:
      ret = tf.squeeze(ret, 2)
    return ret

Direct evidence

conv1d is a 1×1 tf.layers.conv2d, which owns a kernel and a bias.

conv1d · tensorflow/tensor2tensor / tensor2tensor/models/common_layers.py
Pinned revision 0c66117d4297 ↗ · lines 354–360

Read code excerpt

def conv(inputs, filters, kernel_size, **kwargs):
  return conv_internal(tf.layers.conv2d, inputs, filters, kernel_size, **kwargs)


def conv1d(inputs, filters, kernel_size, **kwargs):
  return tf.squeeze(
      conv(tf.expand_dims(inputs, 2), filters, (kernel_size, 1), **kwargs), 2)