Transformer · Shift right · OpenModelAtlas

Shift right

no parameterscommon_layers.shift_left_3d(targets) · transformer.py 123

Move the target one position right, so position t sees only the tokens before it.

{\htmlData{light=shifted-ids shift-right}{I^{in}_{t}}}=\begin{cases}0 & t=0\\ I^{tgt}_{t-1} & t>0\end{cases}

in: $I^{tgt}$ [B batch, T target positions]
out: $I^{in}$ [B batch, T target positions]

The code shifts the embedded targets rather than the IDs (shift_left_3d pads a zero vector); the zero vector is what padding id 0 embeds to, so the result is the same. The function’s name says left; its docstring and its padding say right (common_layers.py 209–215).

Explanation

The output embeddings are offset by one position, which with the mask ensures that the prediction for position $t$ depends only on the known outputs before $t$ (§3.1).

Codecommon_layers.py 209–215 · transformer.py 109–126

common_layers.py 209–215

def shift_left_3d(x, pad_value=None):
  """Shift the second dimension of x right by one."""
  if pad_value is None:
    shifted_targets = tf.pad(x, [[0, 0], [1, 0], [0, 0]])[:, :-1, :]
  else:
    shifted_targets = tf.concat([pad_value, x], axis=1)[:, :-1, :]
  return shifted_targets

transformer.py 109–126

def transformer_prepare_decoder(targets, hparams):
  """Prepare one shard of the model for the decoder.

  Args:
    targets: a Tensor.
    hparams: run hyperparameters

  Returns:
    decoder_input: a Tensor, bottom of decoder stack
    decoder_self_attention_bias: a Tensor, containing large negative values
    to implement masked attention and possibly baises for diagonal alignments
  """
  decoder_self_attention_bias = (
      common_attention.attention_bias_lower_triangle(tf.shape(targets)[1]))
  decoder_input = common_layers.shift_left_3d(targets)
  if hparams.pos == "timing":
    decoder_input = common_attention.add_timing_signal_1d(decoder_input)
  return (decoder_input, decoder_self_attention_bias)

Sources2

Direct evidence

shift_left_3d pads one position at the start and drops the last: the targets shifted right by one.

shift · tensorflow/tensor2tensor / tensor2tensor/models/common_layers.py
Pinned revision 0c66117d4297 ↗ · lines 209–215

Read code excerpt

def shift_left_3d(x, pad_value=None):
  """Shift the second dimension of x right by one."""
  if pad_value is None:
    shifted_targets = tf.pad(x, [[0, 0], [1, 0], [0, 0]])[:, :-1, :]
  else:
    shifted_targets = tf.concat([pad_value, x], axis=1)[:, :-1, :]
  return shifted_targets

Direct evidence

The decoder input is the embedded targets shifted right by one position, plus the timing signal; its self-attention bias is lower-triangular.

prepare-decoder · tensorflow/tensor2tensor / tensor2tensor/models/transformer.py
Pinned revision 0c66117d4297 ↗ · lines 109–126

Read code excerpt

def transformer_prepare_decoder(targets, hparams):
  """Prepare one shard of the model for the decoder.

  Args:
    targets: a Tensor.
    hparams: run hyperparameters

  Returns:
    decoder_input: a Tensor, bottom of decoder stack
    decoder_self_attention_bias: a Tensor, containing large negative values
    to implement masked attention and possibly baises for diagonal alignments
  """
  decoder_self_attention_bias = (
      common_attention.attention_bias_lower_triangle(tf.shape(targets)[1]))
  decoder_input = common_layers.shift_left_3d(targets)
  if hparams.pos == "timing":
    decoder_input = common_attention.add_timing_signal_1d(decoder_input)
  return (decoder_input, decoder_self_attention_bias)