Decoder 6 layers › Masked attention
Causal mask
×no parametersattention_bias_lower_triangle(tf.shape(targets)[1]) · transformer.py 121–122
0 at the current and earlier target positions, −10⁹ at later ones.
- in
- [B batch, T target positions, 512 d]
- out
- [1, 1, T queries, T keys]
Explanation
Masking out the later positions keeps the decoder auto-regressive: position can depend only on positions before it, since its input is the target shifted right (§3.1, §3.2.3).
Codecommon_attention.py 180–191 · transformer.py 109–126
def attention_bias_lower_triangle(length):
"""Create an bias tensor to be added to attention logits.
Args:
length: a Scalar.
Returns:
a `Tensor` with shape [1, 1, length, length].
"""
lower_triangle = tf.matrix_band_part(tf.ones([length, length]), -1, 0)
ret = -1e9 * (1.0 - lower_triangle)
return tf.reshape(ret, [1, 1, length, length])def transformer_prepare_decoder(targets, hparams):
"""Prepare one shard of the model for the decoder.
Args:
targets: a Tensor.
hparams: run hyperparameters
Returns:
decoder_input: a Tensor, bottom of decoder stack
decoder_self_attention_bias: a Tensor, containing large negative values
to implement masked attention and possibly baises for diagonal alignments
"""
decoder_self_attention_bias = (
common_attention.attention_bias_lower_triangle(tf.shape(targets)[1]))
decoder_input = common_layers.shift_left_3d(targets)
if hparams.pos == "timing":
decoder_input = common_attention.add_timing_signal_1d(decoder_input)
return (decoder_input, decoder_self_attention_bias)Sources2
Direct evidence
The lower-triangle bias: 0 on and below the diagonal, −1e9 above it.
causal · tensorflow/tensor2tensor / tensor2tensor/models/common_attention.py
Pinned revision 0c66117d4297 ↗ · lines 180–191
Read code excerpt
def attention_bias_lower_triangle(length):
"""Create an bias tensor to be added to attention logits.
Args:
length: a Scalar.
Returns:
a `Tensor` with shape [1, 1, length, length].
"""
lower_triangle = tf.matrix_band_part(tf.ones([length, length]), -1, 0)
ret = -1e9 * (1.0 - lower_triangle)
return tf.reshape(ret, [1, 1, length, length])Direct evidence
The decoder input is the embedded targets shifted right by one position, plus the timing signal; its self-attention bias is lower-triangular.
prepare-decoder · tensorflow/tensor2tensor / tensor2tensor/models/transformer.py
Pinned revision 0c66117d4297 ↗ · lines 109–126
Read code excerpt
def transformer_prepare_decoder(targets, hparams):
"""Prepare one shard of the model for the decoder.
Args:
targets: a Tensor.
hparams: run hyperparameters
Returns:
decoder_input: a Tensor, bottom of decoder stack
decoder_self_attention_bias: a Tensor, containing large negative values
to implement masked attention and possibly baises for diagonal alignments
"""
decoder_self_attention_bias = (
common_attention.attention_bias_lower_triangle(tf.shape(targets)[1]))
decoder_input = common_layers.shift_left_3d(targets)
if hparams.pos == "timing":
decoder_input = common_attention.add_timing_signal_1d(decoder_input)
return (decoder_input, decoder_self_attention_bias)