Padding mask
×no parametersattention_bias_ignore_padding(embedding_to_padding(encoder_input)) · transformer.py 95–98
Mark the padding positions of the source so that no query attends to them.
- in
- [B batch, S source positions, 512 d]
- out
- [B batch, 1, 1, S keys]
Not in the paper, which does not discuss padding. The code finds padding from the zeroed embedding rows.
Codecommon_attention.py 166–178, 194–205 · transformer.py 78–106
def embedding_to_padding(emb):
"""Input embeddings -> is_padding.
We have hacked symbol_modality to return all-zero embeddings for padding.
Args:
emb: a Tensor with shape [..., depth].
Returns:
a boolean Tensor with shape [...].
"""
emb_sum = tf.reduce_sum(tf.abs(emb), axis=-1)
return tf.equal(emb_sum, 0.0)
def attention_bias_ignore_padding(memory_padding):
"""Create an bias tensor to be added to attention logits.
Args:
memory_padding: a boolean `Tensor` with shape [batch, memory_length].
Returns:
a `Tensor` with shape [batch, 1, 1, memory_length].
"""
ret = tf.to_float(memory_padding) * -1e9
return tf.expand_dims(tf.expand_dims(ret, 1), 1)
def transformer_prepare_encoder(inputs, target_space, hparams):
"""Prepare one shard of the model for the encoder.
Args:
inputs: a Tensor.
target_space: a Tensor.
hparams: run hyperparameters
Returns:
encoder_input: a Tensor, bottom of encoder stack
encoder_self_attention_bias: a Tensor, containing large negative values
to implement masked attention and possibly baises for diagonal
alignments
encoder_padding: a Tensor
"""
# Flatten inputs.
ishape_static = inputs.shape.as_list()
encoder_input = inputs
encoder_padding = common_attention.embedding_to_padding(encoder_input)
encoder_self_attention_bias = common_attention.attention_bias_ignore_padding(
encoder_padding)
# Append target_space_id embedding to inputs.
emb_target_space = common_layers.embedding(
target_space, 32, ishape_static[-1], name="target_space_embedding")
emb_target_space = tf.reshape(emb_target_space, [1, 1, -1])
encoder_input += emb_target_space
if hparams.pos == "timing":
encoder_input = common_attention.add_timing_signal_1d(encoder_input)
return (encoder_input, encoder_self_attention_bias, encoder_padding)Sources2
Direct evidence
Padding positions are those whose embedding is all zeros; they get an attention bias of −1e9.
padding · tensorflow/tensor2tensor / tensor2tensor/models/common_attention.py
Pinned revision 0c66117d4297 ↗ · lines 166–205
Read code excerpt
def embedding_to_padding(emb):
"""Input embeddings -> is_padding.
We have hacked symbol_modality to return all-zero embeddings for padding.
Args:
emb: a Tensor with shape [..., depth].
Returns:
a boolean Tensor with shape [...].
"""
emb_sum = tf.reduce_sum(tf.abs(emb), axis=-1)
return tf.equal(emb_sum, 0.0)
… lines 179–193 omitted …
def attention_bias_ignore_padding(memory_padding):
"""Create an bias tensor to be added to attention logits.
Args:
memory_padding: a boolean `Tensor` with shape [batch, memory_length].
Returns:
a `Tensor` with shape [batch, 1, 1, memory_length].
"""
ret = tf.to_float(memory_padding) * -1e9
return tf.expand_dims(tf.expand_dims(ret, 1), 1)
Direct evidence
The encoder input is the embedded source plus a learned target-space embedding of 32 rows, plus the timing signal; the padding positions become a −1e9 attention bias.
prepare-encoder · tensorflow/tensor2tensor / tensor2tensor/models/transformer.py
Pinned revision 0c66117d4297 ↗ · lines 78–106
Read code excerpt
def transformer_prepare_encoder(inputs, target_space, hparams):
"""Prepare one shard of the model for the encoder.
Args:
inputs: a Tensor.
target_space: a Tensor.
hparams: run hyperparameters
Returns:
encoder_input: a Tensor, bottom of encoder stack
encoder_self_attention_bias: a Tensor, containing large negative values
to implement masked attention and possibly baises for diagonal
alignments
encoder_padding: a Tensor
"""
# Flatten inputs.
ishape_static = inputs.shape.as_list()
encoder_input = inputs
encoder_padding = common_attention.embedding_to_padding(encoder_input)
encoder_self_attention_bias = common_attention.attention_bias_ignore_padding(
encoder_padding)
# Append target_space_id embedding to inputs.
emb_target_space = common_layers.embedding(
target_space, 32, ishape_static[-1], name="target_space_embedding")
emb_target_space = tf.reshape(emb_target_space, [1, 1, -1])
encoder_input += emb_target_space
if hparams.pos == "timing":
encoder_input = common_attention.add_timing_signal_1d(encoder_input)
return (encoder_input, encoder_self_attention_bias, encoder_padding)