Encoder output
×tensorencoder_output
The last encoder layer’s output: 512 features per source position, read by every decoder layer.
- shape
- [B batch, S source positions, 512 d]
Sources
Contextual evidence from Encoder: 6 layers
Each of num_hidden_layers encoder layers: self-attention over the encoder input with the padding bias, then the feed-forward network, each through residual_fn.
encoder · tensorflow/tensor2tensor / tensor2tensor/models/transformer.py
Pinned revision 0c66117d4297 ↗ · lines 129–167
Read code excerpt
def transformer_encoder(encoder_input,
residual_fn,
encoder_self_attention_bias,
hparams,
name="encoder"):
"""A stack of transformer layers.
Args:
encoder_input: a Tensor
residual_fn: a function from (layer_input, layer_output) -> combined_output
encoder_self_attention_bias: bias Tensor for self-attention
(see common_attention.attention_bias())
hparams: hyperparameters for model
name: a string
Returns:
y: a Tensors
"""
x = encoder_input
# Summaries don't work in multi-problem setting yet.
summaries = "problems" not in hparams.values() or len(hparams.problems) == 1
with tf.variable_scope(name):
for layer in xrange(hparams.num_hidden_layers):
with tf.variable_scope("layer_%d" % layer):
x = residual_fn(
x,
common_attention.multihead_attention(
x,
None,
encoder_self_attention_bias,
hparams.attention_key_channels or hparams.hidden_size,
hparams.attention_value_channels or hparams.hidden_size,
hparams.hidden_size,
hparams.num_heads,
hparams.attention_dropout,
summaries=summaries,
name="encoder_self_attention"))
x = residual_fn(x, transformer_ffn_layer(x, hparams))
return xContextual evidence from Encoder: 6 layers
Post-norm: each sublayer output is dropped out, added to its input and layer-normalized.
residual-fn · tensorflow/tensor2tensor / tensor2tensor/models/transformer.py
Pinned revision 0c66117d4297 ↗ · lines 59–61
Read code excerpt
def residual_fn(x, y):
return common_layers.layer_norm(x + tf.nn.dropout(
y, 1.0 - hparams.residual_dropout))Contextual evidence from Decoder: 6 layers
Each decoder layer: masked self-attention, attention over the encoder output with the encoder padding bias, then the feed-forward network, each through residual_fn.
decoder · tensorflow/tensor2tensor / tensor2tensor/models/transformer.py
Pinned revision 0c66117d4297 ↗ · lines 170–226
Read code excerpt
def transformer_decoder(decoder_input,
encoder_output,
residual_fn,
decoder_self_attention_bias,
encoder_decoder_attention_bias,
hparams,
name="decoder"):
"""A stack of transformer layers.
Args:
decoder_input: a Tensor
encoder_output: a Tensor
residual_fn: a function from (layer_input, layer_output) -> combined_output
decoder_self_attention_bias: bias Tensor for self-attention
(see common_attention.attention_bias())
encoder_decoder_attention_bias: bias Tensor for encoder-decoder attention
(see common_attention.attention_bias())
hparams: hyperparameters for model
name: a string
Returns:
y: a Tensors
"""
x = decoder_input
# Summaries don't work in multi-problem setting yet.
summaries = "problems" not in hparams.values() or len(hparams.problems) == 1
with tf.variable_scope(name):
for layer in xrange(hparams.num_hidden_layers):
with tf.variable_scope("layer_%d" % layer):
x = residual_fn(
x,
common_attention.multihead_attention(
x,
None,
decoder_self_attention_bias,
hparams.attention_key_channels or hparams.hidden_size,
hparams.attention_value_channels or hparams.hidden_size,
hparams.hidden_size,
hparams.num_heads,
hparams.attention_dropout,
summaries=summaries,
name="decoder_self_attention"))
x = residual_fn(
x,
common_attention.multihead_attention(
x,
encoder_output,
encoder_decoder_attention_bias,
hparams.attention_key_channels or hparams.hidden_size,
hparams.attention_value_channels or hparams.hidden_size,
hparams.hidden_size,
hparams.num_heads,
hparams.attention_dropout,
summaries=summaries,
name="encdec_attention"))
x = residual_fn(x, transformer_ffn_layer(x, hparams))
return x