Logits
×tensorlogits
Unnormalized scores over the vocabulary for the token at each target position.
- shape
- [B batch, T target positions, 37000 vocabulary]
Sources
Contextual evidence from Tied vocabulary projection
With shared weights the logits are the decoder output times the transposed table.
symbol-top · tensorflow/tensor2tensor / tensor2tensor/models/modalities.py
Pinned revision 0c66117d4297 ↗ · lines 98–120
Read code excerpt
def top(self, body_output, targets):
"""Generate logits.
Args:
body_output: A Tensor with shape [batch, p0, p1, body_input_depth]
targets: A Tensor with shape [batch, p0, p1, 1]
Returns:
logits: A Tensor with shape [batch, p0, p1, ?, vocab_size].
"""
if self._model_hparams.shared_embedding_and_softmax_weights:
scope_name = "shared"
reuse = True
else:
scope_name = "softmax"
reuse = False
with tf.variable_scope(scope_name, reuse=reuse):
var = self._get_weights()
shape = tf.shape(body_output)[:-1]
body_output = tf.reshape(body_output, [-1, self._body_input_depth])
logits = tf.matmul(body_output, var, transpose_b=True)
logits = tf.reshape(logits, tf.concat([shape, [self._vocab_size]], 0))
# insert a channels dimension
return tf.expand_dims(logits, 3)