Latent decoder › Query MLP
Input
×tensor
The features entering the query MLP.
- shape
- [B batch, T positions, 768 d]
Sources
Contextual evidence from Mask vector
The latent decoder owns one position table, the mask token’s vector copied from the token table of a GPT-2 loaded for the purpose and frozen because the feature model is configured frozen, an MLP for the queries, the blocks and a scale for the prepared queries.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Mask vector
Every query is the mask vector: it passes the MLP and a scale, positions are added; the latents take the same position table; the blocks are applied; every position counts as masked.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Mask vector
A GPT-2 is loaded by name and its token table resized to the configured vocabulary; that table is returned.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Mask vector
The feature model is GPT-2, frozen; the hidden state at index −3 is extracted; the encoder and the decoder are of the Perceiver kind; the file’s vocabulary size and mask token ID are defaults that the trainer overwrites with the tokenizer’s; there are as many latents as positions, of 768 features.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from LayerNorm
The Perceiver blocks’ MLP: LayerNorm, an affine layer, exact GELU, dropout, an affine layer.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from LayerNorm
LayerNorm normalizes the final configured axes with population variance, defaults to epsilon 1e-5, and owns learned gain and bias.
torch-module-norm · pytorch/pytorch / torch/nn/modules/normalization.py
Pinned revision ee1b6804381c ↗ · lines 90–203
Read code excerpt
class LayerNorm(Module):
r"""Applies Layer Normalization over a mini-batch of inputs.
This layer implements the operation as described in
the paper `Layer Normalization <https://arxiv.org/abs/1607.06450>`__
.. math::
y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta
The mean and standard-deviation are calculated over the last `D` dimensions, where `D`
is the dimension of :attr:`normalized_shape`. For example, if :attr:`normalized_shape`
is ``(3, 5)`` (a 2-dimensional shape), the mean and standard-deviation are computed over
the last 2 dimensions of the input (i.e. ``input.mean((-2, -1))``).
:math:`\gamma` and :math:`\beta` are learnable affine transform parameters of
:attr:`normalized_shape` if :attr:`elementwise_affine` is ``True``.
The standard-deviation is calculated via the biased estimator, equivalent to
`torch.var(input, unbiased=False)`.
.. note::
Unlike Batch Normalization and Instance Normalization, which applies
scalar scale and bias for each entire channel/plane with the
:attr:`affine` option, Layer Normalization applies per-element scale and
bias with :attr:`elementwise_affine`.
This layer uses statistics computed from input data in both training and
evaluation modes.
Args:
normalized_shape (int or list or torch.Size): input shape from an expected input
of size
.. math::
[* \times \text{normalized\_shape}[0] \times \text{normalized\_shape}[1]
\times \ldots \times \text{normalized\_shape}[-1]]
If a single integer is used, it is treated as a singleton list, and this module will… lines 126–172 omitted …
def __init__(self, normalized_shape: _shape_t, eps: float = 1e-5, elementwise_affine: bool = True,
bias: bool = True, device=None, dtype=None) -> None:
factory_kwargs = {'device': device, 'dtype': dtype}
super().__init__()
if isinstance(normalized_shape, numbers.Integral):
# mypy error: incompatible types in assignment
normalized_shape = (normalized_shape,) # type: ignore[assignment]
self.normalized_shape = tuple(normalized_shape) # type: ignore[arg-type]
self.eps = eps
self.elementwise_affine = elementwise_affine
if self.elementwise_affine:
self.weight = Parameter(torch.empty(self.normalized_shape, **factory_kwargs))
if bias:
self.bias = Parameter(torch.empty(self.normalized_shape, **factory_kwargs))
else:
self.register_parameter('bias', None)
else:
self.register_parameter('weight', None)
self.register_parameter('bias', None)
self.reset_parameters()
def reset_parameters(self) -> None:
if self.elementwise_affine:
init.ones_(self.weight)
if self.bias is not None:
init.zeros_(self.bias)
def forward(self, input: Tensor) -> Tensor:
return F.layer_norm(
input, self.normalized_shape, self.weight, self.bias, self.eps)