Latent decoder
Initial decoder state
×tensor
The same prepared query at every position, told apart only by the position added.
- shape
- [B batch, T positions, 768 d]
Sources
Contextual evidence from Positions to queries
Every query is the mask vector: it passes the MLP and a scale, positions are added; the latents take the same position table; the blocks are applied; every position counts as masked.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Decoder block
A decoder block: Perceiver attention with the queries’ stream first read from the decoder’s state and keys and values from the latents and the state, a scaled residual, an MLP and a second scaled residual.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Decoder block
The decoder’s residual scale: x · (1 + s), with one learned scalar for masked positions and one for unmasked ones, both zero at initialization.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Decoder block
The latent decoder: 12 heads of 64 features, query and key normalization, softcap 50, 6 layers, MLP multiplier 4; the token decoder has 3 blocks.
The implementation, pinned at revision e18c42acdfb7, not yet public