Latent encoder › 6 encoder blocks
Latents after attention
×tensor
The residual features after the attention update, feeding the MLP branch and its bypass.
- shape
- [B batch, T positions, 768 d]
Sources
Contextual evidence from Add attention update
An encoder block adds the Perceiver attention’s output to the latents, then adds the MLP’s output.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from MLP
The Perceiver blocks’ MLP: LayerNorm, an affine layer, exact GELU, dropout, an affine layer.
The implementation, pinned at revision e18c42acdfb7, not yet public