Latent encoder › 6 encoder blocks › Perceiver attention
Keys
×tensor
A normalized 64-feature key for every head and each of the 2T positions of the two streams.
- shape
- [B batch, 12 heads, 2T positions, 64 head features]
Sources
Contextual evidence from Key and value heads
Both streams are normalized; queries come from one stream, keys and values from both, concatenated along positions; queries and keys are normalized per head; attention runs over all positions with softcap; the heads are merged and projected.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Key and value heads
An encoder block adds the Perceiver attention’s output to the latents, then adds the MLP’s output.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Key and value heads
A decoder block: Perceiver attention with the queries’ stream first read from the decoder’s state and keys and values from the latents and the state, a scaled residual, an MLP and a second scaled residual.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Key and value heads
RMSNorm divides by the vector’s norm times width^(−1/2), clamped from below at ε, and multiplies by a learned gain.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Key and value heads
Perceiver attention owns an RMSNorm for each of its two streams, a fused query–key–value projection of the query stream, a fused key–value projection of the other stream, RMSNorms for each head’s queries and keys, and an output projection; no bias.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Attention over both streams
The latent encoder: 12 heads of 64 features, query and key normalization, softcap 50, 6 layers, MLP multiplier 4, dropout 0.1, ε = 10⁻⁵.
The implementation, pinned at revision e18c42acdfb7, not yet public