Latent decoder › 6 decoder blocks › Perceiver attention
The decoder state’s queries, keys and values
×tensor
Three parts of 768 features side by side: queries, keys and values of the stream that asks.
- shape
- [B batch, T positions, 2304 q, k, v]
Sources
Contextual evidence from Query, key and value projection
Perceiver attention owns an RMSNorm for each of its two streams, a fused query–key–value projection of the query stream, a fused key–value projection of the other stream, RMSNorms for each head’s queries and keys, and an output projection; no bias.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Query, key and value projection
Both streams are normalized; queries come from one stream, keys and values from both, concatenated along positions; queries and keys are normalized per head; attention runs over all positions with softcap; the heads are merged and projected.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Query heads
RMSNorm divides by the vector’s norm times width^(−1/2), clamped from below at ε, and multiplies by a learned gain.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Key and value heads
An encoder block adds the Perceiver attention’s output to the latents, then adds the MLP’s output.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Key and value heads
A decoder block: Perceiver attention with the queries’ stream first read from the decoder’s state and keys and values from the latents and the state, a scaled residual, an MLP and a second scaled residual.
The implementation, pinned at revision e18c42acdfb7, not yet public