- in
- [B batch, T positions, 2304 q, k, v]
- out
- [B batch, 12 heads, T positions, 64 head features]
- parameters
- 64 · × 6 blocks = 384
Parameters
64 parameters per instance, across its learned components
| Component | Calculation | Parameters |
|---|---|---|
| Query RMSNorm | 64 | |
| Per-instance total | 64 | |
Across 6 independent blocks: parameters.
Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.
Explanation
Queries come from one stream only; this is what makes the attention a Perceiver’s.
Sources3
Direct evidence
Both streams are normalized; queries come from one stream, keys and values from both, concatenated along positions; queries and keys are normalized per head; attention runs over all positions with softcap; the heads are merged and projected.
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
RMSNorm divides by the vector’s norm times width^(−1/2), clamped from below at ε, and multiplies by a learned gain.
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
Perceiver attention owns an RMSNorm for each of its two streams, a fused query–key–value projection of the query stream, a fused key–value projection of the other stream, RMSNorms for each head’s queries and keys, and an output projection; no bias.
The implementation, pinned at revision e18c42acdfb7, not yet public