LDLM · Perceiver attention

in: $r_n$ [B batch, T positions, 768 d] $z^{p}$ [B batch, T positions, 768 d]
out: $\Delta_{\mathrm{attn}}$ [B batch, T positions, 768 d]
parameters: 3,540,608 · × 6 blocks = 21,243,648
mode: generation

Parameters

3,540,608 parameters per instance, across its learned components

Exact per-instance accounting
Component	Calculation	Parameters
RMSNorm	$768$	768
RMSNorm	$768$	768
Query, key and value projection	$768 \times 2{,}304$	1,769,472
Key and value projection	$768 \times 1{,}536$	1,179,648
Query heads	64 across its components	64
Key and value heads	64 across its components	64
Output projection	$768 \times 768$	589,824
Per-instance total		3,540,608

Across 6 independent blocks: $3{,}540{,}608 \times 6 = 21{,}243{,}648$ parameters.

Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.

Explanation

This is how the decoder reads the latents: only its own state is updated, and the latents enter every block unchanged.

Queries and keys are normalized per head and the scores softcapped at 50; nothing encodes position here, the positions were added to both streams before the blocks.

Sources2

Direct evidence

Perceiver attention owns an RMSNorm for each of its two streams, a fused query–key–value projection of the query stream, a fused key–value projection of the other stream, RMSNorms for each head’s queries and keys, and an output projection; no bias.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

Both streams are normalized; queries come from one stream, keys and values from both, concatenated along positions; queries and keys are normalized per head; attention runs over all positions with softcap; the heads are merged and projected.

The implementation, pinned at revision e18c42acdfb7, not yet public

LDLMIn construction