LDLM · Attention over all positions

Sampler › Denoiser › 12 denoiser blocks › Self-attention

Attention over all positions

no parameters

Each query weighs the values at all T positions by its softcapped scores against their keys.

A_{t,s}=\frac{\exp F_{t,s}}{\sum_{u=1}^{T}\exp F_{t,u}}

O=AV

All 4 steps

S=QK^{\mathsf T}

F=50\tanh\!\Big(\frac{S}{50\sqrt{64}}\Big)

A_{t,s}=\frac{\exp F_{t,s}}{\sum_{u=1}^{T}\exp F_{t,u}}

O=AV

in: $Q$ [B batch, 12 heads, T positions, 64 head features] $K$ [B batch, 12 heads, T positions, 64 head features] $V$ [B batch, 12 heads, T positions, 64 head features]
out: $O$ [B batch, 12 heads, T positions, 64 head features]

Explanation

The fused kernel does not store the score and weight fields; the plate states its arithmetic step by step.

Sources2

Direct evidence

A block normalizes, modulates by the time condition’s shift and scale, projects to queries, keys and values, rotates queries and keys, attends over all positions with softcap, and adds the projected output scaled by a gate; the MLP branch does the same with its own shift, scale and gate.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

Diffusion: self-conditioning on, `t_min` 0.05 and `eps` 10⁻⁵; the adaptive time sampler has 100 bins, a minimum weight of 0.0001, an EMA of 0.9 and updates every 5,000 steps; the denoiser’s EMA decay is 0.9999; scheduler `tanh` with d = 3, solver `euler`, `ode_sampling` false, N = 128 by default; the denoiser has 12 blocks of width 768 with 12 heads, time conditioning `adaln`, a standard MLP, softcap 50 and self-conditioning by concatenation; the timestep embedding’s `max_period` is 10.

The implementation, pinned at revision e18c42acdfb7, not yet public

LDLMIn construction