LDLMIn construction

All models, with their blocks →

Size

306M trained

Published

Meshchaninov et al. · May 2026

Line

Diffusion language model

View

Architecture

Source

LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verifiedHow to read

Perceiver attention

Training Generation

Latent encoder › 6 encoder blocks › Perceiver attention

Attention over both streams

Open inside →×

no parameters

Each query weighs the values at all 2T positions by its softcapped scores against their keys.

A_{t,s}=\frac{\exp F_{t,s}}{\sum_{u=1}^{2T}\exp F_{t,u}}

O=AV

All 4 steps

S=QK^{\mathsf T}

F=50\tanh\!\Big(\frac{S}{50\sqrt{64}}\Big)

A_{t,s}=\frac{\exp F_{t,s}}{\sum_{u=1}^{2T}\exp F_{t,u}}

O=AV

in: $Q$ [B batch, 12 heads, T positions, 64 head features] $K$ [B batch, 12 heads, 2T positions, 64 head features] $V$ [B batch, 12 heads, 2T positions, 64 head features]
out: $O$ [B batch, 12 heads, T positions, 64 head features]

Explanation

The fused kernel does not store the score and weight fields; the plate states its arithmetic step by step.

Sources2

Direct evidence

Both streams are normalized; queries come from one stream, keys and values from both, concatenated along positions; queries and keys are normalized per head; attention runs over all positions with softcap; the heads are merged and projected.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The latent encoder: 12 heads of 64 features, query and key normalization, softcap 50, 6 layers, MLP multiplier 4, dropout 0.1, ε = 10⁻⁵.

The implementation, pinned at revision e18c42acdfb7, not yet public