LDLMIn construction

Meshchaninov et al. · May 2026Diffusion language model

Encoder block

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

u_n[B, T, 768]h^p[B, T, 768]

Perceiver attention Open →

+u^′_n[B, T, 768]

MLP Open →

+u_n+1[B, T, 768]

Latent encoder › 6 encoder blocks

Latents, next block

tensor

The residual features after both mechanism updates.

shape: $u_{n+1}$ [B batch, T positions, 768 d]

Sources

Contextual evidence from Encoder block

An encoder block adds the Perceiver attention’s output to the latents, then adds the MLP’s output.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Encoder block

The encoder adds positions to the features, normalizes the learned queries, applies the blocks, each attending to the features, and normalizes the result.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Encoder block

The latent encoder: 12 heads of 64 features, query and key normalization, softcap 50, 6 layers, MLP multiplier 4, dropout 0.1, ε = 10⁻⁵.

The implementation, pinned at revision e18c42acdfb7, not yet public