LDLMIn construction

All models, with their blocks →

Size
306M trained
Published
Meshchaninov et al. · May 2026
Line
Diffusion language model
View
Architecture
Source

LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verifiedHow to read

Attention over both streams

TrainingGeneration
in
QQ [B batch, 12 heads, T positions, 64 head features]KK [B batch, 12 heads, 2T positions, 64 head features]VV [B batch, 12 heads, 2T positions, 64 head features]
out
OO [B batch, 12 heads, T positions, 64 head features]
mode
generation
Explanation

The fused kernel does not store the score and weight fields; the plate states its arithmetic step by step.

Sources2

Direct evidence

Both streams are normalized; queries come from one stream, keys and values from both, concatenated along positions; queries and keys are normalized per head; attention runs over all positions with softcap; the heads are merged and projected.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The latent decoder: 12 heads of 64 features, query and key normalization, softcap 50, 6 layers, MLP multiplier 4; the token decoder has 3 blocks.

The implementation, pinned at revision e18c42acdfb7, not yet public