LDLMIn construction

Size

306M trained

Published

Meshchaninov et al. · May 2026

Line

Diffusion language model

View

Architecture

Source

LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verifiedHow to read

Gated MLP

g=y_{\mathrm{mlp}}W_{\mathrm{gate}}

u=y_{\mathrm{mlp}}W_{\mathrm{up}}

a = \operatorname{SiLU}(g)

h = a \odot u

\Delta_{\mathrm{mlp}}=hW_{\mathrm{down}}

Token decoder › 3 blocks › Gated MLP

tensor

The gated features supplied to the down projection.

Sources

Contextual evidence from Element-wise product

The gated MLP: down(SiLU(gate x) · up x), three projections without bias.

The implementation, pinned at revision e18c42acdfb7, not yet public