LDLMIn construction

Meshchaninov et al. · May 2026Diffusion language model

Gated MLP

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

y_mlp[B, T, 768]Gate projection

g=y_{\mathrm{mlp}}W_{\mathrm{gate}}

g[B, T, 3072]Up projection

u=y_{\mathrm{mlp}}W_{\mathrm{up}}

u[B, T, 3072]SiLU

a = \operatorname{SiLU}(g)

a[B, T, 3072]Element-wise product

h = a \odot u

h[B, T, 3072]Down projection

\Delta_{\mathrm{mlp}}=hW_{\mathrm{down}}

Δ_mlp[B, T, 768]

Token decoder › 3 blocks › Gated MLP

SwiGLU output

tensor

The SwiGLU output.

shape: $\Delta_{\mathrm{mlp}}$ [B batch, T positions, 768 d]

Sources

Contextual evidence from Add MLP update

The residual update is scale · dropout(x) added to the residual; modulation is x · (1 + scale) + shift.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Add MLP update

A block normalizes, modulates by the time condition’s shift and scale, projects to queries, keys and values, rotates queries and keys, attends over all positions with softcap, and adds the projected output scaled by a gate; the MLP branch does the same with its own shift, scale and gate.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Down projection

The gated MLP: down(SiLU(gate x) · up x), three projections without bias.

The implementation, pinned at revision e18c42acdfb7, not yet public