LDLMIn construction

Meshchaninov et al. · May 2026Diffusion language model

MLP

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

u^′_n[B, T, 768]LayerNorm

\mu,\;\sigma^2

gain · bias Expansion

h=\bar u'_n\,W_{1}+b_{1}

GELU

g=h\,\Phi(h)

g[B, T, 3072]Contraction

\Delta_{\mathrm{mlp}}=g\,W_{2}+b_{2}

Δ_mlp[B, T, 768]

Latent encoder › 6 encoder blocks › MLP

MLP output

tensor

The output of the block’s MLP.

shape: $\Delta_{\mathrm{mlp}}$ [B batch, T positions, 768 d]

Sources

Contextual evidence from Add MLP update

An encoder block adds the Perceiver attention’s output to the latents, then adds the MLP’s output.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Contraction

The Perceiver blocks’ MLP: LayerNorm, an affine layer, exact GELU, dropout, an affine layer.

The implementation, pinned at revision e18c42acdfb7, not yet public