LDLMIn construction

Meshchaninov et al. · May 2026Diffusion language model

Gated MLP

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

y_mlp[B, T, 768]Gate projection

g=y_{\mathrm{mlp}}W_{\mathrm{gate}}

g[B, T, 3072]Up projection

u=y_{\mathrm{mlp}}W_{\mathrm{up}}

u[B, T, 3072]SiLU

a = \operatorname{SiLU}(g)

a[B, T, 3072]Element-wise product

h = a \odot u

h[B, T, 3072]Down projection

\Delta_{\mathrm{mlp}}=hW_{\mathrm{down}}

Δ_mlp[B, T, 768]

Token decoder › 3 blocks › Gated MLP

Up projection

learned

Map 768 input features to 3,072 features in parallel with the gate.

{\htmlData{light=u up}{u}}={\htmlData{light=x}{y_{\mathrm{mlp}}}}W_{\mathrm{up}}

in: $y_{\mathrm{mlp}}$ [B batch, T positions, 768 d]
out: $u$ [B batch, T positions, 3072 d_ff]
parameters: 2,359,296 · × 3 blocks = 7,077,888

Parameters

2,359,296 parameters per instance.

Exact per-instance accounting
Component	Calculation	Parameters
Weight matrix	$768 \times 3{,}072$	2,359,296
Per-instance total		2,359,296

This projection has no bias parameter.

The reference module holds the weight output by input, [out, in].

Across 3 independent blocks: $2{,}359{,}296 \times 3 = 7{,}077{,}888$ parameters.

Derived from the configured matrix dimensions; checkpoint tensors have not been independently counted.

Sources1

Direct evidence

The gated MLP: down(SiLU(gate x) · up x), three projections without bias.

The implementation, pinned at revision e18c42acdfb7, not yet public