LDLMIn construction

Meshchaninov et al. · May 2026Diffusion language model

Denoiser

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

z_t[B, T, 768]z^sc₀[B, T, 768]t[B]Joinalong features Input projection

x_0=[\,z_t\;\hat z_0^{\mathrm{sc}}\,]W_{\mathrm{in}}+b_{\mathrm{in}}

x₀

Time embedding Open →

a_t[B, 768]

12 denoiser blocks Open →

x₁₂ Final LayerNorm

\mu,\;\sigma^2

Time projection

a_tW_o+c_o

Modulate

\bar x\odot(1+s_o)+b_o

Output projection

\hat z_0=x_{\mathrm{out}}W_{\mathrm{out}}+b_{\mathrm{out}}

z₀[B, T, 768]

Denoiser

Output projection

learned

Map the modulated features to the predicted clean latents.

{\htmlData{light=output output-projection}{\hat z_0}}={\htmlData{light=final-modulate}{x_{\mathrm{out}}}}W_{\mathrm{out}}+b_{\mathrm{out}}

in: $x_{\mathrm{out}}$ [B batch, T positions, 768 d]
out: $\hat z_0$ [B batch, T positions, 768 d]
parameters: 590,592

Parameters

590,592 parameters per instance.

Exact per-instance accounting
Component	Calculation	Parameters
Weight matrix	$768 \times 768$	589,824
Output bias	$768$	768
Per-instance total		590,592

The reference module holds the weight output by input, [out, in].

Derived from the configured matrix dimensions; checkpoint tensors have not been independently counted.

Explanation

Its weights and bias start at zero, so an untrained denoiser predicts zero latents.

Sources1

Direct evidence

The output layer: gain-only LayerNorm, a shift and a scale from the time condition, and an affine projection initialized at zero.

The implementation, pinned at revision e18c42acdfb7, not yet public