LDLMIn construction

Meshchaninov et al. · May 2026Diffusion language model

Denoiser

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

z_t[B, T, 768]z^sc₀[B, T, 768]t[B]Joinalong features Input projection

x_0=[\,z_t\;\hat z_0^{\mathrm{sc}}\,]W_{\mathrm{in}}+b_{\mathrm{in}}

x₀

Time embedding Open →

a_t[B, 768]

12 denoiser blocks Open →

x₁₂ Final LayerNorm

\mu,\;\sigma^2

Time projection

a_tW_o+c_o

Modulate

\bar x\odot(1+s_o)+b_o

Output projection

\hat z_0=x_{\mathrm{out}}W_{\mathrm{out}}+b_{\mathrm{out}}

z₀[B, T, 768]

Denoiser

Time projection

learned

An affine layer turns the time condition into a shift and a scale of 768 features for the output.

{\htmlData{light=final-projection}{m_{\mathrm{out}}}}=[\,b_o\;s_o\,]={\htmlData{light=condition time-embedding}{a_t}}W_{o}+c_{o}

in: $a_t$ [B batch, 768 features]
out: $m_{\mathrm{out}}$ [B batch, 1 every position, 1536 two vectors of 768]
parameters: 1,181,184

Parameters

1,181,184 parameters owned by this component

Exact per-instance accounting
Component	Calculation	Parameters
weights + bias	$768 \times 1{,}536 + 1{,}536$	1,181,184
Per-instance total		1,181,184

Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.

A 768 × 1,536 matrix and a bias of 1,536, both zero at initialization.

Sources1

Direct evidence

The output layer: gain-only LayerNorm, a shift and a scale from the time condition, and an affine projection initialized at zero.

The implementation, pinned at revision e18c42acdfb7, not yet public