LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

Sampler

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

TrainingGeneration
Latents at time tz[B, T, 768]Previous estimateẑsc[B, T, 768]Adaptive time grid1,024 stepst[B]
DenoiserOpen →
z0[B, T, 768]
ϵ^\hat\epsilon==(zt−αtz^0)/σt({\htmlData{light=state}{z_t}}-\alpha_t{\htmlData{light=prediction denoiser}{\hat z_0}})/\sigma_tss==−ϵ^/σt-\hat\epsilon/\sigma_tff==−12βtzt−βts-\tfrac12\beta_t{\htmlData{light=state}{z_t}}-\beta_tsmt′{\htmlData{light=step-mean mean}{m_{t'}}}==zt+f (t′−t){\htmlData{light=state}{z_t}}+f\,(t'-{\htmlData{light=time grid}{t}})zt′{\htmlData{light=next-state diffuse}{z_{t'}}}==mt′+βt (t−t′)  ξ{\htmlData{light=step-mean mean}{m_{t'}}}+\sqrt{\beta_t\,({\htmlData{light=time grid}{t}}-t')}\;\xi
mt′[B, T, 768]zt′[B, T, 768]starts as Gaussian noisereplaced by the next latentsstarts as zerosreplaced by this call’s predictionthe next call’s latentsreturned by the last callcall 1 of 1,024 · m: the step’s mean, before its noise · t′: the grid’s next time · α, σ, β: the schedule at t · ξ: fresh Gaussian noise

Sampler

Reverse drift

×

no parameters

The deterministic part of the reverse stochastic differential equation at the current time.

f=−12 βt zt−βt s{\htmlData{light=drift}{f}}=-\tfrac12\,\beta_t\,{\htmlData{light=state}{z_t}}-\beta_t\,{\htmlData{light=score}{s}}
βt=π 9 τ (1+τ2)1+9 τ2\beta_t=\frac{\pi\,9\,\tau\,(1+\tau^2)}{1+9\,\tau^2}
τ=tan⁡ ⁣(π2min⁡(t, 0.95))\tau=\tan\!\big(\tfrac{\pi}{2}\min({\htmlData{light=time grid}{t}},\,0.95)\big)
in
ztz_t [B batch, T positions, 768 d]ss [B batch, T positions, 768 d]tt [B batch]
out
ff [B batch, T positions, 768 d]
Explanation

The rate is evaluated with the time clipped at 0.95: for the first steps, above that time, it keeps its value at 0.95.

The stochastic sampler is used: ode_sampling is false.

Sources3

Direct evidence

The reverse SDE: drift −β x/2 − β · score and diffusion √β; the ODE halves the score term and has no diffusion.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The tangent schedule: α = 1/√(1 + d² tan²(πt/2)), σ = √(1 − α²); β(t) is evaluated with t clipped at 0.95.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

Diffusion: self-conditioning on, `t_min` 0.05 and `eps` 10⁻⁵; the adaptive time sampler has 100 bins, a minimum weight of 0.0001, an EMA of 0.9 and updates every 5,000 steps; the denoiser’s EMA decay is 0.9999; scheduler `tanh` with d = 3, solver `euler`, `ode_sampling` false, N = 128 by default; the denoiser has 12 blocks of width 768 with 12 heads, time conditioning `adaln`, a standard MLP, softcap 50 and self-conditioning by concatenation; the timestep embedding’s `max_period` is 10.

The implementation, pinned at revision e18c42acdfb7, not yet public