LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

Sampler

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generationcall1

Latents at time tz[B, T, 768]Previous estimateẑsc[B, T, 768]Adaptive time grid1,024 steps t[B]

Denoiser Open →

z₀[B, T, 768]

\hat\epsilon

=

({\htmlData{light=state}{z_t}}-\alpha_t{\htmlData{light=prediction denoiser}{\hat z_0}})/\sigma_t

s

=

-\hat\epsilon/\sigma_t

f

=

-\tfrac12\beta_t{\htmlData{light=state}{z_t}}-\beta_ts

{\htmlData{light=step-mean mean}{m_{t'}}}

=

{\htmlData{light=state}{z_t}}+f\,(t'-{\htmlData{light=time grid}{t}})

{\htmlData{light=next-state diffuse}{z_{t'}}}

=

{\htmlData{light=step-mean mean}{m_{t'}}}+\sqrt{\beta_t\,({\htmlData{light=time grid}{t}}-t')}\;\xi

m_t′[B, T, 768]z_t′[B, T, 768]starts as Gaussian noisereplaced by the next latentsstarts as zerosreplaced by this call’s predictionthe next call’s latentsreturned by the last callcall 1 of 1,024 · m: the step’s mean, before its noise · t′: the grid’s next time · α, σ, β: the schedule at t · ξ: fresh Gaussian noise

Sampler

Time of this call

tensor

The grid’s times for call 1: $t=t_{0}$ and the next, $t'=t_{1}$ , the same for every sequence. Their values are the adaptive grid’s, computed from bin weights saved with the checkpoint, which were not read: the plate shows the call’s number and no value of $t$ .

shape: $t$ [B batch]

Sources

Contextual evidence from Adaptive time grid

The generation grid: N + 1 times at equal steps of the weights’ cumulative distribution, interpolated inside bins, clamped to the stopping time, in decreasing order.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Adaptive time grid

Sampling starts from the prior with a zero self-conditioning estimate, uses the averaged denoiser, takes the adaptive grid when it is requested and the sampler has been updated and a uniform grid otherwise, steps N times carrying the state and the clean estimate, and returns the last step’s mean.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Adaptive time grid

Diffusion: self-conditioning on, `t_min` 0.05 and `eps` 10⁻⁵; the adaptive time sampler has 100 bins, a minimum weight of 0.0001, an EMA of 0.9 and updates every 5,000 steps; the denoiser’s EMA decay is 0.9999; scheduler `tanh` with d = 3, solver `euler`, `ode_sampling` false, N = 128 by default; the denoiser has 12 blocks of width 768 with 12 heads, time conditioning `adaln`, a standard MLP, softcap 50 and self-conditioning by concatenation; the timestep embedding’s `max_period` is 10.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Adaptive time grid

Generation: sequences are not truncated at the end token, and the trained adaptive time grid is used.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Adaptive time grid

The documented OpenWebText invocation: the joint task on 1,024-token sequences, decoder noise 3, hidden state −3, 1,024 diffusion steps on the adaptive grid.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Denoiser

The denoiser owns the time embedding’s two affine layers, one rotary table for all blocks, an affine projection of the concatenated input, and the stack.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Denoiser

The time embedding passes its MLP and a SiLU; the noised latents are concatenated with the previous clean estimate, zeros when there is none, and projected; the stack runs in bfloat16.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Denoiser

The denoiser’s stack applies the blocks in order and then the output layer, each with the time condition.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Noise estimate

The denoiser’s clean prediction is converted into the noise estimate and the score with the marginal’s mean and standard deviation.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Noise estimate

The tangent schedule: α = 1/√(1 + d² tan²(πt/2)), σ = √(1 − α²); β(t) is evaluated with t clipped at 0.95.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Reverse drift

The reverse SDE: drift −β x/2 − β · score and diffusion √β; the ODE halves the score term and has no diffusion.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Euler step

One Euler–Maruyama step: the mean is x + drift · dt, and the next state adds diffusion · √(−dt) · noise; the clean estimate is returned with them.

The implementation, pinned at revision e18c42acdfb7, not yet public