LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

Sampler

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

TrainingGeneration
Latents at time tz[B, T, 768]Previous estimateẑsc[B, T, 768]Adaptive time grid1,024 stepst[B]
DenoiserOpen →
z0[B, T, 768]
ϵ^\hat\epsilon==(zt−αtz^0)/σt({\htmlData{light=state}{z_t}}-\alpha_t{\htmlData{light=prediction denoiser}{\hat z_0}})/\sigma_tss==−ϵ^/σt-\hat\epsilon/\sigma_tff==−12βtzt−βts-\tfrac12\beta_t{\htmlData{light=state}{z_t}}-\beta_tsmt′{\htmlData{light=step-mean mean}{m_{t'}}}==zt+f (t′−t){\htmlData{light=state}{z_t}}+f\,(t'-{\htmlData{light=time grid}{t}})zt′{\htmlData{light=next-state diffuse}{z_{t'}}}==mt′+βt (t−t′)  ξ{\htmlData{light=step-mean mean}{m_{t'}}}+\sqrt{\beta_t\,({\htmlData{light=time grid}{t}}-t')}\;\xi
mt′[B, T, 768]zt′[B, T, 768]starts as Gaussian noisereplaced by the next latentsstarts as zerosreplaced by this call’s predictionthe next call’s latentsreturned by the last callcall 1 of 1,024 · m: the step’s mean, before its noise · t′: the grid’s next time · α, σ, β: the schedule at t · ξ: fresh Gaussian noise

Sampler

Adaptive time grid

×

no parameters

1,025 times from 1 down to 0.05, closer together where the diffusion loss changed fastest during training.

Why

The paper adapts the distribution of times during training, so that the denoising loss grows linearly with the sampled time. It does not describe the grid used at generation: that the same bin weights space the sampler’s calls is the implementation’s, with no reason given in the paper. LDLM paper §5.3

tk=max⁡ ⁣(F−1(1−k/N),  0.05)t_k=\max\!\big(F^{-1}(1-{\htmlClass{vl-stream-k}{k}}/N),\;0.05\big)
F(ui+1)=∑j≤ipj and linear between edgesF(u_{i+1})=\textstyle\sum_{j\le i}p_j\ \text{and linear between edges}
k=0,…,N{\htmlClass{vl-stream-k}{k}}=0,\ldots,N
N=1,024N=1,024
out
t0,…,tNt_{0},\ldots,t_{N} [1025 grid points]

The bins cover the times from 0.00001 to 1, and grid points below 0.05 are raised to 0.05; the last always is. A step between two raised points has zero length: it leaves the latents as they are, and still calls the denoiser and replaces the previous estimate. How many such steps there are depends on the saved bin weights, which were not read. The adaptive grid is taken when it is requested, as the documented invocation does, and the sampler has been updated; otherwise the times are 1,025 evenly spaced ones.

Explanation

pip_i is the weight of the bin between the edges uiu_i and ui+1u_{i+1}, and FF is their cumulative distribution. NN is 1,024 in the documented OpenWebText invocation; the configuration’s default is 128.

Sources5

Direct evidence

The generation grid: N + 1 times at equal steps of the weights’ cumulative distribution, interpolated inside bins, clamped to the stopping time, in decreasing order.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

Sampling starts from the prior with a zero self-conditioning estimate, uses the averaged denoiser, takes the adaptive grid when it is requested and the sampler has been updated and a uniform grid otherwise, steps N times carrying the state and the clean estimate, and returns the last step’s mean.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

Diffusion: self-conditioning on, `t_min` 0.05 and `eps` 10⁻⁵; the adaptive time sampler has 100 bins, a minimum weight of 0.0001, an EMA of 0.9 and updates every 5,000 steps; the denoiser’s EMA decay is 0.9999; scheduler `tanh` with d = 3, solver `euler`, `ode_sampling` false, N = 128 by default; the denoiser has 12 blocks of width 768 with 12 heads, time conditioning `adaln`, a standard MLP, softcap 50 and self-conditioning by concatenation; the timestep embedding’s `max_period` is 10.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

Generation: sequences are not truncated at the end token, and the trained adaptive time grid is used.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The documented OpenWebText invocation: the joint task on 1,024-token sequences, decoder noise 3, hidden state −3, 1,024 diffusion steps on the adaptive grid.

The implementation, pinned at revision e18c42acdfb7, not yet public