LDLMIn construction

All models, with their blocks →

Size
306M trained
Published
Meshchaninov et al. · May 2026
Line
Diffusion language model
View
Architecture
Source

LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verifiedHow to read

Sampler

TrainingGeneration
Latents at time tz[B, T, 768]Previous estimateẑsc[B, T, 768]Adaptive time grid1,024 stepst[B]
DenoiserOpen →
ẑ0[B, T, 768]
ϵ^\hat\epsilon==(zt−αtz^0)/σt({\htmlData{light=state}{z_t}}-\alpha_t{\htmlData{light=prediction denoiser}{\hat z_0}})/\sigma_tss==−ϵ^/σt-\hat\epsilon/\sigma_tff==−12βtzt−βts-\tfrac12\beta_t{\htmlData{light=state}{z_t}}-\beta_tsmt′{\htmlData{light=step-mean mean}{m_{t'}}}==zt+f (t′−t){\htmlData{light=state}{z_t}}+f\,(t'-{\htmlData{light=time grid}{t}})zt′{\htmlData{light=next-state diffuse}{z_{t'}}}==mt′+βt (t−t′)  ξ{\htmlData{light=step-mean mean}{m_{t'}}}+\sqrt{\beta_t\,({\htmlData{light=time grid}{t}}-t')}\;\xi
mt′[B, T, 768]zt′[B, T, 768]starts as Gaussian noisereplaced by the next latentsstarts as zerosreplaced by this call’s predictionthe next call’s latentsreturned by the last callcall 1 of 1,024 · m: the step’s mean, before its noise · t′: the grid’s next time · α, σ, β: the schedule at t · ξ: fresh Gaussian noise

Sampler

Previous estimate

×

state · kept between calls

The clean latents as the previous call predicted them: zeros before the first call.

shape
z^0sc\hat z_0^{\mathrm{sc}} ẑ_sc [B batch, T positions, 768 d]

Replaced whole at every call by that call’s prediction.

Explanation

Training gives the denoiser such an estimate for half of the batches; the sampler gives it one at every call after the first.

Sources

Contextual evidence from Denoiser

The denoiser owns the time embedding’s two affine layers, one rotary table for all blocks, an affine projection of the concatenated input, and the stack.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Denoiser

The time embedding passes its MLP and a SiLU; the noised latents are concatenated with the previous clean estimate, zeros when there is none, and projected; the stack runs in bfloat16.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Denoiser

The denoiser’s stack applies the blocks in order and then the output layer, each with the time condition.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Denoiser

Diffusion: self-conditioning on, `t_min` 0.05 and `eps` 10⁻⁵; the adaptive time sampler has 100 bins, a minimum weight of 0.0001, an EMA of 0.9 and updates every 5,000 steps; the denoiser’s EMA decay is 0.9999; scheduler `tanh` with d = 3, solver `euler`, `ode_sampling` false, N = 128 by default; the denoiser has 12 blocks of width 768 with 12 heads, time conditioning `adaln`, a standard MLP, softcap 50 and self-conditioning by concatenation; the timestep embedding’s `max_period` is 10.

The implementation, pinned at revision e18c42acdfb7, not yet public