LDLMIn construction

All models, with their blocks →

Size

306M trained

Published

Meshchaninov et al. · May 2026

Line

Diffusion language model

View

Architecture

Source

LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verifiedHow to read

Time embedding

Training Generationcall1

t[B]Sinusoidal encoding Expansion

e^{\uparrow}=e_tW_{1}+b_{1}

SiLU Contraction

e^{\downarrow}=e^{\sigma}W_{2}+b_{2}

SiLU a_t[B, 768]

Sampler › Denoiser › Time embedding

Diffusion time

tensor

One time between 0 and 1 for each sequence: 1 is pure noise.

shape: $t$ [B batch]

Sources

Contextual evidence from Adaptive time grid

The generation grid: N + 1 times at equal steps of the weights’ cumulative distribution, interpolated inside bins, clamped to the stopping time, in decreasing order.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Adaptive time grid

Sampling starts from the prior with a zero self-conditioning estimate, uses the averaged denoiser, takes the adaptive grid when it is requested and the sampler has been updated and a uniform grid otherwise, steps N times carrying the state and the clean estimate, and returns the last step’s mean.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Adaptive time grid

Diffusion: self-conditioning on, `t_min` 0.05 and `eps` 10⁻⁵; the adaptive time sampler has 100 bins, a minimum weight of 0.0001, an EMA of 0.9 and updates every 5,000 steps; the denoiser’s EMA decay is 0.9999; scheduler `tanh` with d = 3, solver `euler`, `ode_sampling` false, N = 128 by default; the denoiser has 12 blocks of width 768 with 12 heads, time conditioning `adaln`, a standard MLP, softcap 50 and self-conditioning by concatenation; the timestep embedding’s `max_period` is 10.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Adaptive time grid

Generation: sequences are not truncated at the end token, and the trained adaptive time grid is used.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Adaptive time grid

The documented OpenWebText invocation: the joint task on 1,024-token sequences, decoder noise 3, hidden state −3, 1,024 diffusion steps on the adaptive grid.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Sinusoidal encoding

The timestep embedding: cosines and sines of the time multiplied by frequencies falling geometrically from 1 to nearly 1/max_period.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Sinusoidal encoding

The time embedding passes its MLP and a SiLU; the noised latents are concatenated with the previous clean estimate, zeros when there is none, and projected; the stack runs in bfloat16.

The implementation, pinned at revision e18c42acdfb7, not yet public