LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

Add noise

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

TrainingGeneration

Add noise

Sample a time

×

no parameters

Draw one diffusion time per sequence from the adaptive sampler’s 100 bins.

Why

Adaptive. The noise schedule matters for a diffusion model, and the best one depends on the latent space. Here the latent space changes during training, so a fixed schedule is hard to tune: the schedule is adapted while training. LDLM paper §5.3

The aim is a denoising loss that grows linearly with the sampled time, so that all times contribute equally to reducing uncertainty. Sampling times with a density proportional to the loss’s derivative gives that; the derivative is approximated on 100 equal intervals from a moving average of the loss at their edges. LDLM paper §5.3

In the ablation it brings the generative perplexity from 171.9 to 98.5 and Mauve from 11.8 to 81.7 with entropy unchanged. LDLM paper §6

i∼pi\sim p
t∼U(ui,ui+1){\htmlData{light=time sample-time}{t}}\sim\mathcal U(u_i,u_{i+1})
pi∝∣Lˉ(ui+1)−Lˉ(ui)∣p_i\propto\lvert\bar{\mathcal L}(u_{i+1})-\bar{\mathcal L}(u_i)\rvert
out
tt [B batch]
Explanation

The bins divide the times from 0.00001 to 1 equally. Until step 50,000 their weights are equal, so the time is uniform. That step is the end of the warm-up of the encoder’s gradient too: the sampler’s warm_up is set to diffusion_warmup. At that step and from then on, every 5,000 steps the weights are set to the absolute differences of the loss between neighbouring bin edges, divided by their sum, raised to at least 0.0001 and divided by their sum again.

Lˉ(ui)\bar{\mathcal L}(u_i) is the diffusion loss at a bin edge, measured on one validation batch per process with the averaged denoiser, the current encoder and no self-conditioning, and smoothed across updates with decay 0.9.

Sources6

Direct evidence

Times are drawn by the adaptive sampler in resampling mode, otherwise uniformly.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The trainer’s `diffusion_t_min` is the configuration’s `diffusion.eps`; the adaptive time sampler runs in resampling mode from that value to T.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The sampler divides the time range into equal bins, draws a bin by its weight and a time uniformly inside it.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The bin weights are the absolute differences of the smoothed loss at neighbouring bin edges, normalized, with a floor.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The loss at the bin edges is measured on one validation batch with the averaged denoiser and no self-conditioning.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

Diffusion: self-conditioning on, `t_min` 0.05 and `eps` 10⁻⁵; the adaptive time sampler has 100 bins, a minimum weight of 0.0001, an EMA of 0.9 and updates every 5,000 steps; the denoiser’s EMA decay is 0.9999; scheduler `tanh` with d = 3, solver `euler`, `ode_sampling` false, N = 128 by default; the denoiser has 12 blocks of width 768 with 12 heads, time conditioning `adaln`, a standard MLP, softcap 50 and self-conditioning by concatenation; the timestep embedding’s `max_period` is 10.

The implementation, pinned at revision e18c42acdfb7, not yet public