LDLMIn construction

All models, with their blocks →

Size

306M trained

Published

Meshchaninov et al. · May 2026

Line

Diffusion language model

View

Architecture

Source

LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verifiedHow to read

GPT-2 Small

Training Generation

Token IDs[B, T]Frozen GPT-2hidden state −3 h^raw[B, T, 768]Feature statisticsfixed assets Standardize

(h^{\mathrm{raw}}-\mu)/\sigma

h[B, T, 768]

GPT-2 Small

Feature statistics

no parameters

A mean and a standard deviation for each of the 768 features, measured over the validation set before training and saved with the checkpoint.

out: $\mu,\,\sigma$ [2 mean, standard deviation, 768 features]

Explanation

They are measured once, over at most 1,000 validation batches in each process, on valid positions only, with the variance clamped from below at 10⁻⁴. They are tensors the trainer holds and writes into the checkpoint, not parameters: nothing updates them. Their values in the released checkpoint were not read.

Sources2

Contextual evidence

The feature statistics are a mean and a standard deviation per feature over valid positions of the validation set, the variance clamped at 10⁻⁴; features are standardized with them.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence

Training computes the statistics once before the first step; they are saved with the checkpoint.

The implementation, pinned at revision e18c42acdfb7, not yet public