LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

Latent decoder

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

TrainingGeneration
in
z~0\tilde z_0 [B batch, T positions, 768 d]
out
h^\hat h [B batch, T positions, 768 d]
parameters
55,098,650
Parameters

55,098,650 parameters per instance, across its learned components

Exact per-instance accounting
ComponentCalculationParameters
Position table1,024×7681{,}024 \times 768786,432
Mask vector768768768
Query MLP4,723,968 across its components4,723,968
Residual scale1+11 + 12
Decoder block8,264,580×68{,}264{,}580 \times 649,587,480
Per-instance total55,098,650

Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.

Explanation

Every position starts from the same vector, the frozen embedding of <mask>; positions tell them apart, and everything else they read from the latents.

There is no normalization after the last block: the output is compared with the standardized features as it is.

Sources4

Direct evidence

The latent decoder owns one position table, the mask token’s vector copied from the token table of a GPT-2 loaded for the purpose and frozen because the feature model is configured frozen, an MLP for the queries, the blocks and a scale for the prepared queries.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

Every query is the mask vector: it passes the MLP and a scale, positions are added; the latents take the same position table; the blocks are applied; every position counts as masked.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The latent decoder: 12 heads of 64 features, query and key normalization, softcap 50, 6 layers, MLP multiplier 4; the token decoder has 3 blocks.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The feature model is GPT-2, frozen; the hidden state at index −3 is extracted; the encoder and the decoder are of the Perceiver kind; the file’s vocabulary size and mask token ID are defaults that the trainer overwrites with the tokenizer’s; there are as many latents as positions, of 768 features.

The implementation, pinned at revision e18c42acdfb7, not yet public