LDLMIn construction

Contextual evidence from Latent encoder

The encoder owns the learned latent queries, a position table, an input and an output RMSNorm and the blocks.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Latent encoder

The encoder adds positions to the features, normalizes the learned queries, applies the blocks, each attending to the features, and normalizes the result.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Latent encoder

The latent encoder: 12 heads of 64 features, query and key normalization, softcap 50, 6 layers, MLP multiplier 4, dropout 0.1, ε = 10⁻⁵.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Latent encoder

The feature model is GPT-2, frozen; the hidden state at index −3 is extracted; the encoder and the decoder are of the Perceiver kind; the file’s vocabulary size and mask token ID are defaults that the trainer overwrites with the tokenizer’s; there are as many latents as positions, of 768 features.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Decoder noise

Reconstruction: Gaussian noise of fixed standard deviation is added to the latents, the latent decoder runs, the token decoder reads its output detached, and the cross-entropy against the token IDs is taken; the latent decoder’s output is also compared with the standardized features by a mean squared error.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Decoder noise

Joint training: decoder noise of standard deviation 3; weights 1 for the three losses; the diffusion gradient into the encoder rises from 0.001 over 50,000 steps by a straight-through scaling; the token decoder’s input is detached; AdamW at 3·10⁻⁴ for 1,000,000 steps of 512 sequences, gradients clipped at norm 1.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Add noise

The diffusion gradient into the encoder: the weight rises from its minimum by a sigmoid schedule during warm-up, and the clean latents become w · z + (1 − w) · detach(z), unchanged in value.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Add noise

Diffusion: a time is sampled, the latents are noised by the marginal, with probability one half a first call without gradient gives the self-conditioning estimate, the denoiser predicts the clean latents, and the mean squared error against them is taken; the three losses are summed with their weights.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Add noise

The marginal: x_t = μ x_0 + σ ε.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Add noise

The tangent schedule: α = 1/√(1 + d² tan²(πt/2)), σ = √(1 − α²); β(t) is evaluated with t clipped at 0.95.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Add noise

Times are drawn by the adaptive sampler in resampling mode, otherwise uniformly.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Add noise

Diffusion: self-conditioning on, `t_min` 0.05 and `eps` 10⁻⁵; the adaptive time sampler has 100 bins, a minimum weight of 0.0001, an EMA of 0.9 and updates every 5,000 steps; the denoiser’s EMA decay is 0.9999; scheduler `tanh` with d = 3, solver `euler`, `ode_sampling` false, N = 128 by default; the denoiser has 12 blocks of width 768 with 12 heads, time conditioning `adaln`, a standard MLP, softcap 50 and self-conditioning by concatenation; the timestep embedding’s `max_period` is 10.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Encoder gradient scale

The warm-up schedule: a sigmoid centred at 0.8 of the warm-up, normalized to run from 0 to 1.

The implementation, pinned at revision e18c42acdfb7, not yet public