Denoiser › 12 denoiser blocks
Time condition
×supplied · no parameters
One vector for each sequence, made from the diffusion time; the same enters every block.
- shape
- [B batch, 768 features]
Sources
Contextual evidence from Time embedding
The timestep embedding: cosines and sines of the time multiplied by frequencies falling geometrically from 1 to nearly 1/max_period.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Time embedding
The denoiser owns the time embedding’s two affine layers, one rotary table for all blocks, an affine projection of the concatenated input, and the stack.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Time embedding
The time embedding passes its MLP and a SiLU; the noised latents are concatenated with the previous clean estimate, zeros when there is none, and projected; the stack runs in bfloat16.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Time embedding
Diffusion: self-conditioning on, `t_min` 0.05 and `eps` 10⁻⁵; the adaptive time sampler has 100 bins, a minimum weight of 0.0001, an EMA of 0.9 and updates every 5,000 steps; the denoiser’s EMA decay is 0.9999; scheduler `tanh` with d = 3, solver `euler`, `ode_sampling` false, N = 128 by default; the denoiser has 12 blocks of width 768 with 12 heads, time conditioning `adaln`, a standard MLP, softcap 50 and self-conditioning by concatenation; the timestep embedding’s `max_period` is 10.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Time projection
A block owns two gain-only LayerNorms, a fused query–key–value projection and an output projection without bias, an MLP of the configured kind, and, with modulation, an affine layer from the time condition to six vectors, initialized at zero.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Time projection
A block normalizes, modulates by the time condition’s shift and scale, projects to queries, keys and values, rotates queries and keys, attends over all positions with softcap, and adds the projected output scaled by a gate; the MLP branch does the same with its own shift, scale and gate.
The implementation, pinned at revision e18c42acdfb7, not yet public