- in
- [B batch]
- out
- [B batch, 768 features]
- parameters
- 2,361,600
Parameters
2,361,600 parameters per instance, across its learned components
| Component | Calculation | Parameters |
|---|---|---|
| Expansion | 1,181,184 | |
| Contraction | 1,180,416 | |
| Per-instance total | 2,361,600 | |
Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.
Explanation
The time is the same for every position of a sequence, so the condition has no position axis.
Sources4
Direct evidence
The timestep embedding: cosines and sines of the time multiplied by frequencies falling geometrically from 1 to nearly 1/max_period.
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
The denoiser owns the time embedding’s two affine layers, one rotary table for all blocks, an affine projection of the concatenated input, and the stack.
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
The time embedding passes its MLP and a SiLU; the noised latents are concatenated with the previous clean estimate, zeros when there is none, and projected; the stack runs in bfloat16.
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
Diffusion: self-conditioning on, `t_min` 0.05 and `eps` 10⁻⁵; the adaptive time sampler has 100 bins, a minimum weight of 0.0001, an EMA of 0.9 and updates every 5,000 steps; the denoiser’s EMA decay is 0.9999; scheduler `tanh` with d = 3, solver `euler`, `ode_sampling` false, N = 128 by default; the denoiser has 12 blocks of width 768 with 12 heads, time conditioning `adaln`, a standard MLP, softcap 50 and self-conditioning by concatenation; the timestep embedding’s `max_period` is 10.
The implementation, pinned at revision e18c42acdfb7, not yet public