Denoiser › 12 denoiser blocks
Time projection
×learned
This block’s affine layer turns the time condition into six vectors of 768 features: a shift, a scale and a gate for each of the two branches.
- in
- [B batch, 768 features]
- out
- [B batch, 1 every position, 4608 six vectors of 768]
- parameters
- 3,543,552 · × 12 blocks = 42,522,624
Parameters
3,543,552 parameters owned by this component
| Component | Calculation | Parameters |
|---|---|---|
| weights + bias | 3,543,552 | |
| Per-instance total | 3,543,552 | |
Across 12 independent blocks: parameters.
Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.
A 768 × 4,608 matrix and a bias of 4,608, both zero at initialization.
Explanation
Each block owns its projection, so each block reads the time in its own way.
Sources2
Direct evidence
A block owns two gain-only LayerNorms, a fused query–key–value projection and an output projection without bias, an MLP of the configured kind, and, with modulation, an affine layer from the time condition to six vectors, initialized at zero.
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
A block normalizes, modulates by the time condition’s shift and scale, projects to queries, keys and values, rotates queries and keys, attends over all positions with softcap, and adds the projected output scaled by a gate; the MLP branch does the same with its own shift, scale and gate.
The implementation, pinned at revision e18c42acdfb7, not yet public