Latent decoder › 6 decoder blocks
Residual scale
×learned
Scale the attention update by one learned number.
- in
- [B batch, T positions, 768 d]
- out
- [B batch, T positions, 768 d]
- parameters
- 2 · × 6 blocks = 12
mask_of_mask_tokens is all ones, so the scale is scale_masked at every position.
Parameters
2 parameters owned by this component
| Component | Calculation | Parameters |
|---|---|---|
| learned vector | 1 | |
| learned vector | 1 | |
| Per-instance total | 2 | |
Across 6 independent blocks: parameters.
Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.
Two scalars, one for masked positions and one for unmasked ones, both zero at initialization. Every position of the decoder counts as masked, so the second cancels out of the expression: it is owned, handed to the optimizer, and receives no gradient.
Sources3
Direct evidence
The decoder’s residual scale: x · (1 + s), with one learned scalar for masked positions and one for unmasked ones, both zero at initialization.
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
A decoder block: Perceiver attention with the queries’ stream first read from the decoder’s state and keys and values from the latents and the state, a scaled residual, an MLP and a second scaled residual.
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
Every query is the mask vector: it passes the MLP and a scale, positions are added; the latents take the same position table; the blocks are applied; every position counts as masked.
The implementation, pinned at revision e18c42acdfb7, not yet public