Latent decoder
Position table
×learned
A learned vector for each of the 1,024 positions, scaled by 768^(−1/2).
- out
- [B batch, T positions, 768 d]
- parameters
- 786,432
One table, added both to the latents and to the queries: the latent width equals the embedding width, so no second table is built.
Parameters
786,432 parameters owned by this component
| Component | Calculation | Parameters |
|---|---|---|
| table | 786,432 | |
| Per-instance total | 786,432 | |
Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.
1,024 rows of 768 features.
Explanation
The rows are multiplied by before they are added. They are the same for every sequence of the batch.
Sources3
Direct evidence
The position table is a learned embedding whose rows are scaled by width^(−1/2).
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
The latent decoder owns one position table, the mask token’s vector copied from the token table of a GPT-2 loaded for the purpose and frozen because the feature model is configured frozen, an MLP for the queries, the blocks and a scale for the prepared queries.
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
Every query is the mask vector: it passes the MLP and a scale, positions are added; the latents take the same position table; the blocks are applied; every position counts as masked.
The implementation, pinned at revision e18c42acdfb7, not yet public