- in
- [B batch, T positions, 768 d]
- out
- [B batch, T positions, 50259 vocabulary]
- parameters
- 66,915,072
Parameters
66,915,072 parameters per instance, across its learned components
| Component | Calculation | Parameters |
|---|---|---|
| Token-decoder block | 28,316,160 | |
| Vocabulary readout | 38,598,912 | |
| Per-instance total | 66,915,072 | |
Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.
Explanation
There is no normalization between the last block and the readout.
The readout is a matrix of its own: it is tied neither to GPT-2’s token table nor to anything else.
Sources4
Direct evidence
The token decoder: a rotary table, the configured number of blocks, and a linear readout to the vocabulary without bias.
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
The token decoder’s blocks: no softcap, no time conditioning, the gated MLP with the configured multiplier.
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
The latent decoder: 12 heads of 64 features, query and key normalization, softcap 50, 6 layers, MLP multiplier 4; the token decoder has 3 blocks.
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
The feature model is GPT-2, frozen; the hidden state at index −3 is extracted; the encoder and the decoder are of the Perceiver kind; the file’s vocabulary size and mask token ID are defaults that the trainer overwrites with the tokenizer’s; there are as many latents as positions, of 768 features.
The implementation, pinned at revision e18c42acdfb7, not yet public