- in
- [B batch, T positions]
- out
- [B batch, T positions, 768 d]
- parameters
- 124,441,344
Parameters
124,441,344 parameters per instance, across its learned components
| Component | Calculation | Parameters |
|---|---|---|
| Frozen GPT-2 | 124,441,344 | |
| Per-instance total | 124,441,344 | |
Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.
Explanation
Nothing here is trained: GPT-2 is in evaluation mode and takes no gradient, and the statistics are measured once.
GPT-2, the latent encoder and the two decoders run under bfloat16 autocast. The Perceiver attention switches autocast off around its query and key normalization and its kernel; the queries, keys and values it hands them are already bfloat16.
The standardized features are both what the encoder reads and what the latent decoder is trained to reproduce.
Sources5
Direct evidence
With the feature model frozen, its hidden state is computed without gradient, standardized, and passed to the encoder; the same tensor is the target of the feature loss.
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
The feature model is wrapped to return one entry of its hidden states, chosen by index; GPT-2’s token table is resized to the configured vocabulary.
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
With an extraction layer other than −1 the encoder loads the wrapped feature model; when it is frozen it is put in evaluation mode and takes no gradient.
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
The feature statistics are a mean and a standard deviation per feature over valid positions of the validation set, the variance clamped at 10⁻⁴; features are standardized with them.
The implementation, pinned at revision e18c42acdfb7, not yet public
Direct evidence
The feature model is GPT-2, frozen; the hidden state at index −3 is extracted; the encoder and the decoder are of the Perceiver kind; the file’s vocabulary size and mask token ID are defaults that the trainer overwrites with the tokenizer’s; there are as many latents as positions, of 768 features.
The implementation, pinned at revision e18c42acdfb7, not yet public