Reconstructed features
×tensor
The latent decoder’s reconstruction of the standardized features.
- shape
- [B batch, T positions, 768 d]
Sources
Contextual evidence from Latent decoder
The latent decoder owns one position table, the mask token’s vector copied from the token table of a GPT-2 loaded for the purpose and frozen because the feature model is configured frozen, an MLP for the queries, the blocks and a scale for the prepared queries.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Latent decoder
Every query is the mask vector: it passes the MLP and a scale, positions are added; the latents take the same position table; the blocks are applied; every position counts as masked.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Latent decoder
The latent decoder: 12 heads of 64 features, query and key normalization, softcap 50, 6 layers, MLP multiplier 4; the token decoder has 3 blocks.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Latent decoder
The feature model is GPT-2, frozen; the hidden state at index −3 is extracted; the encoder and the decoder are of the Perceiver kind; the file’s vocabulary size and mask token ID are defaults that the trainer overwrites with the tokenizer’s; there are as many latents as positions, of 768 features.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Token decoder
The token decoder: a rotary table, the configured number of blocks, and a linear readout to the vocabulary without bias.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from Token decoder
The token decoder’s blocks: no softcap, no time conditioning, the gated MLP with the configured multiplier.
The implementation, pinned at revision e18c42acdfb7, not yet public