Llama 3.1

All models, with previews →

Meta · Jul 2024Dense decoder · follows Llama 3

Reference code meta-llama/llama3 @ a0940f9, which computes as Llama 3.1 but for its scaled RoPE (meta-llama/llama-models @ 0e0b8c5, model.py 45–69); configuration meta-llama/llama-models @ 0e0b8c5 · checkpoint, tokenizer and tensors not verified · How to read

Token embedding

×

learnedself.tok_embeddings(tokens)

Look up a learned feature vector for each token ID.

x0=E[I]{\htmlData{light=embedded embedding}{x_0}}=E[{\htmlData{light=token-ids tokenize}{I}}]
in
II [B batch, T positions]
out
x0x_0 [B batch, T positions, 4096 d]
parameters
525,336,576

Position enters only through RoPE on queries and keys.

Parameters

525,336,576 parameters per instance.

Exact per-instance accounting
ComponentCalculationParameters
Token embedding128,256×4,096128{,}256 \times 4{,}096525,336,576
Per-instance total525,336,576

The vocabulary projection owns a separate matrix.

Derived from the configured table dimensions; checkpoint tensors have not been independently counted.

Codemodel.py 253–260, 279–280 · sku_list.py 19, 231–248
model.py 253–260
        super().__init__()
        self.params = params
        self.vocab_size = params.vocab_size
        self.n_layers = params.n_layers

        self.tok_embeddings = VocabParallelEmbedding(
            params.vocab_size, params.dim, init_method=lambda x: x
        )
model.py 279–280
        _bsz, seqlen = tokens.shape
        h = self.tok_embeddings(tokens)
sku_list.py 19
LLAMA3_VOCAB_SIZE = 128256
sku_list.py 231–248
        Model(
            core_model_id=CoreModelId.llama3_1_8b,
            description="Llama 3.1 8b model",
            huggingface_repo="meta-llama/Llama-3.1-8B",
            arch_args={
                "dim": 4096,
                "n_layers": 32,
                "n_heads": 32,
                "n_kv_heads": 8,
                "vocab_size": LLAMA3_VOCAB_SIZE,
                "ffn_dim_multiplier": 1.3,
                "multiple_of": 1024,
                "norm_eps": 1e-05,
                "rope_theta": 500000.0,
                "use_scaled_rope": True,
            },
            pth_file_count=1,
        ),
Sources2

Direct evidence

The model constructs a vocabulary-by-feature token embedding and looks up the supplied token IDs.

model-embedding · meta-llama/llama3 / llama/model.py
Pinned revision a0940f9cf706 ↗ · lines 253–280

Read code excerpt
        super().__init__()
        self.params = params
        self.vocab_size = params.vocab_size
        self.n_layers = params.n_layers

        self.tok_embeddings = VocabParallelEmbedding(
            params.vocab_size, params.dim, init_method=lambda x: x
        )

… lines 261–278 omitted …

        _bsz, seqlen = tokens.shape
        h = self.tok_embeddings(tokens)

Direct evidence

Meta's Llama 3.1 8B base catalogue entry supplies dimensions, head counts, the vocabulary (LLAMA3_VOCAB_SIZE, line 19), MLP rounding, epsilon, rope_theta and use_scaled_rope: True.

config · meta-llama/llama-models / models/sku_list.py
Pinned revision 0e0b8c519242 ↗ · lines 19–248

Read code excerpt
LLAMA3_VOCAB_SIZE = 128256

… lines 20–230 omitted …

        Model(
            core_model_id=CoreModelId.llama3_1_8b,
            description="Llama 3.1 8b model",
            huggingface_repo="meta-llama/Llama-3.1-8B",
            arch_args={
                "dim": 4096,
                "n_layers": 32,
                "n_heads": 32,
                "n_kv_heads": 8,
                "vocab_size": LLAMA3_VOCAB_SIZE,
                "ffn_dim_multiplier": 1.3,
                "multiple_of": 1024,
                "norm_eps": 1e-05,
                "rope_theta": 500000.0,
                "use_scaled_rope": True,
            },
            pth_file_count=1,
        ),