Llama 3.1

Meta · Jul 2024Dense decoder · follows Llama 3

Merge and project

Reference code meta-llama/llama3 @ a0940f9, which computes as Llama 3.1 but for its scaled RoPE (meta-llama/llama-models @ 0e0b8c5, model.py 45–69); configuration meta-llama/llama-models @ 0e0b8c5 · checkpoint, tokenizer and tensors not verified · How to read

Full sequence Prefill DecodeP1,000

0 127 1,000 4,096 131,071

H[B, 32, 1, 128]Merge heads

32\times 128\;\to\;4096

H[B, 1, 4096]Output projection

\Delta_{\mathrm{attn}}=HW_O

Δ_attn[B, 1, 4096]

32 independent transformer blocks › Attention

Merged field

tensoroutput · 189

The head outputs joined along the feature axis.

shape: $H$ [B batch, 1 positions, 4096 d]

Codemodel.py 188–189

model.py 188–189

        output = torch.matmul(scores, values)  # (bs, n_local_heads, seqlen, head_dim)
        output = output.transpose(1, 2).contiguous().view(bsz, seqlen, -1)

Sources1

Contextual evidence

The head output is transposed and reshaped into the full model-width field.

attn-merge · meta-llama/llama3 / llama/model.py
Pinned revision a0940f9cf706 ↗ · lines 188–189

Read code excerpt

        output = torch.matmul(scores, values)  # (bs, n_local_heads, seqlen, head_dim)
        output = output.transpose(1, 2).contiguous().view(bsz, seqlen, -1)