Llama 3.1

All models, with previews →

Meta · Jul 2024Dense decoder · follows Llama 3

Merge and project

Reference code meta-llama/llama3 @ a0940f9, which computes as Llama 3.1 but for its scaled RoPE (meta-llama/llama-models @ 0e0b8c5, model.py 45–69); configuration meta-llama/llama-models @ 0e0b8c5 · checkpoint, tokenizer and tensors not verified · How to read

Full sequencePrefillDecode

32 independent transformer blocks › Attention

Merge heads

×

no parametersoutput.transpose(1, 2).contiguous().view(bsz, seqlen, -1)

Join all retrieved head features into the model-width field.

[B, 32, T, 128][B,\,32,\,T,\,128]
→  [B, T, 32, 128]\to\;[B,\,T,\,32,\,128]
→  [B, T, 4096]\to\;[B,\,T,\,4096]
in
HH [B batch, 32 q heads, 1 positions, 128 d_h]
out
HH [B batch, 1 positions, 4096 d]
Explanation

Heads are concatenated in order: head ii fills features 128i,…,128i+127{128i},\dots,{128i+127}.

Codemodel.py 188–189
model.py 188–189
        output = torch.matmul(scores, values)  # (bs, n_local_heads, seqlen, head_dim)
        output = output.transpose(1, 2).contiguous().view(bsz, seqlen, -1)
Sources1

Direct evidence

The head output is transposed and reshaped into the full model-width field.

attn-merge · meta-llama/llama3 / llama/model.py
Pinned revision a0940f9cf706 ↗ · lines 188–189

Read code excerpt
        output = torch.matmul(scores, values)  # (bs, n_local_heads, seqlen, head_dim)
        output = output.transpose(1, 2).contiguous().view(bsz, seqlen, -1)