32 independent transformer blocks › Attention
Merged field
×tensoroutput · 189
The head outputs joined along the feature axis.
- shape
- [B batch, 1 position, 4096 d]
Codemodel.py 188–189
output = torch.matmul(scores, values) # (bs, n_local_heads, seqlen, head_dim)
output = output.transpose(1, 2).contiguous().view(bsz, seqlen, -1)Sources1
Contextual evidence
The head output is transposed and reshaped into the full model-width field.
attn-merge · meta-llama/llama3 / llama/model.py
Pinned revision a0940f9cf706 ↗ · lines 188–189
Read code excerpt
output = torch.matmul(scores, values) # (bs, n_local_heads, seqlen, head_dim)
output = output.transpose(1, 2).contiguous().view(bsz, seqlen, -1)