Llama 3 8B · Merge and project

in: $H$ [B batch, 32 q heads, T positions, 128 d_h]
out: $\Delta_{\mathrm{attn}}$ [B batch, T positions, 4096 d]
parameters: 16,777,216 · × 32 blocks = 536,870,912

Parameters

16,777,216 parameters per block, across its learned components

Exact per-instance accounting
Component	Calculation	Parameters
Output projection	$4{,}096 \times 4{,}096$	16,777,216
Per-instance total		16,777,216

Across 32 independent blocks: $16{,}777{,}216 \times 32 = 536{,}870{,}912$ parameters.

The group references these owners; it adds no weights of its own.

Explanation

Merging moves the head axis beside the feature axis and concatenates 32 × 128 = 4,096 features into $H$ . The output projection then combines information across heads at every position: $\Delta_{\mathrm{attn}}=HW_O$ . Only the projection owns parameters.

Operations: Merge heads · Output projection
Intermediate states: Merged field

Codemodel.py 121–248 · 2 more files

model.py 121–127

        self.wo = RowParallelLinear(
            args.n_heads * self.head_dim,
            args.dim,
            bias=False,
            input_is_parallel=True,
            init_method=lambda x: x,
        )

model.py 189–190

        output = output.transpose(1, 2).contiguous().view(bsz, seqlen, -1)
        return self.wo(output)

model.py 236–248

        self.attention_norm = RMSNorm(args.dim, eps=args.norm_eps)
        self.ffn_norm = RMSNorm(args.dim, eps=args.norm_eps)

    def forward(
        self,
        x: torch.Tensor,
        start_pos: int,
        freqs_cis: torch.Tensor,
        mask: Optional[torch.Tensor],
    ):
        h = x + self.attention(self.attention_norm(x), start_pos, freqs_cis, mask)
        out = h + self.feed_forward(self.ffn_norm(h))
        return out

layers.py 258–262


        # Parameters.
        # Note: torch.nn.functional.linear performs XA^T + b and as a result
        # we allocate the transpose.
        self.weight = Parameter(torch.Tensor(self.output_size_per_partition, self.in_features))

layers.py 286–290

    def forward(self, input_: torch.Tensor) -> torch.Tensor:  # type: ignore
        # Set up backprop all-reduce.
        input_parallel = copy_to_model_parallel_region(input_)
        # Matrix multiply.
        output_parallel = F.linear(input_parallel, self.weight, self.bias)

layers.py 345–349


        # Parameters.
        # Note: torch.nn.functional.linear performs XA^T + b and as a result
        # we allocate the transpose.
        self.weight = Parameter(torch.Tensor(self.out_features, self.input_size_per_partition))

layers.py 374–380

        # Set up backprop all-reduce.
        if self.input_is_parallel:
            input_parallel = input_
        else:
            input_parallel = scatter_to_model_parallel_region(input_)
        # Matrix multiply.
        output_parallel = F.linear(input_parallel, self.weight)

sku_list.py 19

LLAMA3_VOCAB_SIZE = 128256

sku_list.py 188–207

def llama3_base_models() -> list[Model]:
    return [
        Model(
            core_model_id=CoreModelId.llama3_8b,
            description="Llama 3 8b model",
            huggingface_repo="meta-llama/Llama-3-8B",
            arch_args={
                "dim": 4096,
                "n_layers": 32,
                "n_heads": 32,
                "n_kv_heads": 8,
                "vocab_size": LLAMA3_VOCAB_SIZE,
                "ffn_dim_multiplier": 1.3,
                "multiple_of": 1024,
                "norm_eps": 1e-05,
                "rope_theta": 500000.0,
                "use_scaled_rope": False,
            },
            pth_file_count=1,
        ),

Sources4

Evidence for the operations this group contains. The group itself is a presentation and has no implementation of its own.

Contextual evidence from Merge heads

The head output is transposed and reshaped into the full model-width field.

attn-merge · meta-llama/llama3 / llama/model.py
Pinned revision a0940f9cf706 ↗ · lines 188–189

Read code excerpt

        output = torch.matmul(scores, values)  # (bs, n_local_heads, seqlen, head_dim)
        output = output.transpose(1, 2).contiguous().view(bsz, seqlen, -1)

Contextual evidence from Output projection

A bias-free learned output projection transforms the merged field at model width.

attn-output · meta-llama/llama3 / llama/model.py
Pinned revision a0940f9cf706 ↗ · lines 121–190

Read code excerpt

        self.wo = RowParallelLinear(
            args.n_heads * self.head_dim,
            args.dim,
            bias=False,
            input_is_parallel=True,
            init_method=lambda x: x,
        )

… lines 128–188 omitted …

        output = output.transpose(1, 2).contiguous().view(bsz, seqlen, -1)
        return self.wo(output)

Contextual evidence from Output projection

Meta’s original Llama 3 8B base catalogue entry supplies dimensions, head counts, vocabulary, MLP rounding settings, epsilon and unscaled RoPE settings.

config · meta-llama/llama-models / models/sku_list.py
Pinned revision 0e0b8c519242 ↗ · lines 19–207

Read code excerpt

LLAMA3_VOCAB_SIZE = 128256

… lines 20–187 omitted …

def llama3_base_models() -> list[Model]:
    return [
        Model(
            core_model_id=CoreModelId.llama3_8b,
            description="Llama 3 8b model",
            huggingface_repo="meta-llama/Llama-3-8B",
            arch_args={
                "dim": 4096,
                "n_layers": 32,
                "n_heads": 32,
                "n_kv_heads": 8,
                "vocab_size": LLAMA3_VOCAB_SIZE,
                "ffn_dim_multiplier": 1.3,
                "multiple_of": 1024,
                "norm_eps": 1e-05,
                "rope_theta": 500000.0,
                "use_scaled_rope": False,
            },
            pth_file_count=1,
        ),

Contextual evidence from Output projection

Attention receives attention_norm(x) and its update is added to x; the MLP receives ffn_norm(h) and its update is added to h.

handoff · meta-llama/llama3 / llama/model.py
Pinned revision a0940f9cf706 ↗ · lines 236–248

Read code excerpt

        self.attention_norm = RMSNorm(args.dim, eps=args.norm_eps)
        self.ffn_norm = RMSNorm(args.dim, eps=args.norm_eps)

    def forward(
        self,
        x: torch.Tensor,
        start_pos: int,
        freqs_cis: torch.Tensor,
        mask: Optional[torch.Tensor],
    ):
        h = x + self.attention(self.attention_norm(x), start_pos, freqs_cis, mask)
        out = h + self.feed_forward(self.ffn_norm(h))
        return out

Contextual evidence from Output projection

FairScale 0.4.13 stores linear weights with output features first and calls F.linear. This is operation-context evidence; the Meta requirements leave the FairScale version unpinned.

linear-storage · facebookresearch/fairscale / fairscale/nn/model_parallel/layers.py
Pinned revision 50b06d251730 ↗ · lines 258–380

Read code excerpt


        # Parameters.
        # Note: torch.nn.functional.linear performs XA^T + b and as a result
        # we allocate the transpose.
        self.weight = Parameter(torch.Tensor(self.output_size_per_partition, self.in_features))

… lines 263–285 omitted …

    def forward(self, input_: torch.Tensor) -> torch.Tensor:  # type: ignore
        # Set up backprop all-reduce.
        input_parallel = copy_to_model_parallel_region(input_)
        # Matrix multiply.
        output_parallel = F.linear(input_parallel, self.weight, self.bias)

… lines 291–344 omitted …


        # Parameters.
        # Note: torch.nn.functional.linear performs XA^T + b and as a result
        # we allocate the transpose.
        self.weight = Parameter(torch.Tensor(self.out_features, self.input_size_per_partition))

… lines 350–373 omitted …

        # Set up backprop all-reduce.
        if self.input_is_parallel:
            input_parallel = input_
        else:
            input_parallel = scatter_to_model_parallel_region(input_)
        # Matrix multiply.
        output_parallel = F.linear(input_parallel, self.weight)