Llama 3

All models, with previews →

Meta · Apr 2024Dense decoder

Transformer block

Reference code meta-llama/llama3 @ a0940f9, configuration meta-llama/llama-models @ 0e0b8c5 · checkpoint, tokenizer and tensors not verified · How to read

32 independent transformer blocks

Attention residual addition

×

no parametersx + self.attention(…)

Add the mechanism’s update to the residual stream element by element.

xn′=xn+Δattn{\htmlData{light=residual attention-add}{x'_n}}={\htmlData{light=block-input}{x_n}}+{\htmlData{light=attention}{\Delta_{\mathrm{attn}}}}
in
xnx_n [B batch, T positions, 4096 d]Δattn\Delta_{\mathrm{attn}} [B batch, T positions, 4096 d]
out
xn′x'_n [B batch, T positions, 4096 d]
Codemodel.py 236–248
model.py 236–248
        self.attention_norm = RMSNorm(args.dim, eps=args.norm_eps)
        self.ffn_norm = RMSNorm(args.dim, eps=args.norm_eps)

    def forward(
        self,
        x: torch.Tensor,
        start_pos: int,
        freqs_cis: torch.Tensor,
        mask: Optional[torch.Tensor],
    ):
        h = x + self.attention(self.attention_norm(x), start_pos, freqs_cis, mask)
        out = h + self.feed_forward(self.ffn_norm(h))
        return out
Sources1

Direct evidence

Two independently normalized mechanism branches are added to their corresponding residual inputs.

block-residuals · meta-llama/llama3 / llama/model.py
Pinned revision a0940f9cf706 ↗ · lines 236–248

Read code excerpt
        self.attention_norm = RMSNorm(args.dim, eps=args.norm_eps)
        self.ffn_norm = RMSNorm(args.dim, eps=args.norm_eps)

    def forward(
        self,
        x: torch.Tensor,
        start_pos: int,
        freqs_cis: torch.Tensor,
        mask: Optional[torch.Tensor],
    ):
        h = x + self.attention(self.attention_norm(x), start_pos, freqs_cis, mask)
        out = h + self.feed_forward(self.ffn_norm(h))
        return out