DeepSeek-V3.2-Exp

All models, with their blocks →

Size
671B
Published
DeepSeek · Sep 2025
Line
Latent attention and experts
View
Source

deepseek‑ai/DeepSeek‑V3.2‑Exp @ 87e509a · checkpoint, tokenizer and tensors not verifiedHow to read

Transformer blocks

61 transformer blocks

Dense-stack output

×

tensorresidual · 303

The residual stream after the 3 dense blocks.

shape
x3x_{3} [B batch, T positions, 7168 d]
Sources

Contextual evidence from Dense transformer block

A block applies attention-norm, MLA, ffn-norm and the FFN; the first n_dense_layers blocks use the dense MLP, the rest MoE. Residual additions happen inside the next normalization.

block · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 826–851

Read code excerpt
        self.attn = MLA(args)
        self.ffn = MLP(args.dim, args.inter_dim) if layer_id < args.n_dense_layers else MoE(args)
        self.attn_norm = RMSNorm(args.dim)
        self.ffn_norm = RMSNorm(args.dim)

… lines 830–843 omitted …

        if residual is None:
            x, residual = self.attn_norm(x), x
        else:
            x, residual = self.attn_norm(x, residual)
        x = self.attn(x, start_pos, freqs_cis, mask)
        x, residual = self.ffn_norm(x, residual)
        x = self.ffn(x)
        return x, residual