DeepSeek-V3.2-Exp

All models, with previews →

DeepSeek · Sep 2025Latent attention and experts

Transformer blocks

Reference code deepseek-ai/DeepSeek-V3.2-Exp @ 87e509a · checkpoint, tokenizer and tensors not verified · How to read

61 transformer blocks

Embedded state

×

tensorh · 904

The residual stream entering the first block.

shape
x0x_0 [B batch, T positions, 7168 d]
Sources

Contextual evidence from Token embedding

The token embedding is a learned table of vocabulary × width rows, looked up by token ID.

embedding · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 108–127

Read code excerpt
        self.weight = nn.Parameter(torch.empty(self.part_vocab_size, self.dim))

… lines 109–126 omitted …

        y = F.embedding(x, self.weight)

Contextual evidence from Token embedding

The released 671B configuration: width, layers, dense layers, heads, experts and routing, MLA ranks and head widths, indexer heads and top-k.

config · deepseek-ai/DeepSeek-V3.2-Exp / inference/config_671B_v3.2.json
Pinned revision 87e509a2e5a1 ↗ · lines 1–26

Read code excerpt
{
    "vocab_size": 129280,
    "dim": 7168,
    "inter_dim": 18432,
    "moe_inter_dim": 2048,
    "n_layers": 61,
    "n_dense_layers": 3,
    "n_heads": 128,
    "n_routed_experts": 256,
    "n_shared_experts": 1,
    "n_activated_experts": 8,
    "n_expert_groups": 8,
    "n_limited_groups": 4,
    "route_scale": 2.5,
    "score_func": "sigmoid",
    "q_lora_rank": 1536,
    "kv_lora_rank": 512,
    "qk_nope_head_dim": 128,
    "qk_rope_head_dim": 64,
    "v_head_dim": 128,
    "dtype": "fp8",
    "scale_fmt": "ue8m0",
    "index_n_heads": 64,
    "index_head_dim": 128,
    "index_topk": 2048
}

Contextual evidence from Dense transformer block

A block applies attention-norm, MLA, ffn-norm and the FFN; the first n_dense_layers blocks use the dense MLP, the rest MoE. Residual additions happen inside the next normalization.

block · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 826–851

Read code excerpt
        self.attn = MLA(args)
        self.ffn = MLP(args.dim, args.inter_dim) if layer_id < args.n_dense_layers else MoE(args)
        self.attn_norm = RMSNorm(args.dim)
        self.ffn_norm = RMSNorm(args.dim)

… lines 830–843 omitted …

        if residual is None:
            x, residual = self.attn_norm(x), x
        else:
            x, residual = self.attn_norm(x, residual)
        x = self.attn(x, start_pos, freqs_cis, mask)
        x, residual = self.ffn_norm(x, residual)
        x = self.ffn(x)
        return x, residual