DeepSeek-V3.2-Exp

All models, with previews →

DeepSeek · Sep 2025Latent attention and experts

MoE transformer block

Reference code deepseek-ai/DeepSeek-V3.2-Exp @ 87e509a · checkpoint, tokenizer and tensors not verified · How to read

in
xnx_n [B batch, T positions, 7168 d]RR [T positions, 32 pairs]MM [T queries, T keys]
out
xn+1x_{n+1} [B batch, T positions, 7168 d]
parameters
11,521,245,696 · × 58 MoE blocks = 668,232,250,368
Parameters

11,521,245,696 parameters per MoE block, across its learned components

Exact per-instance accounting
ComponentCalculationParameters
Attention RMSNorm7,1687{,}1687,168
Multi-head latent attention201,066,752 across its components201,066,752
MoE RMSNorm7,1687{,}1687,168
Mixture of experts11,320,164,608 across its components11,320,164,608
Per-instance total11,521,245,696

Across 58 independent MoE blocks: 11,521,245,696×58=668,232,250,36811{,}521{,}245{,}696 \times 58 = 668{,}232{,}250{,}368 parameters.

Derived from the pinned reference configuration and inference code. Checkpoint tensors have not been independently counted; FP8 block scales and the multi-token-prediction module are not counted.

In numbers

Weights · one block, by sublayer

MoE block · FP810.7 GiB
Attention · FP8193.1 MiB
MoE · FP810.5 GiB
Two RMSNorms · float3256 KiB

11,521,245,696 parameters per block: attention 201,066,752 (1.7%), the MoE 11,320,164,608 (98.3%), the two RMSNorms 14,336; × 58 MoE blocks = 668,232,250,368.

config_671B_v3.2.json 5–8config_671B_v3.2.json 17model.py 826–829model.py 844–851config_671B_v3.2.json 21model.py 876model.py 183–187model.py 284model.py 317–318model.py 449model.py 674–675generate.py 109
Codemodel.py 826–829, 844–851
model.py 826–829
        self.attn = MLA(args)
        self.ffn = MLP(args.dim, args.inter_dim) if layer_id < args.n_dense_layers else MoE(args)
        self.attn_norm = RMSNorm(args.dim)
        self.ffn_norm = RMSNorm(args.dim)
model.py 844–851
        if residual is None:
            x, residual = self.attn_norm(x), x
        else:
            x, residual = self.attn_norm(x, residual)
        x = self.attn(x, start_pos, freqs_cis, mask)
        x, residual = self.ffn_norm(x, residual)
        x = self.ffn(x)
        return x, residual
Sources1

Direct evidence

A block applies attention-norm, MLA, ffn-norm and the FFN; the first n_dense_layers blocks use the dense MLP, the rest MoE. Residual additions happen inside the next normalization.

block · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 826–851

Read code excerpt
        self.attn = MLA(args)
        self.ffn = MLP(args.dim, args.inter_dim) if layer_id < args.n_dense_layers else MoE(args)
        self.attn_norm = RMSNorm(args.dim)
        self.ffn_norm = RMSNorm(args.dim)

… lines 830–843 omitted …

        if residual is None:
            x, residual = self.attn_norm(x), x
        else:
            x, residual = self.attn_norm(x, residual)
        x = self.attn(x, start_pos, freqs_cis, mask)
        x, residual = self.ffn_norm(x, residual)
        x = self.ffn(x)
        return x, residual