DeepSeek-V3.2-Exp

All models, with their blocks →

Size
671B
Published
DeepSeek · Sep 2025
Line
Latent attention and experts
View
Source

deepseek‑ai/DeepSeek‑V3.2‑Exp @ 87e509a · checkpoint, tokenizer and tensors not verifiedHow to read

61 transformer blocks

MoE transformer block

Open inside →×

learnedBlock(layer_id, args)

MLA and a mixture of experts, each added to the residual stream; blocks 4–61.

xn′=xn+Δattnx'_n=x_n+\Delta_{\mathrm{attn}}
xn+1=xn′+Δffnx_{n+1}=x'_n+\Delta_{\mathrm{ffn}}
All 43 steps
xattn=RMSNorm⁡(xn)x_{\mathrm{attn}}=\operatorname{RMSNorm}(x_n)
Sparse latent attention:\text{Sparse latent attention:}
cQ=xWDQc^{{\htmlClass{vl-stream-q}{Q}}}=xW_{DQ}
c^Q=RMSNorm⁡(cQ)\hat c^{{\htmlClass{vl-stream-q}{Q}}}=\operatorname{RMSNorm}(c^{{\htmlClass{vl-stream-q}{Q}}})
cKV=xWDKVc^{KV}=xW_{DKV}
c^KV=RMSNorm⁡(cKV)\hat c^{KV}=\operatorname{RMSNorm}(c^{KV})
qC=c^QWUQ{\htmlClass{vl-stream-q}{q^{C}}}=\hat c^{{\htmlClass{vl-stream-q}{Q}}}W_{UQ}
QC=reshape⁡(qC){\htmlClass{vl-stream-q}{Q^{C}}}=\operatorname{reshape}({\htmlClass{vl-stream-q}{q^{C}}})
qpe=c^QWQR{\htmlClass{vl-stream-q}{q^{\mathrm{pe}}}}=\hat c^{{\htmlClass{vl-stream-q}{Q}}}W_{QR}
Qpe=reshape⁡(qpe){\htmlClass{vl-stream-q}{Q^{\mathrm{pe}}}}=\operatorname{reshape}({\htmlClass{vl-stream-q}{q^{\mathrm{pe}}}})
QtR=RoPE⁡(Qtpe,Rt){\htmlClass{vl-stream-q}{Q^{R}_{t}}}=\operatorname{RoPE}({\htmlClass{vl-stream-q}{Q^{\mathrm{pe}}_{t}}},{\htmlClass{vl-stream-context}{R_t}})
Q=[QC; QR]{\htmlClass{vl-stream-q}{Q}}=[{\htmlClass{vl-stream-q}{Q^{C}}};\,{\htmlClass{vl-stream-q}{Q^{R}}}]
kC=c^KVWUK{\htmlClass{vl-stream-k}{k^{C}}}=\hat c^{KV}W_{UK}
KC=reshape⁡(kC){\htmlClass{vl-stream-k}{K^{C}}}=\operatorname{reshape}({\htmlClass{vl-stream-k}{k^{C}}})
kpe=xWKR{\htmlClass{vl-stream-k}{k^{\mathrm{pe}}}}=xW_{KR}
ktR=RoPE⁡(ktpe,Rt){\htmlClass{vl-stream-k}{k^{R}_{t}}}=\operatorname{RoPE}({\htmlClass{vl-stream-k}{k^{\mathrm{pe}}_{t}}},{\htmlClass{vl-stream-context}{R_t}})
Ki=[KiC; kR]{\htmlClass{vl-stream-k}{K_i}}=[{\htmlClass{vl-stream-k}{K^{C}_i}};\,{\htmlClass{vl-stream-k}{k^{R}}}]
v=c^KVWUV{\htmlClass{vl-stream-v}{v}}=\hat c^{KV}W_{UV}
V=reshape⁡(v){\htmlClass{vl-stream-v}{V}}=\operatorname{reshape}({\htmlClass{vl-stream-v}{v}})
w=xWww=xW_{w}
St,sI=164128∑h=164wt,h ReLU⁡ ⁣(Qh,tIˉ⋅ksIˉ)S^{I}_{t,s}=\frac{1}{\sqrt{64}\sqrt{128}}\sum_{h=1}^{64}w_{t,h}\,\operatorname{ReLU}\!\big(\bar {\htmlClass{vl-stream-q}{Q^{I}_{h,t}}}\cdot\bar {\htmlClass{vl-stream-k}{k^{I}_{s}}}\big)
S~I=SI+M\tilde S^{I}=S^{I}+{\htmlClass{vl-stream-context}{M}}
Mt,ssel={Mt,ss∈top⁡2,048(S~tI)−∞otherwise{\htmlClass{vl-stream-context}{M^{\mathrm{sel}}_{t,s}}}=\begin{cases}{\htmlClass{vl-stream-context}{M_{t,s}}}&s\in\operatorname{top}_{2,048}(\tilde S^{I}_{t})\\-\infty&\text{otherwise}\end{cases}
xn′=xn+Δattnx'_n=x_n+\Delta_{\mathrm{attn}}
xffn=RMSNorm⁡(xn′)x_{\mathrm{ffn}}=\operatorname{RMSNorm}(x'_n)
Mixture of experts:\text{Mixture of experts:}
z=xffnWrz=x_{\mathrm{ffn}}W_{r}
s=σ(z)s=\sigma(z)
Eisel=[ i∈top⁡8(s+b) within the top 4 groups ]E^{\mathrm{sel}}_i=\big[\,i\in\operatorname{top}_{8}(s+b)\text{ within the top }4\text{ groups}\,\big]
Gi=2.5 si Eisel∑jsj EjselG_{i}=2.5\,\frac{s_i\,E^{\mathrm{sel}}_i}{\sum_{j}s_j\,E^{\mathrm{sel}}_j}
g=xffnW1g=x_{\mathrm{ffn}}W_{1}
u=xffnW3u=x_{\mathrm{ffn}}W_{3}
a=g σ(g)a=g\,\sigma(g)
h=a⊙uh=a\odot u
Ei=hW2E_i=hW_{2}
yrouted=∑i=1256Gi Eiy^{\mathrm{routed}}=\sum_{i=1}^{256}G_i\,E_i
g=xffnW1g=x_{\mathrm{ffn}}W_{1}
u=xffnW3u=x_{\mathrm{ffn}}W_{3}
a=g σ(g)a=g\,\sigma(g)
h=a⊙uh=a\odot u
yshared=hW2y^{\mathrm{shared}}=hW_{2}
Δffn=yrouted+yshared\Delta_{\mathrm{ffn}}=y^{\mathrm{routed}}+y^{\mathrm{shared}}
xn+1=xn′+Δffnx_{n+1}=x'_n+\Delta_{\mathrm{ffn}}
in
x3x_{3} [B batch, T positions, 7168 d]RR [T positions, 32 pairs]MM [T queries, T keys]
out
xn+1x_{n+1} [B batch, T positions, 7168 d]
parameters
11,521,245,696 · × 58 MoE blocks = 668,232,250,368

Each residual addition runs in float32 inside the next RMSNorm, which returns the normalized sum and the sum, both cast back to bfloat16, the sum as the new stream; the first block’s norm receives no residual (model.py 302–306, 844–849, 907).

Parameters

11,521,245,696 parameters per MoE block, across its learned components

Exact per-instance accounting
ComponentCalculationParameters
Attention RMSNorm7,1687{,}1687,168
Sparse latent attention201,066,752 across its components201,066,752
MoE RMSNorm7,1687{,}1687,168
Mixture of experts11,320,164,608 across its components11,320,164,608
Per-instance total11,521,245,696

Across 58 independent MoE blocks: 11,521,245,696×58=668,232,250,36811{,}521{,}245{,}696 \times 58 = 668{,}232{,}250{,}368 parameters.

Derived from the pinned reference configuration and inference code. Checkpoint tensors have not been independently counted; FP8 block scales and the multi-token-prediction module are not counted.

Codemodel.py 826–829, 844–851
model.py 826–829
        self.attn = MLA(args)
        self.ffn = MLP(args.dim, args.inter_dim) if layer_id < args.n_dense_layers else MoE(args)
        self.attn_norm = RMSNorm(args.dim)
        self.ffn_norm = RMSNorm(args.dim)
model.py 844–851
        if residual is None:
            x, residual = self.attn_norm(x), x
        else:
            x, residual = self.attn_norm(x, residual)
        x = self.attn(x, start_pos, freqs_cis, mask)
        x, residual = self.ffn_norm(x, residual)
        x = self.ffn(x)
        return x, residual
Sources1

Direct evidence

A block applies attention-norm, MLA, ffn-norm and the FFN; the first n_dense_layers blocks use the dense MLP, the rest MoE. Residual additions happen inside the next normalization.

block · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 826–851

Read code excerpt
        self.attn = MLA(args)
        self.ffn = MLP(args.dim, args.inter_dim) if layer_id < args.n_dense_layers else MoE(args)
        self.attn_norm = RMSNorm(args.dim)
        self.ffn_norm = RMSNorm(args.dim)

… lines 830–843 omitted …

        if residual is None:
            x, residual = self.attn_norm(x), x
        else:
            x, residual = self.attn_norm(x, residual)
        x = self.attn(x, start_pos, freqs_cis, mask)
        x, residual = self.ffn_norm(x, residual)
        x = self.ffn(x)
        return x, residual