DeepSeek V4.1

All models, with their blocks →

Size
552B + 196B Engram
Published
DeepSeek · Sep 2026
Line
New parts in the block · after DeepSeek-V3.2-Exp
View
Source

deepseek‑ai/DeepSeek‑V4.1‑Flash @ 2cba9e4 · parameter count checked against the checkpoint’s tensor headers · text model onlyHow to read

Pooled group

Full sequencePrefillDecode

40 layers › 2 groups of 6 layers

Source layer, pairs pooled

Open inside →×

learnedBlock(layer_id, args)

The residual is four parallel streams. Each sublayer reads them as one weighted sum and writes its output back into all four, with weights computed from the streams themselves. Here: compressed sparse attention and a mixture of experts; layers 2, 8, 14.

hffn=∑jpn,j′ Xn,j′h_{\mathrm{ffn}}=\sum_{j}p'_{n,j}\,X'_{n,j}
Xn+1,k=ukffn Δffn+∑jCjkffn Xn,j′X_{n+1,{\htmlClass{vl-stream-k}{k}}}=u^{\mathrm{ffn}}_{{\htmlClass{vl-stream-k}{k}}}\,\Delta_{\mathrm{ffn}}+\sum_{j}C^{\mathrm{ffn}}_{jk}\,X'_{n,j}
All 64 steps
Stream weights for attention:\text{Stream weights for attention:}
xf=reshape⁡(Xn)x^{f}=\operatorname{reshape}(X_n)
m=xfWmix⊤mean⁡((xf)2)+εm=\frac{x^{f}W_{\mathrm{mix}}^{\top}}{\sqrt{\operatorname{mean}\big((x^{f})^{2}\big)+\varepsilon}}
pn,j′=σ(s1 mj+bj)+ϵp'_{n,j}=\sigma\big(s_{1}\,m_j+b_j\big)+\epsilon
ukattn=2 σ(s2 m4+k+b4+k)u^{\mathrm{attn}}_{{\htmlClass{vl-stream-k}{k}}}=2\,\sigma\big(s_{2}\,m_{4+{\htmlClass{vl-stream-k}{k}}}+b_{4+{\htmlClass{vl-stream-k}{k}}}\big)
Cjkattn=Sinkhorn⁡ ⁣(softmax⁡k(s3 m8+4(j−1)+k+b8+4(j−1)+k))C^{\mathrm{attn}}_{jk}=\operatorname{Sinkhorn}\!\Big(\operatorname{softmax}_{{\htmlClass{vl-stream-k}{k}}}\big(s_{3}\,m_{8+4(j-1)+{\htmlClass{vl-stream-k}{k}}}+b_{8+4(j-1)+{\htmlClass{vl-stream-k}{k}}}\big)\Big)
hattn=∑jpn,j Xn,jh_{\mathrm{attn}}=\sum_{j}p_{n,j}\,X_{n,j}
Compressed sparse attention:\text{Compressed sparse attention:}
qa=xattnWDQ{\htmlClass{vl-stream-q}{q^{a}}}=x_{\mathrm{attn}}W_{DQ}
q=cQWUQ{\htmlClass{vl-stream-q}{q}}=c^{{\htmlClass{vl-stream-q}{Q}}}W_{UQ}
[B, T, 32768]  →  [B, 64, T, 512][B,\,T,\,32768]\;\to\;[B,\,64,\,T,\,512]
Q…,448:←Rt Q…,448:{\htmlClass{vl-stream-q}{Q_{\ldots,448:}}}\leftarrow {\htmlClass{vl-stream-context}{R_t}}\,{\htmlClass{vl-stream-q}{Q_{\ldots,448:}}}
wa=xattnWKVw^{a}=x_{\mathrm{attn}}W_{KV}
Wt,448:←Rt wt,448:nW_{t,448:}\leftarrow {\htmlClass{vl-stream-context}{R_t}}\,w^{n}_{t,448:}
W←FP8⁡(W)W\leftarrow\operatorname{FP8}(W)
kc=xattnWC{\htmlClass{vl-stream-k}{k^{c}}}=x_{\mathrm{attn}}W_{C}
gc=xattnWGg^{c}=x_{\mathrm{attn}}W_{G}
kjp=∑i∈{2j, 2j+1}softmax⁡i(gic)⊙kic{\htmlClass{vl-stream-k}{k^{p}_{j}}}=\sum_{i\in\{2j,\,2j+1\}}\operatorname{softmax}_{i}\big(g^{c}_{i}\big)\odot {\htmlClass{vl-stream-k}{k^{c}_{i}}}
c=γC⊙kpmean⁡((kp)2)+εc=\gamma_{C}\odot\frac{{\htmlClass{vl-stream-k}{k^{p}}}}{\sqrt{\operatorname{mean}\big(({\htmlClass{vl-stream-k}{k^{p}}})^{2}\big)+\varepsilon}}
Zj,448:←R2j cj,448:Z_{j,448:}\leftarrow {\htmlClass{vl-stream-context}{R_{2j}}}\,c_{j,448:}
Z←FP4⁡(Z)Z\leftarrow\operatorname{FP4}(Z)
ki=c WIK{\htmlClass{vl-stream-k}{k^{i}}}=c\,W_{IK}
kn=γIK⊙kimean⁡((ki)2)+ε{\htmlClass{vl-stream-k}{k^{n}}}=\gamma_{IK}\odot\frac{{\htmlClass{vl-stream-k}{k^{i}}}}{\sqrt{\operatorname{mean}\big(({\htmlClass{vl-stream-k}{k^{i}}})^{2}\big)+\varepsilon}}
Kj,64:I←R2j kj,64:n{\htmlClass{vl-stream-k}{K^{I}_{j,64:}}}\leftarrow {\htmlClass{vl-stream-context}{R_{2j}}}\,{\htmlClass{vl-stream-k}{k^{n}_{j,64:}}}
KI←FP4⁡(KI){\htmlClass{vl-stream-k}{K^{I}}}\leftarrow\operatorname{FP4}({\htmlClass{vl-stream-k}{K^{I}}})
qi=cQWIQ{\htmlClass{vl-stream-q}{q^{i}}}=c^{{\htmlClass{vl-stream-q}{Q}}}W_{IQ}
wI=xattnWIW /128⋅32w^{I}=x_{\mathrm{attn}}W_{IW}\,/\sqrt{128\cdot32}
St,jI=∑hwt,hI ReLU⁡(Qh,tI⋅KjI)S^{I}_{t,j}=\sum_{h}w^{I}_{t,h}\,\operatorname{ReLU}\big({\htmlClass{vl-stream-q}{Q^{I}_{h,t}}}\cdot {\htmlClass{vl-stream-k}{K^{I}_{j}}}\big)
It=top⁡512(St,jI  :  j<⌊(t+1)/2⌋)I_{t}=\operatorname{top}_{512}\big(S^{I}_{t,j}\;:\;j<\lfloor(t+1)/2\rfloor\big)
Jtw={max⁡(0, t−127),…,t}J^{w}_{t}=\{\max(0,\,t-127),\ldots,t\}
Y=[ W;  Z ]Y=[\,W;\;Z\,]
J=[ Jw;  I+∣W∣ ]J=[\,J^{w};\;I+\lvert W\rvert\,]
Gt=Y[ Jt ]G_{t}=Y[\,J_{t}\,]
Sh,t,i=Qh,t⋅Gt,i /512S_{h,t,i}={\htmlClass{vl-stream-q}{Q_{h,t}}}\cdot G_{t,i}\,/\sqrt{512}
Ah,t,i=exp⁡Sh,t,iexp⁡sh+∑jexp⁡Sh,t,jA_{h,t,i}=\frac{\exp S_{h,t,i}}{\exp s_{h}+\sum_{j}\exp S_{h,t,j}}
Oh,t=∑iAh,t,i Gt,iO_{h,t}=\sum_{i}A_{h,t,i}\,G_{t,i}
Oˉh,t,448:=Rt−1 Oh,t,448:\bar O_{h,t,448:}={\htmlClass{vl-stream-context}{R_t^{-1}}}\,O_{h,t,448:}
ot,rg=[ Oˉh,t ]h∈r WOA,ro^{g}_{t,r}=\big[\,\bar O_{h,t}\,\big]_{h\in r}\,W_{OA,r}
Δattn=ogWOB\Delta_{\mathrm{attn}}=o^{g}W_{OB}
Xn,k′=ukattn Δattn+∑jCjkattn Xn,jX'_{n,{\htmlClass{vl-stream-k}{k}}}=u^{\mathrm{attn}}_{{\htmlClass{vl-stream-k}{k}}}\,\Delta_{\mathrm{attn}}+\sum_{j}C^{\mathrm{attn}}_{jk}\,X_{n,j}
Stream weights for the feed-forward:\text{Stream weights for the feed-forward:}
xf=reshape⁡(Xn′)x^{f}=\operatorname{reshape}(X'_n)
m=xfWmix⊤mean⁡((xf)2)+εm=\frac{x^{f}W_{\mathrm{mix}}^{\top}}{\sqrt{\operatorname{mean}\big((x^{f})^{2}\big)+\varepsilon}}
pn+1,j=σ(s1 mj+bj)+ϵp_{n+1,j}=\sigma\big(s_{1}\,m_j+b_j\big)+\epsilon
ukffn=2 σ(s2 m4+k+b4+k)u^{\mathrm{ffn}}_{{\htmlClass{vl-stream-k}{k}}}=2\,\sigma\big(s_{2}\,m_{4+{\htmlClass{vl-stream-k}{k}}}+b_{4+{\htmlClass{vl-stream-k}{k}}}\big)
Cjkffn=Sinkhorn⁡ ⁣(softmax⁡k(s3 m8+4(j−1)+k+b8+4(j−1)+k))C^{\mathrm{ffn}}_{jk}=\operatorname{Sinkhorn}\!\Big(\operatorname{softmax}_{{\htmlClass{vl-stream-k}{k}}}\big(s_{3}\,m_{8+4(j-1)+{\htmlClass{vl-stream-k}{k}}}+b_{8+4(j-1)+{\htmlClass{vl-stream-k}{k}}}\big)\Big)
hffn=∑jpn,j′ Xn,j′h_{\mathrm{ffn}}=\sum_{j}p'_{n,j}\,X'_{n,j}
Mixture of experts:\text{Mixture of experts:}
z=xffnWRz=x_{\mathrm{ffn}}W_{{\htmlClass{vl-stream-context}{R}}}
s=softplus⁡(z)s=\sqrt{\operatorname{softplus}(z)}
T=top⁡6(s+b)\mathcal T=\operatorname{top}_{6}(s+b)
Gi=1.5 si∑j∈Tsj  (i∈T)G_{i}=1.5\,\frac{s_{i}}{\sum_{j\in\mathcal T}s_{j}}\;(i\in\mathcal T)
g=xffnW1g=x_{\mathrm{ffn}}W_{1}
u=xffnW3u=x_{\mathrm{ffn}}W_{3}
gˉ=min⁡(g, 10)\bar g=\min(g,\,10)
uˉ=clip⁡(u, −10, 10)\bar u=\operatorname{clip}(u,\,-10,\,10)
a=gˉ σ(gˉ)a=\bar g\,\sigma(\bar g)
h=a⊙uˉh=a\odot\bar u
Ei=hW2E_i=hW_{2}
yrouted=∑i∈TGi Eiy^{\mathrm{routed}}=\sum_{i\in\mathcal T}G_{i}\,E_{i}
g=xffnW1g=x_{\mathrm{ffn}}W_{1}
u=xffnW3u=x_{\mathrm{ffn}}W_{3}
gˉ=min⁡(g, 10)\bar g=\min(g,\,10)
uˉ=clip⁡(u, −10, 10)\bar u=\operatorname{clip}(u,\,-10,\,10)
a=gˉ σ(gˉ)a=\bar g\,\sigma(\bar g)
h=a⊙uˉh=a\odot\bar u
yshared=hW2y^{\mathrm{shared}}=hW_{2}
Δffn=yrouted+yshared\Delta_{\mathrm{ffn}}=y^{\mathrm{routed}}+y^{\mathrm{shared}}
Xn+1,k=ukffn Δffn+∑jCjkffn Xn,j′X_{n+1,{\htmlClass{vl-stream-k}{k}}}=u^{\mathrm{ffn}}_{{\htmlClass{vl-stream-k}{k}}}\,\Delta_{\mathrm{ffn}}+\sum_{j}C^{\mathrm{ffn}}_{jk}\,X'_{n,j}
in
XnX_n [B batch, T positions, 4 streams, 5120 d]pnp_n [B batch, T positions, 4 streams]
out
Xn+1X_{n+1} [B batch, T positions, 4 streams, 5120 d]
parameters
13,765,227,766 · × 2 groups = 27,530,455,532

Between forward calls: the layer keeps a ring of 128 slots holding the window keys and values of up to the last 128 positions (model.py 663–668, 708–719); the source layer keeps ⌊T/2⌋ rows of compressed keys and values, 512 features each, and as many index keys of 128 (model.py 669–679, 517–525); a pooling source also keeps its pair buffer, the projections and pooling scores of two tokens: 2 slots of 512 in each of two tensors (model.py 449–456).

Parameters

13,765,227,766 parameters per instance, across its learned components

Exact per-instance accounting
ComponentCalculationParameters
Stream weights for attention491,547 across its components491,547
Attention RMSNorm5,1205{,}1205,120
Compressed sparse attention137,333,184 across its components137,333,184
Stream weights for the feed-forward491,547 across its components491,547
MoE RMSNorm5,1205{,}1205,120
Mixture of experts13,626,901,248 across its components13,626,901,248
Per-instance total13,765,227,766

Across 2 independent groups: 13,765,227,766×2=27,530,455,53213{,}765{,}227{,}766 \times 2 = 27{,}530{,}455{,}532 parameters.

Derived from the pinned reference implementation and configuration; equal to the element counts of the checkpoint’s own tensors on the text path. The quantisation scales, the vision encoder and the DSpark blocks are not counted.

Codemodel.py 907–994
model.py 907–946
class Block(nn.Module):
    """A block whose residual stream is `hc_mult` parallel copies (Hyper-Connections).

    Attention and FFN each sit between `hc_pre` (collapse the copies into one sublayer input) and
    `hc_post` (expand back out, mixing the residual in through `comb`). `hc_mixes` derives all three
    coefficient sets from the stream itself, `comb` made doubly stochastic by Sinkhorn.

    The coefficients a sublayer computes are used by the *next* one -- see `forward`.
    """

    attention_cls = Attention

    def __init__(
        self,
        layer_id: int,
        args: ModelArgs,
        engram_layout: EngramLayout | None = None,
    ):
        super().__init__()
        self.layer_id = layer_id
        self.norm_eps = args.norm_eps
        self.attn = self.attention_cls(layer_id, args)
        self.ffn = MoE(layer_id, args)
        self.engram = None
        if engram_layout is not None and layer_id in engram_layout.layer_ids:
            self.engram = Engram(args, layer_id, engram_layout)
        self.attn_norm = RMSNorm(args.dim, self.norm_eps)
        self.ffn_norm = RMSNorm(args.dim, self.norm_eps)
        self.hc_mult = hc_mult = args.hc_mult
        self.hc_sinkhorn_iters = args.hc_sinkhorn_iters
        self.hc_eps = args.hc_eps
        mix_hc = (2 + hc_mult) * hc_mult
        hc_dim = hc_mult * args.dim
        with set_dtype(torch.float32):
            self.hc_attn_fn = nn.Parameter(torch.empty(mix_hc, hc_dim))
            self.hc_ffn_fn = nn.Parameter(torch.empty(mix_hc, hc_dim))
            self.hc_attn_base = nn.Parameter(torch.empty(mix_hc))
            self.hc_ffn_base = nn.Parameter(torch.empty(mix_hc))
            self.hc_attn_scale = nn.Parameter(torch.empty(3))
            self.hc_ffn_scale = nn.Parameter(torch.empty(3))
model.py 968–994
    def forward(
        self,
        x: torch.Tensor,
        start_pos: int,
        pre_mix: torch.Tensor,
        image_mask: torch.Tensor | None,
        *attn_args,
    ) -> tuple[torch.Tensor, torch.Tensor]:
        """`pre_mix` collapses the hc_mult copies down to one input for this block's attention. Each
        sub-block's own `hc_mixes` produces the mix for the *next* one, so attention uses what the
        previous layer's FFN produced and the FFN uses what this attention produced.

        image_mask: [b, s] bool, True inside image spans (selects the VL routing bias)."""
        residual = x
        attn_pre, attn_post, attn_comb = self.hc_mixes(x, self.hc_attn_fn, self.hc_attn_scale, self.hc_attn_base)
        x = self.hc_pre(x, pre_mix)
        x = self.attn_norm(x)
        x = self.attn(x, start_pos, *attn_args)
        x = self.hc_post(x, residual, attn_post, attn_comb)

        residual = x
        ffn_pre, ffn_post, ffn_comb = self.hc_mixes(x, self.hc_ffn_fn, self.hc_ffn_scale, self.hc_ffn_base)
        x = self.hc_pre(x, attn_pre)
        x = self.ffn_norm(x)
        x = self.ffn(x, image_mask)
        x = self.hc_post(x, residual, ffn_post, ffn_comb)
        return x, ffn_pre
model.py 948–966
    def hc_mixes(self, x: torch.Tensor, hc_fn: torch.Tensor, hc_scale: torch.Tensor, hc_base: torch.Tensor):
        """x: [b,s,hc,d], hc_fn: [mix_hc, hc*d], hc_scale: [3], hc_base: [mix_hc]. Returns the
        pre / post / comb coefficients, split out of one projection of the flattened stream."""
        # normalized over the whole flattened hc*d stream, one statistic per token
        x = x.flatten(2).float()
        rsqrt = torch.rsqrt(x.square().mean(-1, keepdim=True) + self.norm_eps)
        mixes = F.linear(x, hc_fn) * rsqrt
        return hc_split_sinkhorn(mixes, hc_scale, hc_base, self.hc_mult, self.hc_sinkhorn_iters, self.hc_eps)

    def hc_pre(self, x: torch.Tensor, pre_mix: torch.Tensor):
        """Collapse the hc copies into one, weighted by pre_mix. [b,s,hc,d] x [b,s,hc] -> [b,s,d]"""
        y = torch.sum(pre_mix.unsqueeze(-1) * x.float(), dim=2)
        return y.to(x.dtype)

    def hc_post(self, x: torch.Tensor, residual: torch.Tensor, post: torch.Tensor, comb: torch.Tensor):
        """Expand the sublayer output back to hc copies and mix the residual in through `comb`.
        x: [b,s,d], residual: [b,s,hc,d], post: [b,s,hc], comb: [b,s,hc,hc] -> [b,s,hc,d]"""
        y = post.unsqueeze(-1) * x.unsqueeze(-2) + torch.sum(comb.unsqueeze(-1) * residual.unsqueeze(-2), dim=2)
        return y.type_as(x)
Sources3

Direct evidence

A block owns its attention and mixture, two RMSNorms, and for each sublayer a projection of the flattened streams to 24 mix numbers with 24 offsets and 3 scales.

block-init · deepseek-ai/DeepSeek-V4.1-Flash / inference/model.py
Pinned revision 2cba9e42aa02 ↗ · lines 907–946

Read code excerpt
class Block(nn.Module):
    """A block whose residual stream is `hc_mult` parallel copies (Hyper-Connections).

    Attention and FFN each sit between `hc_pre` (collapse the copies into one sublayer input) and
    `hc_post` (expand back out, mixing the residual in through `comb`). `hc_mixes` derives all three
    coefficient sets from the stream itself, `comb` made doubly stochastic by Sinkhorn.

    The coefficients a sublayer computes are used by the *next* one -- see `forward`.
    """

    attention_cls = Attention

    def __init__(
        self,
        layer_id: int,
        args: ModelArgs,
        engram_layout: EngramLayout | None = None,
    ):
        super().__init__()
        self.layer_id = layer_id
        self.norm_eps = args.norm_eps
        self.attn = self.attention_cls(layer_id, args)
        self.ffn = MoE(layer_id, args)
        self.engram = None
        if engram_layout is not None and layer_id in engram_layout.layer_ids:
            self.engram = Engram(args, layer_id, engram_layout)
        self.attn_norm = RMSNorm(args.dim, self.norm_eps)
        self.ffn_norm = RMSNorm(args.dim, self.norm_eps)
        self.hc_mult = hc_mult = args.hc_mult
        self.hc_sinkhorn_iters = args.hc_sinkhorn_iters
        self.hc_eps = args.hc_eps
        mix_hc = (2 + hc_mult) * hc_mult
        hc_dim = hc_mult * args.dim
        with set_dtype(torch.float32):
            self.hc_attn_fn = nn.Parameter(torch.empty(mix_hc, hc_dim))
            self.hc_ffn_fn = nn.Parameter(torch.empty(mix_hc, hc_dim))
            self.hc_attn_base = nn.Parameter(torch.empty(mix_hc))
            self.hc_ffn_base = nn.Parameter(torch.empty(mix_hc))
            self.hc_attn_scale = nn.Parameter(torch.empty(3))
            self.hc_ffn_scale = nn.Parameter(torch.empty(3))

Direct evidence

Attention reads the streams with the mix the previous sublayer computed and writes with its own; the mixture reads with the mix attention computed and writes with its own, and hands its reading mix to the next layer.

block-forward · deepseek-ai/DeepSeek-V4.1-Flash / inference/model.py
Pinned revision 2cba9e42aa02 ↗ · lines 968–994

Read code excerpt
    def forward(
        self,
        x: torch.Tensor,
        start_pos: int,
        pre_mix: torch.Tensor,
        image_mask: torch.Tensor | None,
        *attn_args,
    ) -> tuple[torch.Tensor, torch.Tensor]:
        """`pre_mix` collapses the hc_mult copies down to one input for this block's attention. Each
        sub-block's own `hc_mixes` produces the mix for the *next* one, so attention uses what the
        previous layer's FFN produced and the FFN uses what this attention produced.

        image_mask: [b, s] bool, True inside image spans (selects the VL routing bias)."""
        residual = x
        attn_pre, attn_post, attn_comb = self.hc_mixes(x, self.hc_attn_fn, self.hc_attn_scale, self.hc_attn_base)
        x = self.hc_pre(x, pre_mix)
        x = self.attn_norm(x)
        x = self.attn(x, start_pos, *attn_args)
        x = self.hc_post(x, residual, attn_post, attn_comb)

        residual = x
        ffn_pre, ffn_post, ffn_comb = self.hc_mixes(x, self.hc_ffn_fn, self.hc_ffn_scale, self.hc_ffn_base)
        x = self.hc_pre(x, attn_pre)
        x = self.ffn_norm(x)
        x = self.ffn(x, image_mask)
        x = self.hc_post(x, residual, ffn_post, ffn_comb)
        return x, ffn_pre

Direct evidence

The mix numbers are a projection of the flattened streams divided by their root mean square; reading is a weighted sum of the streams; writing adds the sublayer output, weighted per stream, to a combination of the streams.

block-hc · deepseek-ai/DeepSeek-V4.1-Flash / inference/model.py
Pinned revision 2cba9e42aa02 ↗ · lines 948–966

Read code excerpt
    def hc_mixes(self, x: torch.Tensor, hc_fn: torch.Tensor, hc_scale: torch.Tensor, hc_base: torch.Tensor):
        """x: [b,s,hc,d], hc_fn: [mix_hc, hc*d], hc_scale: [3], hc_base: [mix_hc]. Returns the
        pre / post / comb coefficients, split out of one projection of the flattened stream."""
        # normalized over the whole flattened hc*d stream, one statistic per token
        x = x.flatten(2).float()
        rsqrt = torch.rsqrt(x.square().mean(-1, keepdim=True) + self.norm_eps)
        mixes = F.linear(x, hc_fn) * rsqrt
        return hc_split_sinkhorn(mixes, hc_scale, hc_base, self.hc_mult, self.hc_sinkhorn_iters, self.hc_eps)

    def hc_pre(self, x: torch.Tensor, pre_mix: torch.Tensor):
        """Collapse the hc copies into one, weighted by pre_mix. [b,s,hc,d] x [b,s,hc] -> [b,s,d]"""
        y = torch.sum(pre_mix.unsqueeze(-1) * x.float(), dim=2)
        return y.to(x.dtype)

    def hc_post(self, x: torch.Tensor, residual: torch.Tensor, post: torch.Tensor, comb: torch.Tensor):
        """Expand the sublayer output back to hc copies and mix the residual in through `comb`.
        x: [b,s,d], residual: [b,s,hc,d], post: [b,s,hc], comb: [b,s,hc,hc] -> [b,s,hc,d]"""
        y = post.unsqueeze(-1) * x.unsqueeze(-2) + torch.sum(comb.unsqueeze(-1) * residual.unsqueeze(-2), dim=2)
        return y.type_as(x)