40 layers › 4 groups of 4 layers
learnedBlock(layer_id, args)
The residual is four parallel streams. Each sublayer reads them as one weighted sum and writes its output back into all four, with weights computed from the streams themselves. Here: compressed sparse attention and a mixture of experts; layers 24, 28, 32, 36.
All 56 steps
- in
- [B batch, T positions, 4 streams, 5120 d] [B batch, T positions, 4 streams] [B batch, T compressed positions, 512 d_h] [B batch, T compressed positions, 128 index width] [B batch, T positions, T compressed positions]
- out
- [B batch, T positions, 4 streams, 5120 d]
- parameters
- 13,759,918,710 · × 4 groups = 55,039,674,840
Between forward calls: the layer keeps a ring of 128 slots holding the window keys and values of up to the last 128 positions (model.py 663–668, 708–719). This layer reads compressed rows, index keys and candidates from its source and computes its own selected positions; it allocates none of those shared buffers (model.py 654–661, 669, 500, 517, 724–737, 763, 1166–1180).
Parameters
13,759,918,710 parameters per instance, across its learned components
| Component | Calculation | Parameters |
|---|---|---|
| Stream weights for attention | 491,547 across its components | 491,547 |
| Attention RMSNorm | 5,120 | |
| Compressed sparse attention | 132,024,128 across its components | 132,024,128 |
| Stream weights for the feed-forward | 491,547 across its components | 491,547 |
| MoE RMSNorm | 5,120 | |
| Mixture of experts | 13,626,901,248 across its components | 13,626,901,248 |
| Per-instance total | 13,759,918,710 | |
Across 4 independent groups: parameters.
Derived from the pinned reference implementation and configuration; equal to the element counts of the checkpoint’s own tensors on the text path. The quantisation scales, the vision encoder and the DSpark blocks are not counted.
Codemodel.py 907–994
class Block(nn.Module):
"""A block whose residual stream is `hc_mult` parallel copies (Hyper-Connections).
Attention and FFN each sit between `hc_pre` (collapse the copies into one sublayer input) and
`hc_post` (expand back out, mixing the residual in through `comb`). `hc_mixes` derives all three
coefficient sets from the stream itself, `comb` made doubly stochastic by Sinkhorn.
The coefficients a sublayer computes are used by the *next* one -- see `forward`.
"""
attention_cls = Attention
def __init__(
self,
layer_id: int,
args: ModelArgs,
engram_layout: EngramLayout | None = None,
):
super().__init__()
self.layer_id = layer_id
self.norm_eps = args.norm_eps
self.attn = self.attention_cls(layer_id, args)
self.ffn = MoE(layer_id, args)
self.engram = None
if engram_layout is not None and layer_id in engram_layout.layer_ids:
self.engram = Engram(args, layer_id, engram_layout)
self.attn_norm = RMSNorm(args.dim, self.norm_eps)
self.ffn_norm = RMSNorm(args.dim, self.norm_eps)
self.hc_mult = hc_mult = args.hc_mult
self.hc_sinkhorn_iters = args.hc_sinkhorn_iters
self.hc_eps = args.hc_eps
mix_hc = (2 + hc_mult) * hc_mult
hc_dim = hc_mult * args.dim
with set_dtype(torch.float32):
self.hc_attn_fn = nn.Parameter(torch.empty(mix_hc, hc_dim))
self.hc_ffn_fn = nn.Parameter(torch.empty(mix_hc, hc_dim))
self.hc_attn_base = nn.Parameter(torch.empty(mix_hc))
self.hc_ffn_base = nn.Parameter(torch.empty(mix_hc))
self.hc_attn_scale = nn.Parameter(torch.empty(3))
self.hc_ffn_scale = nn.Parameter(torch.empty(3)) def forward(
self,
x: torch.Tensor,
start_pos: int,
pre_mix: torch.Tensor,
image_mask: torch.Tensor | None,
*attn_args,
) -> tuple[torch.Tensor, torch.Tensor]:
"""`pre_mix` collapses the hc_mult copies down to one input for this block's attention. Each
sub-block's own `hc_mixes` produces the mix for the *next* one, so attention uses what the
previous layer's FFN produced and the FFN uses what this attention produced.
image_mask: [b, s] bool, True inside image spans (selects the VL routing bias)."""
residual = x
attn_pre, attn_post, attn_comb = self.hc_mixes(x, self.hc_attn_fn, self.hc_attn_scale, self.hc_attn_base)
x = self.hc_pre(x, pre_mix)
x = self.attn_norm(x)
x = self.attn(x, start_pos, *attn_args)
x = self.hc_post(x, residual, attn_post, attn_comb)
residual = x
ffn_pre, ffn_post, ffn_comb = self.hc_mixes(x, self.hc_ffn_fn, self.hc_ffn_scale, self.hc_ffn_base)
x = self.hc_pre(x, attn_pre)
x = self.ffn_norm(x)
x = self.ffn(x, image_mask)
x = self.hc_post(x, residual, ffn_post, ffn_comb)
return x, ffn_pre def hc_mixes(self, x: torch.Tensor, hc_fn: torch.Tensor, hc_scale: torch.Tensor, hc_base: torch.Tensor):
"""x: [b,s,hc,d], hc_fn: [mix_hc, hc*d], hc_scale: [3], hc_base: [mix_hc]. Returns the
pre / post / comb coefficients, split out of one projection of the flattened stream."""
# normalized over the whole flattened hc*d stream, one statistic per token
x = x.flatten(2).float()
rsqrt = torch.rsqrt(x.square().mean(-1, keepdim=True) + self.norm_eps)
mixes = F.linear(x, hc_fn) * rsqrt
return hc_split_sinkhorn(mixes, hc_scale, hc_base, self.hc_mult, self.hc_sinkhorn_iters, self.hc_eps)
def hc_pre(self, x: torch.Tensor, pre_mix: torch.Tensor):
"""Collapse the hc copies into one, weighted by pre_mix. [b,s,hc,d] x [b,s,hc] -> [b,s,d]"""
y = torch.sum(pre_mix.unsqueeze(-1) * x.float(), dim=2)
return y.to(x.dtype)
def hc_post(self, x: torch.Tensor, residual: torch.Tensor, post: torch.Tensor, comb: torch.Tensor):
"""Expand the sublayer output back to hc copies and mix the residual in through `comb`.
x: [b,s,d], residual: [b,s,hc,d], post: [b,s,hc], comb: [b,s,hc,hc] -> [b,s,hc,d]"""
y = post.unsqueeze(-1) * x.unsqueeze(-2) + torch.sum(comb.unsqueeze(-1) * residual.unsqueeze(-2), dim=2)
return y.type_as(x)Sources3
Direct evidence
A block owns its attention and mixture, two RMSNorms, and for each sublayer a projection of the flattened streams to 24 mix numbers with 24 offsets and 3 scales.
block-init · deepseek-ai/DeepSeek-V4.1-Flash / inference/model.py
Pinned revision 2cba9e42aa02 ↗ · lines 907–946
Read code excerpt
class Block(nn.Module):
"""A block whose residual stream is `hc_mult` parallel copies (Hyper-Connections).
Attention and FFN each sit between `hc_pre` (collapse the copies into one sublayer input) and
`hc_post` (expand back out, mixing the residual in through `comb`). `hc_mixes` derives all three
coefficient sets from the stream itself, `comb` made doubly stochastic by Sinkhorn.
The coefficients a sublayer computes are used by the *next* one -- see `forward`.
"""
attention_cls = Attention
def __init__(
self,
layer_id: int,
args: ModelArgs,
engram_layout: EngramLayout | None = None,
):
super().__init__()
self.layer_id = layer_id
self.norm_eps = args.norm_eps
self.attn = self.attention_cls(layer_id, args)
self.ffn = MoE(layer_id, args)
self.engram = None
if engram_layout is not None and layer_id in engram_layout.layer_ids:
self.engram = Engram(args, layer_id, engram_layout)
self.attn_norm = RMSNorm(args.dim, self.norm_eps)
self.ffn_norm = RMSNorm(args.dim, self.norm_eps)
self.hc_mult = hc_mult = args.hc_mult
self.hc_sinkhorn_iters = args.hc_sinkhorn_iters
self.hc_eps = args.hc_eps
mix_hc = (2 + hc_mult) * hc_mult
hc_dim = hc_mult * args.dim
with set_dtype(torch.float32):
self.hc_attn_fn = nn.Parameter(torch.empty(mix_hc, hc_dim))
self.hc_ffn_fn = nn.Parameter(torch.empty(mix_hc, hc_dim))
self.hc_attn_base = nn.Parameter(torch.empty(mix_hc))
self.hc_ffn_base = nn.Parameter(torch.empty(mix_hc))
self.hc_attn_scale = nn.Parameter(torch.empty(3))
self.hc_ffn_scale = nn.Parameter(torch.empty(3))Direct evidence
Attention reads the streams with the mix the previous sublayer computed and writes with its own; the mixture reads with the mix attention computed and writes with its own, and hands its reading mix to the next layer.
block-forward · deepseek-ai/DeepSeek-V4.1-Flash / inference/model.py
Pinned revision 2cba9e42aa02 ↗ · lines 968–994
Read code excerpt
def forward(
self,
x: torch.Tensor,
start_pos: int,
pre_mix: torch.Tensor,
image_mask: torch.Tensor | None,
*attn_args,
) -> tuple[torch.Tensor, torch.Tensor]:
"""`pre_mix` collapses the hc_mult copies down to one input for this block's attention. Each
sub-block's own `hc_mixes` produces the mix for the *next* one, so attention uses what the
previous layer's FFN produced and the FFN uses what this attention produced.
image_mask: [b, s] bool, True inside image spans (selects the VL routing bias)."""
residual = x
attn_pre, attn_post, attn_comb = self.hc_mixes(x, self.hc_attn_fn, self.hc_attn_scale, self.hc_attn_base)
x = self.hc_pre(x, pre_mix)
x = self.attn_norm(x)
x = self.attn(x, start_pos, *attn_args)
x = self.hc_post(x, residual, attn_post, attn_comb)
residual = x
ffn_pre, ffn_post, ffn_comb = self.hc_mixes(x, self.hc_ffn_fn, self.hc_ffn_scale, self.hc_ffn_base)
x = self.hc_pre(x, attn_pre)
x = self.ffn_norm(x)
x = self.ffn(x, image_mask)
x = self.hc_post(x, residual, ffn_post, ffn_comb)
return x, ffn_preDirect evidence
The mix numbers are a projection of the flattened streams divided by their root mean square; reading is a weighted sum of the streams; writing adds the sublayer output, weighted per stream, to a combination of the streams.
block-hc · deepseek-ai/DeepSeek-V4.1-Flash / inference/model.py
Pinned revision 2cba9e42aa02 ↗ · lines 948–966
Read code excerpt
def hc_mixes(self, x: torch.Tensor, hc_fn: torch.Tensor, hc_scale: torch.Tensor, hc_base: torch.Tensor):
"""x: [b,s,hc,d], hc_fn: [mix_hc, hc*d], hc_scale: [3], hc_base: [mix_hc]. Returns the
pre / post / comb coefficients, split out of one projection of the flattened stream."""
# normalized over the whole flattened hc*d stream, one statistic per token
x = x.flatten(2).float()
rsqrt = torch.rsqrt(x.square().mean(-1, keepdim=True) + self.norm_eps)
mixes = F.linear(x, hc_fn) * rsqrt
return hc_split_sinkhorn(mixes, hc_scale, hc_base, self.hc_mult, self.hc_sinkhorn_iters, self.hc_eps)
def hc_pre(self, x: torch.Tensor, pre_mix: torch.Tensor):
"""Collapse the hc copies into one, weighted by pre_mix. [b,s,hc,d] x [b,s,hc] -> [b,s,d]"""
y = torch.sum(pre_mix.unsqueeze(-1) * x.float(), dim=2)
return y.to(x.dtype)
def hc_post(self, x: torch.Tensor, residual: torch.Tensor, post: torch.Tensor, comb: torch.Tensor):
"""Expand the sublayer output back to hc copies and mix the residual in through `comb`.
x: [b,s,d], residual: [b,s,hc,d], post: [b,s,hc], comb: [b,s,hc,hc] -> [b,s,hc,d]"""
y = post.unsqueeze(-1) * x.unsqueeze(-2) + torch.sum(comb.unsqueeze(-1) * residual.unsqueeze(-2), dim=2)
return y.type_as(x)