DeepSeek V4.1 · Up projection

40 layers › 2 groups of 6 layers › 5 reusing layers › Mixture of experts › Shared expert

Up projection

learnedself.w3(x) · 844

Map 5,120 input features to 2,304 features in parallel with the gate.

{\htmlData{light=up}{u}}={\htmlData{light=x}{x_{\mathrm{ffn}}}}W_{3}

in: $x_{\mathrm{ffn}}$ [B batch, T positions, 5120 d]
out: $u$ [B batch, T positions, 2304 d_ff]
parameters: 11,796,480 · × 2 groups × 5 layers = 117,964,800

Parameters

11,796,480 parameters per instance.

Exact per-instance accounting
Component	Calculation	Parameters
Weight matrix	$5{,}120 \times 2{,}304$	11,796,480
Per-instance total		11,796,480

This projection has no bias parameter.

The reference module holds the weight output by input, [out, in].

Across 5 independent layers: $11{,}796{,}480 \times 5 = 58{,}982{,}400$ parameters.

Derived from the configured matrix dimensions; the model total matches the checkpoint’s tensors.

Codemodel.py 830–851

model.py 830–851

class Expert(nn.Module):
    """One SwiGLU FFN. The clamps come straight from training, where they keep fp8/fp4 activations in
    range: the up branch is clamped on both sides, the gate branch only from above."""

    def __init__(self, dim: int, inter_dim: int, dtype=None, swiglu_limit=0.0):
        super().__init__()
        self.w1 = Linear(dim, inter_dim, dtype=dtype)
        self.w2 = Linear(inter_dim, dim, dtype=dtype)
        self.w3 = Linear(dim, inter_dim, dtype=dtype)
        self.swiglu_limit = swiglu_limit

    def forward(self, x: torch.Tensor, weights: torch.Tensor | None = None) -> torch.Tensor:
        dtype = x.dtype
        gate = self.w1(x).float()
        up = self.w3(x).float()
        if self.swiglu_limit > 0:
            up = torch.clamp(up, min=-self.swiglu_limit, max=self.swiglu_limit)
            gate = torch.clamp(gate, max=self.swiglu_limit)
        x = F.silu(gate) * up
        if weights is not None:
            x = weights * x
        return self.w2(x.to(dtype))

Sources1

Direct evidence

An expert is SwiGLU with its gate and up branches clamped; the routing weight multiplies the hidden features before the down projection.

expert · deepseek-ai/DeepSeek-V4.1-Flash / inference/model.py
Pinned revision 2cba9e42aa02 ↗ · lines 830–851

Read code excerpt

class Expert(nn.Module):
    """One SwiGLU FFN. The clamps come straight from training, where they keep fp8/fp4 activations in
    range: the up branch is clamped on both sides, the gate branch only from above."""

    def __init__(self, dim: int, inter_dim: int, dtype=None, swiglu_limit=0.0):
        super().__init__()
        self.w1 = Linear(dim, inter_dim, dtype=dtype)
        self.w2 = Linear(inter_dim, dim, dtype=dtype)
        self.w3 = Linear(dim, inter_dim, dtype=dtype)
        self.swiglu_limit = swiglu_limit

    def forward(self, x: torch.Tensor, weights: torch.Tensor | None = None) -> torch.Tensor:
        dtype = x.dtype
        gate = self.w1(x).float()
        up = self.w3(x).float()
        if self.swiglu_limit > 0:
            up = torch.clamp(up, min=-self.swiglu_limit, max=self.swiglu_limit)
            gate = torch.clamp(gate, max=self.swiglu_limit)
        x = F.silu(gate) * up
        if weights is not None:
            x = weights * x
        return self.w2(x.to(dtype))