DeepSeek-V3.2-Exp · Gate projection

61 transformer blocks › 58 MoE blocks › Mixture of experts › 256 routed experts

Gate projection

learnedself.w1(x)

Map 7,168 input features to 2,048 gate features.

{\htmlData{light=g gate}{g}}={\htmlData{light=x}{x_{\mathrm{ffn}}}}W_{1}

in: $x_{\mathrm{ffn}}$ [B batch, T positions, 7168 d]
out: $g$ [B batch, T positions, 2048 expert d_ff]
parameters: 14,680,064 · × 58 MoE blocks × 256 experts = 217,969,590,272

Parameters

14,680,064 parameters per instance.

Exact per-instance accounting
Component	Calculation	Parameters
Weight matrix	$7{,}168 \times 2{,}048$	14,680,064
Per-instance total		14,680,064

This projection has no bias parameter.

The reference module holds the weight output by input, [out, in].

Across 256 independent experts: $14{,}680{,}064 \times 256 = 3{,}758{,}096{,}384$ parameters.

Derived from the configured matrix dimensions; checkpoint tensors have not been independently counted.

Codemodel.py 15–744

model.py 730–732

        self.w1 = Linear(dim, inter_dim)
        self.w2 = Linear(inter_dim, dim)
        self.w3 = Linear(dim, inter_dim)

model.py 744

        return self.w2((F.silu(self.w1(x).float()) * self.w3(x).float()).type_as(x))

model.py 15

block_size = 128

model.py 179–193

    def __init__(self, in_features: int, out_features: int, bias: bool = False, dtype = None):
        super().__init__()
        self.in_features = in_features
        self.out_features = out_features
        self.weight = nn.Parameter(torch.empty(out_features, in_features, dtype=dtype or Linear.dtype))
        if self.weight.element_size() == 1:
            scale_out_features = (out_features + block_size - 1) // block_size
            scale_in_features = (in_features + block_size - 1) // block_size
            self.weight.scale = self.scale = nn.Parameter(torch.empty(scale_out_features, scale_in_features, dtype=torch.float32))
        else:
            self.register_parameter("scale", None)
        if bias:
            self.bias = nn.Parameter(torch.empty(out_features))
        else:
            self.register_parameter("bias", None)

Sources2

Direct evidence

Each routed expert is its own SwiGLU MLP of moe_inter_dim hidden features.

expert · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 730–744

Read code excerpt

        self.w1 = Linear(dim, inter_dim)
        self.w2 = Linear(inter_dim, dim)
        self.w3 = Linear(dim, inter_dim)

… lines 733–743 omitted …

        return self.w2((F.silu(self.w1(x).float()) * self.w3(x).float()).type_as(x))

Direct evidence

Linear layers have no bias unless asked. FP8 weights carry one float32 scale per 128 × 128 block; the scales are quantization metadata and are excluded from parameter counts.

linear · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 15–193

Read code excerpt

block_size = 128

… lines 16–178 omitted …

    def __init__(self, in_features: int, out_features: int, bias: bool = False, dtype = None):
        super().__init__()
        self.in_features = in_features
        self.out_features = out_features
        self.weight = nn.Parameter(torch.empty(out_features, in_features, dtype=dtype or Linear.dtype))
        if self.weight.element_size() == 1:
            scale_out_features = (out_features + block_size - 1) // block_size
            scale_in_features = (in_features + block_size - 1) // block_size
            self.weight.scale = self.scale = nn.Parameter(torch.empty(scale_out_features, scale_in_features, dtype=torch.float32))
        else:
            self.register_parameter("scale", None)
        if bias:
            self.bias = nn.Parameter(torch.empty(out_features))
        else:
            self.register_parameter("bias", None)