DeepSeek-V3.2-Exp · Select experts

61 transformer blocks › 58 MoE blocks › Mixture of experts › Router

Select experts

learnedscores.topk(self.topk, dim=-1)[1]

Keep 4 of 8 expert groups, then the top 8 experts, ranked with a selection bias.

{\htmlData{light=selected select}{E^{\mathrm{sel}}_i}}=\big[\,i\in\operatorname{top}_{8}({\htmlData{light=affinities sigmoid}{s}}+b)\text{ within the top }4\text{ groups}\,\big]

in: $s$ [B batch, T positions, 256 experts]
out: $E^{\mathrm{sel}}$ [B batch, T positions, 256 experts]
parameters: 256 · × 58 MoE blocks = 14,848

Parameters

256 parameters owned by this component

Exact per-instance accounting
Component	Calculation	Parameters
learned vector	$256$	256
Per-instance total		256

Across 58 independent MoE blocks: $256 \times 58 = 14{,}848$ parameters.

Derived from the pinned reference configuration and inference code. Checkpoint tensors have not been independently counted; FP8 block scales and the multi-token-prediction module are not counted.

Explanation

With bias $b$ added to the affinities, the 256 experts form 8 groups of 32. Each group is ranked by the sum of its two highest biased affinities; experts outside the best 4 groups are excluded, and the 8 highest remaining experts are selected.

The bias $b$ (256 float32 values) is added only when choosing experts; the weights use the unbiased affinities (a parameter of the gate, model.py 675, 693–694, 705).

Codemodel.py 668–675, 687–709

model.py 668–675

        self.dim = args.dim
        self.topk = args.n_activated_experts
        self.n_groups = args.n_expert_groups
        self.topk_groups = args.n_limited_groups
        self.score_func = args.score_func
        self.route_scale = args.route_scale
        self.weight = nn.Parameter(torch.empty(args.n_routed_experts, args.dim))
        self.bias = nn.Parameter(torch.empty(args.n_routed_experts, dtype=torch.float32)) if self.dim == 7168 else None

model.py 687–709

        scores = linear(x.float(), self.weight.float())
        if self.score_func == "softmax":
            scores = scores.softmax(dim=-1)
        else:
            scores = scores.sigmoid()
        original_scores = scores
        if self.bias is not None:
            scores = scores + self.bias
        if self.n_groups > 1:
            scores = scores.view(x.size(0), self.n_groups, -1)
            if self.bias is None:
                group_scores = scores.amax(dim=-1)
            else:
                group_scores = scores.topk(2, dim=-1)[0].sum(dim=-1)
            indices = group_scores.topk(self.topk_groups, dim=-1)[1]
            mask = scores.new_ones(x.size(0), self.n_groups, dtype=bool).scatter_(1, indices, False)
            scores = scores.masked_fill_(mask.unsqueeze(-1), float("-inf")).flatten(1)
        indices = scores.topk(self.topk, dim=-1)[1]
        weights = original_scores.gather(1, indices)
        if self.score_func == "sigmoid":
            weights /= weights.sum(dim=-1, keepdim=True)
        weights *= self.route_scale
        return weights, indices

Sources2

Direct evidence

The router owns one score vector per routed expert and, at this width, a per-expert bias used only for selection.

gate-init · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 668–675

Read code excerpt

        self.dim = args.dim
        self.topk = args.n_activated_experts
        self.n_groups = args.n_expert_groups
        self.topk_groups = args.n_limited_groups
        self.score_func = args.score_func
        self.route_scale = args.route_scale
        self.weight = nn.Parameter(torch.empty(args.n_routed_experts, args.dim))
        self.bias = nn.Parameter(torch.empty(args.n_routed_experts, dtype=torch.float32)) if self.dim == 7168 else None

Direct evidence

Routing: sigmoid scores, bias added for selection, the best groups kept by their top-two scores, the top-8 experts chosen, their unbiased scores renormalized and scaled.

gate-forward · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 687–709

Read code excerpt

        scores = linear(x.float(), self.weight.float())
        if self.score_func == "softmax":
            scores = scores.softmax(dim=-1)
        else:
            scores = scores.sigmoid()
        original_scores = scores
        if self.bias is not None:
            scores = scores + self.bias
        if self.n_groups > 1:
            scores = scores.view(x.size(0), self.n_groups, -1)
            if self.bias is None:
                group_scores = scores.amax(dim=-1)
            else:
                group_scores = scores.topk(2, dim=-1)[0].sum(dim=-1)
            indices = group_scores.topk(self.topk_groups, dim=-1)[1]
            mask = scores.new_ones(x.size(0), self.n_groups, dtype=bool).scatter_(1, indices, False)
            scores = scores.masked_fill_(mask.unsqueeze(-1), float("-inf")).flatten(1)
        indices = scores.topk(self.topk, dim=-1)[1]
        weights = original_scores.gather(1, indices)
        if self.score_func == "sigmoid":
            weights /= weights.sum(dim=-1, keepdim=True)
        weights *= self.route_scale
        return weights, indices