DeepSeek-V3.2-Exp · Normalize weights

61 transformer blocks › 58 MoE blocks › Mixture of experts › Router

Normalize weights

no parametersweights /= weights.sum(dim=-1, keepdim=True); weights *= self.route_scale

Normalize the selected affinities to sum to one, then scale by 2.5.

{\htmlData{light=output normalize}{G_{i}}}=2.5\,\frac{{\htmlData{light=affinities sigmoid}{s_i}}\,{\htmlData{light=selected select}{E^{\mathrm{sel}}_i}}}{\sum_{j}{\htmlData{light=affinities sigmoid}{s_j}}\,{\htmlData{light=selected select}{E^{\mathrm{sel}}_j}}}

in: $s$ [B batch, T positions, 256 experts] $E^{\mathrm{sel}}$ [B batch, T positions, 256 experts]
out: $G$ [B batch, T positions, 256 experts]

Explanation

Unselected experts get weight zero, so each position’s weights sum to 2.5.

Codemodel.py 687–709

model.py 687–709

        scores = linear(x.float(), self.weight.float())
        if self.score_func == "softmax":
            scores = scores.softmax(dim=-1)
        else:
            scores = scores.sigmoid()
        original_scores = scores
        if self.bias is not None:
            scores = scores + self.bias
        if self.n_groups > 1:
            scores = scores.view(x.size(0), self.n_groups, -1)
            if self.bias is None:
                group_scores = scores.amax(dim=-1)
            else:
                group_scores = scores.topk(2, dim=-1)[0].sum(dim=-1)
            indices = group_scores.topk(self.topk_groups, dim=-1)[1]
            mask = scores.new_ones(x.size(0), self.n_groups, dtype=bool).scatter_(1, indices, False)
            scores = scores.masked_fill_(mask.unsqueeze(-1), float("-inf")).flatten(1)
        indices = scores.topk(self.topk, dim=-1)[1]
        weights = original_scores.gather(1, indices)
        if self.score_func == "sigmoid":
            weights /= weights.sum(dim=-1, keepdim=True)
        weights *= self.route_scale
        return weights, indices

Sources1

Direct evidence

Routing: sigmoid scores, bias added for selection, the best groups kept by their top-two scores, the top-8 experts chosen, their unbiased scores renormalized and scaled.

gate-forward · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 687–709

Read code excerpt

        scores = linear(x.float(), self.weight.float())
        if self.score_func == "softmax":
            scores = scores.softmax(dim=-1)
        else:
            scores = scores.sigmoid()
        original_scores = scores
        if self.bias is not None:
            scores = scores + self.bias
        if self.n_groups > 1:
            scores = scores.view(x.size(0), self.n_groups, -1)
            if self.bias is None:
                group_scores = scores.amax(dim=-1)
            else:
                group_scores = scores.topk(2, dim=-1)[0].sum(dim=-1)
            indices = group_scores.topk(self.topk_groups, dim=-1)[1]
            mask = scores.new_ones(x.size(0), self.n_groups, dtype=bool).scatter_(1, indices, False)
            scores = scores.masked_fill_(mask.unsqueeze(-1), float("-inf")).flatten(1)
        indices = scores.topk(self.topk, dim=-1)[1]
        weights = original_scores.gather(1, indices)
        if self.score_func == "sigmoid":
            weights /= weights.sum(dim=-1, keepdim=True)
        weights *= self.route_scale
        return weights, indices