DeepSeek-V3.2-Exp · Weighted expert sum

61 transformer blocks › 58 MoE blocks › Mixture of experts

Weighted expert sum

no parametersy[idx] += expert(x[idx]) * weights[idx, top, None]

Sum the selected experts’ outputs, each scaled by its routing weight.

{\htmlData{light=routed combine}{y^{\mathrm{routed}}}}=\sum_{i=1}^{256}{\htmlData{light=gates router}{G_i}}\,{\htmlData{light=expert-outputs experts}{E_i}}

in: $G$ [B batch, T positions, 256 experts] $E_i$ [B batch, T positions, 7168 d]
out: $y^{\mathrm{routed}}$ [B batch, T positions, 7168 d]

Accumulated in float32 and cast back at the end (model.py 793, 800, 804).

Codemodel.py 790–803

model.py 790–803

        shape = x.size()
        x = x.view(-1, self.dim)
        weights, indices = self.gate(x)
        y = torch.zeros_like(x, dtype=torch.float32)
        counts = torch.bincount(indices.flatten(), minlength=self.n_routed_experts).tolist()
        for i in range(self.experts_start_idx, self.experts_end_idx):
            if counts[i] == 0:
                continue
            expert = self.experts[i]
            idx, top = torch.where(indices == i)
            y[idx] += expert(x[idx]) * weights[idx, top, None]
        y += self.shared_experts(x)
        if world_size > 1:
            dist.all_reduce(y)

Sources1

Direct evidence

Each token’s output is the router-weighted sum of its selected experts plus the shared expert applied to every token.

moe-forward · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 790–803

Read code excerpt

        shape = x.size()
        x = x.view(-1, self.dim)
        weights, indices = self.gate(x)
        y = torch.zeros_like(x, dtype=torch.float32)
        counts = torch.bincount(indices.flatten(), minlength=self.n_routed_experts).tolist()
        for i in range(self.experts_start_idx, self.experts_end_idx):
            if counts[i] == 0:
                continue
            expert = self.experts[i]
            idx, top = torch.where(indices == i)
            y[idx] += expert(x[idx]) * weights[idx, top, None]
        y += self.shared_experts(x)
        if world_size > 1:
            dist.all_reduce(y)