DeepSeek-V3.2-Exp · Routed expert

in: $x_{\mathrm{ffn}}$ [B batch, T positions, 7168 d]
out: $E_i$ [B batch, T positions, 7168 d]
parameters: 44,040,192 · × 58 MoE blocks × 256 experts = 653,908,770,816

Parameters

44,040,192 parameters per expert, across its learned components

Exact per-instance accounting
Component	Calculation	Parameters
Gate projection	$7{,}168 \times 2{,}048$	14,680,064
Up projection	$7{,}168 \times 2{,}048$	14,680,064
Down projection	$2{,}048 \times 7{,}168$	14,680,064
Per-instance total		44,040,192

Across 256 independent experts: $44{,}040{,}192 \times 256 = 11{,}274{,}289{,}152$ parameters.

Derived from the pinned reference configuration and inference code. Checkpoint tensors have not been independently counted; FP8 block scales and the multi-token-prediction module are not counted.

Explanation

The reference runs expert $i$ only on the positions routed to it and adds its weighted output back at those positions (model.py 794–800).

Codemodel.py 730–803

model.py 730–732

        self.w1 = Linear(dim, inter_dim)
        self.w2 = Linear(inter_dim, dim)
        self.w3 = Linear(dim, inter_dim)

model.py 744

        return self.w2((F.silu(self.w1(x).float()) * self.w3(x).float()).type_as(x))

model.py 770–778

        self.n_routed_experts = args.n_routed_experts
        self.n_local_experts = args.n_routed_experts // world_size
        self.n_activated_experts = args.n_activated_experts
        self.experts_start_idx = rank * self.n_local_experts
        self.experts_end_idx = self.experts_start_idx + self.n_local_experts
        self.gate = Gate(args)
        self.experts = nn.ModuleList([Expert(args.dim, args.moe_inter_dim) if self.experts_start_idx <= i < self.experts_end_idx else None
                                      for i in range(self.n_routed_experts)])
        self.shared_experts = MLP(args.dim, args.n_shared_experts * args.moe_inter_dim, reduce_output=False)

model.py 790–803

        shape = x.size()
        x = x.view(-1, self.dim)
        weights, indices = self.gate(x)
        y = torch.zeros_like(x, dtype=torch.float32)
        counts = torch.bincount(indices.flatten(), minlength=self.n_routed_experts).tolist()
        for i in range(self.experts_start_idx, self.experts_end_idx):
            if counts[i] == 0:
                continue
            expert = self.experts[i]
            idx, top = torch.where(indices == i)
            y[idx] += expert(x[idx]) * weights[idx, top, None]
        y += self.shared_experts(x)
        if world_size > 1:
            dist.all_reduce(y)

Sources3

Direct evidence

Each routed expert is its own SwiGLU MLP of moe_inter_dim hidden features.

expert · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 730–744

Read code excerpt

        self.w1 = Linear(dim, inter_dim)
        self.w2 = Linear(inter_dim, dim)
        self.w3 = Linear(dim, inter_dim)

… lines 733–743 omitted …

        return self.w2((F.silu(self.w1(x).float()) * self.w3(x).float()).type_as(x))

Direct evidence

The MoE layer holds the router, the routed experts and one shared expert MLP of n_shared_experts × moe_inter_dim features.

moe-init · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 770–778

Read code excerpt

        self.n_routed_experts = args.n_routed_experts
        self.n_local_experts = args.n_routed_experts // world_size
        self.n_activated_experts = args.n_activated_experts
        self.experts_start_idx = rank * self.n_local_experts
        self.experts_end_idx = self.experts_start_idx + self.n_local_experts
        self.gate = Gate(args)
        self.experts = nn.ModuleList([Expert(args.dim, args.moe_inter_dim) if self.experts_start_idx <= i < self.experts_end_idx else None
                                      for i in range(self.n_routed_experts)])
        self.shared_experts = MLP(args.dim, args.n_shared_experts * args.moe_inter_dim, reduce_output=False)

Direct evidence

Each token’s output is the router-weighted sum of its selected experts plus the shared expert applied to every token.

moe-forward · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 790–803

Read code excerpt

        shape = x.size()
        x = x.view(-1, self.dim)
        weights, indices = self.gate(x)
        y = torch.zeros_like(x, dtype=torch.float32)
        counts = torch.bincount(indices.flatten(), minlength=self.n_routed_experts).tolist()
        for i in range(self.experts_start_idx, self.experts_end_idx):
            if counts[i] == 0:
                continue
            expert = self.experts[i]
            idx, top = torch.where(indices == i)
            y[idx] += expert(x[idx]) * weights[idx, top, None]
        y += self.shared_experts(x)
        if world_size > 1:
            dist.all_reduce(y)