DeepSeek-V3.2-Exp

All models, with their blocks →

Size
671B
Published
DeepSeek · Sep 2025
Line
Latent attention and experts
View
Source

deepseek‑ai/DeepSeek‑V3.2‑Exp @ 87e509a · checkpoint, tokenizer and tensors not verifiedHow to read

Mixture of experts

61 transformer blocks › 58 MoE blocks › Mixture of experts

Routed output

×

tensory · 800

The weighted sum of the selected experts.

shape
yroutedy^{\mathrm{routed}} [B batch, T positions, 7168 d]
Sources

Contextual evidence from Weighted expert sum

Each token’s output is the router-weighted sum of its selected experts plus the shared expert applied to every token.

moe-forward · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 790–803

Read code excerpt
        shape = x.size()
        x = x.view(-1, self.dim)
        weights, indices = self.gate(x)
        y = torch.zeros_like(x, dtype=torch.float32)
        counts = torch.bincount(indices.flatten(), minlength=self.n_routed_experts).tolist()
        for i in range(self.experts_start_idx, self.experts_end_idx):
            if counts[i] == 0:
                continue
            expert = self.experts[i]
            idx, top = torch.where(indices == i)
            y[idx] += expert(x[idx]) * weights[idx, top, None]
        y += self.shared_experts(x)
        if world_size > 1:
            dist.all_reduce(y)