DeepSeek-V3.2-Exp · Input features

61 transformer blocks › 58 MoE blocks › Mixture of experts › Shared expert

Input features

tensorx · 633

The normalized features entering this SwiGLU shared expert.

shape: $x_{\mathrm{ffn}}$ [B batch, T positions, 7168 d]

Sources

Contextual evidence from MoE RMSNorm

RMSNorm owns one gain per feature and computes in float32; with a residual argument it first adds the residual, fusing the residual addition into the next normalization.

rmsnorm · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 280–306

Read code excerpt

    def __init__(self, dim: int, eps: float = 1e-6):
        super().__init__()
        self.dim = dim
        self.eps = eps
        self.weight = nn.Parameter(torch.ones(dim, dtype=torch.float32))

    def forward(self, x: torch.Tensor, residual: Optional[torch.Tensor] = None):
        """
        Forward pass for RMSNorm.

        Args:
            x (torch.Tensor): Input tensor.

        Returns:
            torch.Tensor: Normalized tensor with the same shape as input.
        """
        dtype = x.dtype
        if residual is None:
            x = x.float()
            var = x.pow(2).mean(-1, keepdim=True)
            x = x * torch.rsqrt(var + self.eps)
            return (self.weight * x).to(dtype)
        else:
            x = residual = x.float() + residual.float()
            var = x.pow(2).mean(-1, keepdim=True)
            x = x * torch.rsqrt(var + self.eps)
            return (self.weight * x).to(dtype), residual.to(dtype)

Contextual evidence from MoE RMSNorm

A block applies attention-norm, MLA, ffn-norm and the FFN; the first n_dense_layers blocks use the dense MLP, the rest MoE. Residual additions happen inside the next normalization.

block · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 826–851

Read code excerpt

        self.attn = MLA(args)
        self.ffn = MLP(args.dim, args.inter_dim) if layer_id < args.n_dense_layers else MoE(args)
        self.attn_norm = RMSNorm(args.dim)
        self.ffn_norm = RMSNorm(args.dim)

… lines 830–843 omitted …

        if residual is None:
            x, residual = self.attn_norm(x), x
        else:
            x, residual = self.attn_norm(x, residual)
        x = self.attn(x, start_pos, freqs_cis, mask)
        x, residual = self.ffn_norm(x, residual)
        x = self.ffn(x)
        return x, residual

Contextual evidence from Gate projection

The dense feed-forward network is a SwiGLU MLP: w2(silu(w1 x) ⊙ w3 x).

mlp · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 629–643

Read code excerpt

        self.w1 = ColumnParallelLinear(dim, inter_dim)
        self.w2 = RowParallelLinear(inter_dim, dim, reduce_output=reduce_output)
        self.w3 = ColumnParallelLinear(dim, inter_dim)

… lines 632–642 omitted …

        return self.w2((F.silu(self.w1(x).float()) * self.w3(x).float()).type_as(x))

Contextual evidence from Gate projection

Linear layers have no bias unless asked. FP8 weights carry one float32 scale per 128 × 128 block; the scales are quantization metadata and are excluded from parameter counts.

linear · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 15–193

Read code excerpt

block_size = 128

… lines 16–178 omitted …

    def __init__(self, in_features: int, out_features: int, bias: bool = False, dtype = None):
        super().__init__()
        self.in_features = in_features
        self.out_features = out_features
        self.weight = nn.Parameter(torch.empty(out_features, in_features, dtype=dtype or Linear.dtype))
        if self.weight.element_size() == 1:
            scale_out_features = (out_features + block_size - 1) // block_size
            scale_in_features = (in_features + block_size - 1) // block_size
            self.weight.scale = self.scale = nn.Parameter(torch.empty(scale_out_features, scale_in_features, dtype=torch.float32))
        else:
            self.register_parameter("scale", None)
        if bias:
            self.bias = nn.Parameter(torch.empty(out_features))
        else:
            self.register_parameter("bias", None)