Kimi K3 · Gate projection · OpenModelAtlas

93 layers › 68 KDA layers › Latent mixture of experts › 896 routed experts

Gate projection

learnedself.w1(x)

Map 3,584 input features to 3,072 gate features.

{\htmlData{light=g gate}{g}}={\htmlData{light=x}{x^{c}}}W_{\mathrm{gate}}

in: $x^{c}$ [B batch, 1 positions, 3584]
out: $g$ [B batch, 1 positions, 3072 expert d_ff]
parameters: 11,010,048 · × 68 KDA layers × 896 experts = 670,820,204,544

Parameters

11,010,048 parameters per instance.

Exact per-instance accounting
Component	Calculation	Parameters
Weight matrix	$3{,}584 \times 3{,}072$	11,010,048
Per-instance total		11,010,048

This projection has no bias parameter.

The reference module holds the weight output by input, [out, in].

Across 896 independent experts: $11{,}010{,}048 \times 896 = 9{,}865{,}003{,}008$ parameters.

Derived from the configured matrix dimensions; the model total matches the checkpoint’s tensors.

Codemodeling_kimi_linear.py 242–270

modeling_kimi_linear.py 242–270

class KimiBlockSparseMLP(nn.Module):
    def __init__(self, config: KimiLinearConfig, hidden_size=None, intermediate_size=None):
        super().__init__()
        self.config = config
        self.ffn_dim = config.intermediate_size if intermediate_size is None else intermediate_size
        self.hidden_dim = config.hidden_size if hidden_size is None else hidden_size

        self.w1 = nn.Linear(self.hidden_dim, self.ffn_dim, bias=False)   # gate
        self.w2 = nn.Linear(self.ffn_dim, self.hidden_dim, bias=False)   # down
        self.w3 = nn.Linear(self.hidden_dim, self.ffn_dim, bias=False)   # up

        if config.hidden_act == "situ":
            beta, linear_beta = _get_situ_activation_params(config)
            self.act_fn = SituAndMul(
                beta=beta,
                linear_beta=linear_beta,
            )
        else:
            self.act_fn = ACT2FN[config.hidden_act]

    def forward(self, hidden_states):
        if self.config.hidden_act == "situ":
            gate_up = torch.cat([self.w1(hidden_states), self.w3(hidden_states)], dim=-1)
            current_hidden_states = self.act_fn(gate_up)
        else:
            current_hidden_states = self.act_fn(
                self.w1(hidden_states)) * self.w3(hidden_states)
        current_hidden_states = self.w2(current_hidden_states)
        return current_hidden_states

Sources1

Direct evidence

Each routed expert is its own gated MLP of moe_intermediate_size hidden features on the routed-expert width, with the SiTU activation.

expert · moonshotai/Kimi-K3 / modeling_kimi_linear.py
Pinned revision f831ab668142 ↗ · lines 242–270

Read code excerpt

class KimiBlockSparseMLP(nn.Module):
    def __init__(self, config: KimiLinearConfig, hidden_size=None, intermediate_size=None):
        super().__init__()
        self.config = config
        self.ffn_dim = config.intermediate_size if intermediate_size is None else intermediate_size
        self.hidden_dim = config.hidden_size if hidden_size is None else hidden_size

        self.w1 = nn.Linear(self.hidden_dim, self.ffn_dim, bias=False)   # gate
        self.w2 = nn.Linear(self.ffn_dim, self.hidden_dim, bias=False)   # down
        self.w3 = nn.Linear(self.hidden_dim, self.ffn_dim, bias=False)   # up

        if config.hidden_act == "situ":
            beta, linear_beta = _get_situ_activation_params(config)
            self.act_fn = SituAndMul(
                beta=beta,
                linear_beta=linear_beta,
            )
        else:
            self.act_fn = ACT2FN[config.hidden_act]

    def forward(self, hidden_states):
        if self.config.hidden_act == "situ":
            gate_up = torch.cat([self.w1(hidden_states), self.w3(hidden_states)], dim=-1)
            current_hidden_states = self.act_fn(gate_up)
        else:
            current_hidden_states = self.act_fn(
                self.w1(hidden_states)) * self.w3(hidden_states)
        current_hidden_states = self.w2(current_hidden_states)
        return current_hidden_states