Kimi K3 · Up projection · OpenModelAtlas

93 layers › 68 KDA layers › Latent mixture of experts › Shared experts

Up projection

learnedself.up_proj(x)

Map 7,168 input features to 6,144 features in parallel with the gate.

{\htmlData{light=u up}{u}}={\htmlData{light=x}{x_{\mathrm{ffn}}}}W_{\mathrm{up}}

in: $x_{\mathrm{ffn}}$ [B batch, T positions, 7168 d]
out: $u$ [B batch, T positions, 6144 shared d_ff]
parameters: 44,040,192 · × 68 KDA layers = 2,994,733,056

Parameters

44,040,192 parameters per instance.

Exact per-instance accounting
Component	Calculation	Parameters
Weight matrix	$7{,}168 \times 6{,}144$	44,040,192
Per-instance total		44,040,192

This projection has no bias parameter.

The reference module holds the weight output by input, [out, in].

Across 68 independent KDA layers: $44{,}040{,}192 \times 68 = 2{,}994{,}733{,}056$ parameters.

Derived from the configured matrix dimensions; the model total matches the checkpoint’s tensors.

Codemodeling_kimi_linear.py 273–301

modeling_kimi_linear.py 273–301

class KimiMLP(nn.Module):
    def __init__(self, config: KimiLinearConfig, hidden_size=None, intermediate_size=None):
        super().__init__()
        self.config = config
        self.hidden_size = config.hidden_size if hidden_size is None else hidden_size
        self.intermediate_size = config.intermediate_size if intermediate_size is None else intermediate_size
        self.gate_proj = nn.Linear(
            self.hidden_size, self.intermediate_size, bias=False)
        self.up_proj = nn.Linear(
            self.hidden_size, self.intermediate_size, bias=False)
        self.down_proj = nn.Linear(
            self.intermediate_size, self.hidden_size, bias=False)
        if config.hidden_act == "situ":
            beta, linear_beta = _get_situ_activation_params(config)
            self.act_fn = SituAndMul(
                beta=beta,
                linear_beta=linear_beta,
            )
        else:
            self.act_fn = ACT2FN[config.hidden_act]

    def forward(self, x):
        if self.config.hidden_act == "situ":
            gate_up = torch.cat([self.gate_proj(x), self.up_proj(x)], dim=-1)
            down_proj = self.down_proj(self.act_fn(gate_up))
        else:
            down_proj = self.down_proj(self.act_fn(
                self.gate_proj(x)) * self.up_proj(x))
        return down_proj

Sources1

Direct evidence

The dense feed-forward network: down(SiTU(gate x, up x)).

mlp · moonshotai/Kimi-K3 / modeling_kimi_linear.py
Pinned revision f831ab668142 ↗ · lines 273–301

Read code excerpt

class KimiMLP(nn.Module):
    def __init__(self, config: KimiLinearConfig, hidden_size=None, intermediate_size=None):
        super().__init__()
        self.config = config
        self.hidden_size = config.hidden_size if hidden_size is None else hidden_size
        self.intermediate_size = config.intermediate_size if intermediate_size is None else intermediate_size
        self.gate_proj = nn.Linear(
            self.hidden_size, self.intermediate_size, bias=False)
        self.up_proj = nn.Linear(
            self.hidden_size, self.intermediate_size, bias=False)
        self.down_proj = nn.Linear(
            self.intermediate_size, self.hidden_size, bias=False)
        if config.hidden_act == "situ":
            beta, linear_beta = _get_situ_activation_params(config)
            self.act_fn = SituAndMul(
                beta=beta,
                linear_beta=linear_beta,
            )
        else:
            self.act_fn = ACT2FN[config.hidden_act]

    def forward(self, x):
        if self.config.hidden_act == "situ":
            gate_up = torch.cat([self.gate_proj(x), self.up_proj(x)], dim=-1)
            down_proj = self.down_proj(self.act_fn(gate_up))
        else:
            down_proj = self.down_proj(self.act_fn(
                self.gate_proj(x)) * self.up_proj(x))
        return down_proj