Kimi K2 · Down projection · OpenModelAtlas

61 transformer blocks › 60 MoE blocks › Mixture of experts › 384 routed experts

Down projection

learnedself.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))

Map 2,048 gated features back to 7,168 features.

{\htmlData{light=output down}{E_i}}={\htmlData{light=h product}{h}}W_{2}

in: $h$ [B batch, T positions, 2048 expert d_ff]
out: $E_i$ [B batch, T positions, 7168 d]
parameters: 14,680,064 · × 60 MoE blocks × 384 experts = 338,228,674,560

Parameters

14,680,064 parameters per instance.

Exact per-instance accounting
Component	Calculation	Parameters
Weight matrix	$2{,}048 \times 7{,}168$	14,680,064
Per-instance total		14,680,064

This projection has no bias parameter.

The reference module holds the weight output by input, [out, in].

Across 384 independent experts: $14{,}680{,}064 \times 384 = 5{,}637{,}144{,}576$ parameters.

Derived from the configured matrix dimensions; the model total matches the checkpoint’s tensors.

Codemodeling_deepseek.py 486–515 · config.json 38–46

modeling_deepseek.py 486–515

        if hasattr(config, "ep_size") and config.ep_size > 1:
            assert config.ep_size == dist.get_world_size()
            self.ep_size = config.ep_size
            self.experts_per_rank = config.n_routed_experts // config.ep_size
            self.ep_rank = dist.get_rank()
            self.experts = nn.ModuleList(
                [
                    (
                        DeepseekV3MLP(
                            config, intermediate_size=config.moe_intermediate_size
                        )
                        if i >= self.ep_rank * self.experts_per_rank
                        and i < (self.ep_rank + 1) * self.experts_per_rank
                        else None
                    )
                    for i in range(config.n_routed_experts)
                ]
            )
        else:
            self.ep_size = 1
            self.experts_per_rank = config.n_routed_experts
            self.ep_rank = 0
            self.experts = nn.ModuleList(
                [
                    DeepseekV3MLP(
                        config, intermediate_size=config.moe_intermediate_size
                    )
                    for i in range(config.n_routed_experts)
                ]
            )

config.json 38–46

  "quantization_config": {
    "activation_scheme": "dynamic",
    "fmt": "e4m3",
    "quant_method": "fp8",
    "weight_block_size": [
      128,
      128
    ]
  },

Sources2

Direct evidence

Each routed expert is its own SwiGLU MLP of moe_intermediate_size hidden features.

expert · moonshotai/Kimi-K2-Instruct / modeling_deepseek.py
Pinned revision fd1984e2b7a3 ↗ · lines 486–515

Read code excerpt

        if hasattr(config, "ep_size") and config.ep_size > 1:
            assert config.ep_size == dist.get_world_size()
            self.ep_size = config.ep_size
            self.experts_per_rank = config.n_routed_experts // config.ep_size
            self.ep_rank = dist.get_rank()
            self.experts = nn.ModuleList(
                [
                    (
                        DeepseekV3MLP(
                            config, intermediate_size=config.moe_intermediate_size
                        )
                        if i >= self.ep_rank * self.experts_per_rank
                        and i < (self.ep_rank + 1) * self.experts_per_rank
                        else None
                    )
                    for i in range(config.n_routed_experts)
                ]
            )
        else:
            self.ep_size = 1
            self.experts_per_rank = config.n_routed_experts
            self.ep_rank = 0
            self.experts = nn.ModuleList(
                [
                    DeepseekV3MLP(
                        config, intermediate_size=config.moe_intermediate_size
                    )
                    for i in range(config.n_routed_experts)
                ]
            )

Direct evidence

Weights are stored in FP8 (e4m3) with one scale per 128 × 128 block; the scales are quantization metadata and are excluded from parameter counts.

linear · moonshotai/Kimi-K2-Instruct / config.json
Pinned revision fd1984e2b7a3 ↗ · lines 38–46

Read code excerpt

  "quantization_config": {
    "activation_scheme": "dynamic",
    "fmt": "e4m3",
    "quant_method": "fp8",
    "weight_block_size": [
      128,
      128
    ]
  },