Kimi K2 · Architecture · OpenModelAtlas

in: Raw text
out: $L$ [B, T, 163840]
parameters: 1,026,408,232,448 in all

Parameters

1,026,408,232,448 parameters in Kimi K2

Exact model ownership reconciliation
Component	Calculation	Parameters
Token embedding	$163{,}840 \times 7{,}168$	1,174,405,120
Transformer blocks	1,024,059,415,040 across its components	1,024,059,415,040
Final RMSNorm	$7{,}168$	7,168
Vocabulary projection	$7{,}168 \times 163{,}840$	1,174,405,120
Model total		1,026,408,232,448

Derived exact parameter total: 1,026,408,232,448. Sources contains the pinned configuration and implementation used for this derivation.

Derived from the pinned configuration and implementation, and equal to the count of the checkpoint’s tensors read from its safetensors headers (FP8 block scales and stored rotary buffers excluded).

In numbers

Fact sheet

Blocks: 61: the first dense, then 60 MoEconfig.json 15 config.json 31
Width: 7,168config.json 17
Attention: 64 heads; keys 128 + 64 rotary features, values 128; latents: query 1,536, key–value 512config.json 20 config.json 29 config.json 35–37 config.json 67
Feed-forward: 384 routed + 1 shared experts; 8 routed chosen per token; 2,048 hidden features eachThe first block has a dense SwiGLU of 18,432 hidden features instead.config.json 19 config.json 23–30
Vocabulary: 163,840config.json 68
Context: 131,072 positions (max_position_embeddings)config.json 21
Parameters: 1,026.4 B in all · 32.9 B used per token (3.2%)Both counts include the embedding table and output head whole; without those two tables, 30.5 B per token.config.json 26 config.json 30 modeling_deepseek.py 523–533 modeling_deepseek.py 536–609
KV cache: 2.4 MiB per position, all layersEvery head’s keys (192) and values (128) in bfloat16, as the implementation caches them. Written in Prefill and Decode; the Full sequence view draws none.modeling_deepseek.py 810–814 config.json 64

Weights · whole model

Kimi K2 · FP8958.3 GiB

Tied readout · FP8956.1 GiB

1,026,408,232,448 parameters: FP8 matrices, block scales left out; the embedding table, the output head, the norms and the router in bfloat16, the configuration’s torch_dtype. The separate output head holds 1,174,405,120 of them (0.11%), 2.2 GiB in bfloat16; tied to the embedding table, the weights would take 956.1 GiB.

config.json 17 config.json 68 config.json 64 config.json 38–46 modeling_deepseek.py 409–415 config.json 61 modeling_deepseek.py 1522

Sources

Direct evidence

The released configuration: width, layers, dense layers, heads, routed and shared experts, routing (sigmoid scores, one expert group, top-8 of 384, route scale 2.827), MLA ranks and head widths, vocabulary, YaRN RoPE.

config · moonshotai/Kimi-K2-Instruct / config.json
Pinned revision fd1984e2b7a3 ↗ · lines 1–68

Read code excerpt

{
  "architectures": [
    "DeepseekV3ForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "auto_map": {
    "AutoConfig": "configuration_deepseek.DeepseekV3Config",
    "AutoModel": "modeling_deepseek.DeepseekV3Model",
    "AutoModelForCausalLM": "modeling_deepseek.DeepseekV3ForCausalLM"
  },
  "aux_loss_alpha": 0.001,
  "bos_token_id": 163584,
  "eos_token_id": 163585,
  "first_k_dense_replace": 1,
  "hidden_act": "silu",
  "hidden_size": 7168,
  "initializer_range": 0.02,
  "intermediate_size": 18432,
  "kv_lora_rank": 512,
  "max_position_embeddings": 131072,
  "model_type": "kimi_k2",
  "moe_intermediate_size": 2048,
  "moe_layer_freq": 1,
  "n_group": 1,
  "n_routed_experts": 384,
  "n_shared_experts": 1,
  "norm_topk_prob": true,
  "num_attention_heads": 64,
  "num_experts_per_tok": 8,
  "num_hidden_layers": 61,
  "num_key_value_heads": 64,
  "num_nextn_predict_layers": 0,
  "pretraining_tp": 1,
  "q_lora_rank": 1536,
  "qk_nope_head_dim": 128,
  "qk_rope_head_dim": 64,
  "quantization_config": {
    "activation_scheme": "dynamic",
    "fmt": "e4m3",
    "quant_method": "fp8",
    "weight_block_size": [
      128,
      128
    ]
  },
  "rms_norm_eps": 1e-06,
  "rope_theta": 50000.0,
  "routed_scaling_factor": 2.827,
  "rope_scaling": {
    "beta_fast": 1.0,
    "beta_slow": 1.0,
    "factor": 32.0,
    "mscale": 1.0,
    "mscale_all_dim": 1.0,
    "original_max_position_embeddings": 4096,
    "type": "yarn"
  },
  "scoring_func": "sigmoid",
  "seq_aux": true,
  "tie_word_embeddings": false,
  "topk_group": 1,
  "topk_method": "noaux_tc",
  "torch_dtype": "bfloat16",
  "transformers_version": "4.48.3",
  "use_cache": true,
  "v_head_dim": 128,
  "vocab_size": 163840

Direct evidence

YaRN RoPE as the implementation builds it: the YaRN class and its defaults, the softmax scale (1/√192 times the square of the YaRN attention correction for mscale_all_dim), and the choice of YaRN whenever the configuration’s rope_scaling type is yarn.

model-args · moonshotai/Kimi-K2-Instruct / modeling_deepseek.py
Pinned revision fd1984e2b7a3 ↗ · lines 262–742

Read code excerpt

class DeepseekV3YarnRotaryEmbedding(DeepseekV3RotaryEmbedding):

    def __init__(
        self,
        dim,
        max_position_embeddings=2048,
        base=10000,
        device=None,
        scaling_factor=1.0,
        original_max_position_embeddings=4096,
        beta_fast=32,
        beta_slow=1,
        mscale=1,
        mscale_all_dim=0,
    ):
        self.scaling_factor = scaling_factor
        self.original_max_position_embeddings = original_max_position_embeddings
        self.beta_fast = beta_fast
        self.beta_slow = beta_slow
        self.mscale = mscale
        self.mscale_all_dim = mscale_all_dim
        super().__init__(dim, max_position_embeddings, base, device)

… lines 284–689 omitted …

        self.softmax_scale = self.q_head_dim ** (-0.5)
        if self.config.rope_scaling is not None:
            mscale_all_dim = self.config.rope_scaling.get("mscale_all_dim", 0)
            scaling_factor = self.config.rope_scaling["factor"]
            if mscale_all_dim:
                mscale = yarn_get_mscale(scaling_factor, mscale_all_dim)
                self.softmax_scale = self.softmax_scale * mscale * mscale

… lines 697–697 omitted …

    def _init_rope(self):
        if self.config.rope_scaling is None:
            self.rotary_emb = DeepseekV3RotaryEmbedding(
                self.qk_rope_head_dim,
                max_position_embeddings=self.max_position_embeddings,
                base=self.rope_theta,
            )
        else:
            scaling_type = self.config.rope_scaling["type"]
            scaling_factor = self.config.rope_scaling["factor"]
            if scaling_type == "linear":
                self.rotary_emb = DeepseekV3LinearScalingRotaryEmbedding(
                    self.qk_rope_head_dim,
                    max_position_embeddings=self.max_position_embeddings,
                    scaling_factor=scaling_factor,
                    base=self.rope_theta,
                )
            elif scaling_type == "dynamic":
                self.rotary_emb = DeepseekV3DynamicNTKScalingRotaryEmbedding(
                    self.qk_rope_head_dim,
                    max_position_embeddings=self.max_position_embeddings,
                    scaling_factor=scaling_factor,
                    base=self.rope_theta,
                )
            elif scaling_type == "yarn":
                kwargs = {
                    key: self.config.rope_scaling[key]
                    for key in [
                        "original_max_position_embeddings",
                        "beta_fast",
                        "beta_slow",
                        "mscale",
                        "mscale_all_dim",
                    ]
                    if key in self.config.rope_scaling
                }
                self.rotary_emb = DeepseekV3YarnRotaryEmbedding(
                    self.qk_rope_head_dim,
                    max_position_embeddings=self.max_position_embeddings,
                    scaling_factor=scaling_factor,
                    base=self.rope_theta,
                    **kwargs,
                )
            else:
                raise ValueError(f"Unknown RoPE scaling type {scaling_type}")

Direct evidence

The model embeds tokens, numbers positions from the cached length, builds a 4D causal mask, applies all blocks, normalizes and projects to vocabulary logits with an untied head, computing every position’s logits in float32.

transformer · moonshotai/Kimi-K2-Instruct / modeling_deepseek.py
Pinned revision fd1984e2b7a3 ↗ · lines 1360–1616

Read code excerpt

        self.embed_tokens = nn.Embedding(
            config.vocab_size, config.hidden_size, self.padding_idx
        )
        self.layers = nn.ModuleList(
            [
                DeepseekV3DecoderLayer(config, layer_idx)
                for layer_idx in range(config.num_hidden_layers)
            ]
        )
        self._use_flash_attention_2 = config._attn_implementation == "flash_attention_2"
        self.norm = DeepseekV3RMSNorm(config.hidden_size, eps=config.rms_norm_eps)

… lines 1371–1422 omitted …

        past_key_values_length = 0
        if use_cache:
            use_legacy_cache = not isinstance(past_key_values, Cache)
            if use_legacy_cache:
                past_key_values = DynamicCache.from_legacy_cache(past_key_values)
            past_key_values_length = past_key_values.get_usable_length(seq_length)

        if position_ids is None:
            device = input_ids.device if input_ids is not None else inputs_embeds.device
            position_ids = torch.arange(
                past_key_values_length,
                seq_length + past_key_values_length,
                dtype=torch.long,
                device=device,
            )
            position_ids = position_ids.unsqueeze(0)

… lines 1439–1442 omitted …

        if self._use_flash_attention_2:
            # 2d mask is passed through the layers
            attention_mask = (
                attention_mask
                if (attention_mask is not None and 0 in attention_mask)
                else None
            )
        else:
            # 4d mask is passed through the layers
            attention_mask = _prepare_4d_causal_attention_mask(
                attention_mask,
                (batch_size, seq_length),
                inputs_embeds,
                past_key_values_length,
            )

… lines 1458–1521 omitted …

        self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)

… lines 1523–1613 omitted …

        hidden_states = outputs[0]
        logits = self.lm_head(hidden_states)
        logits = logits.float()