61 transformer blocks › 1 dense block
Block input
×tensorresidual · 1198
The residual features entering this block.
- shape
- [B batch, T positions, 7168 d]
Sources
Contextual evidence from Token embedding
The token embedding is a learned table of vocabulary × width rows, looked up by token ID.
embedding · moonshotai/Kimi-K2-Instruct / modeling_deepseek.py
Pinned revision fd1984e2b7a3 ↗ · lines 1360–1441
Read code excerpt
self.embed_tokens = nn.Embedding(
config.vocab_size, config.hidden_size, self.padding_idx
)… lines 1363–1439 omitted …
if inputs_embeds is None:
inputs_embeds = self.embed_tokens(input_ids)Contextual evidence from Token embedding
The released configuration: width, layers, dense layers, heads, routed and shared experts, routing (sigmoid scores, one expert group, top-8 of 384, route scale 2.827), MLA ranks and head widths, vocabulary, YaRN RoPE.
config · moonshotai/Kimi-K2-Instruct / config.json
Pinned revision fd1984e2b7a3 ↗ · lines 1–68
Read code excerpt
{
"architectures": [
"DeepseekV3ForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"auto_map": {
"AutoConfig": "configuration_deepseek.DeepseekV3Config",
"AutoModel": "modeling_deepseek.DeepseekV3Model",
"AutoModelForCausalLM": "modeling_deepseek.DeepseekV3ForCausalLM"
},
"aux_loss_alpha": 0.001,
"bos_token_id": 163584,
"eos_token_id": 163585,
"first_k_dense_replace": 1,
"hidden_act": "silu",
"hidden_size": 7168,
"initializer_range": 0.02,
"intermediate_size": 18432,
"kv_lora_rank": 512,
"max_position_embeddings": 131072,
"model_type": "kimi_k2",
"moe_intermediate_size": 2048,
"moe_layer_freq": 1,
"n_group": 1,
"n_routed_experts": 384,
"n_shared_experts": 1,
"norm_topk_prob": true,
"num_attention_heads": 64,
"num_experts_per_tok": 8,
"num_hidden_layers": 61,
"num_key_value_heads": 64,
"num_nextn_predict_layers": 0,
"pretraining_tp": 1,
"q_lora_rank": 1536,
"qk_nope_head_dim": 128,
"qk_rope_head_dim": 64,
"quantization_config": {
"activation_scheme": "dynamic",
"fmt": "e4m3",
"quant_method": "fp8",
"weight_block_size": [
128,
128
]
},
"rms_norm_eps": 1e-06,
"rope_theta": 50000.0,
"routed_scaling_factor": 2.827,
"rope_scaling": {
"beta_fast": 1.0,
"beta_slow": 1.0,
"factor": 32.0,
"mscale": 1.0,
"mscale_all_dim": 1.0,
"original_max_position_embeddings": 4096,
"type": "yarn"
},
"scoring_func": "sigmoid",
"seq_aux": true,
"tie_word_embeddings": false,
"topk_group": 1,
"topk_method": "noaux_tc",
"torch_dtype": "bfloat16",
"transformers_version": "4.48.3",
"use_cache": true,
"v_head_dim": 128,
"vocab_size": 163840Contextual evidence from Attention RMSNorm
RMSNorm owns one gain per feature and computes in float32.
rmsnorm · moonshotai/Kimi-K2-Instruct / modeling_deepseek.py
Pinned revision fd1984e2b7a3 ↗ · lines 94–111
Read code excerpt
class DeepseekV3RMSNorm(nn.Module):
def __init__(self, hidden_size, eps=1e-6):
"""
DeepseekV3RMSNorm is equivalent to T5LayerNorm
"""
super().__init__()
self.weight = nn.Parameter(torch.ones(hidden_size))
self.variance_epsilon = eps
def forward(self, hidden_states):
input_dtype = hidden_states.dtype
hidden_states = hidden_states.to(torch.float32)
variance = hidden_states.pow(2).mean(-1, keepdim=True)
hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
return self.weight * hidden_states.to(input_dtype)
ALL_LAYERNORM_LAYERS.append(DeepseekV3RMSNorm)Contextual evidence from Attention RMSNorm
A block applies input RMSNorm, attention and a residual addition, then post-attention RMSNorm, the FFN and a second residual addition; the first first_k_dense_replace blocks use the dense MLP, the rest MoE.
block · moonshotai/Kimi-K2-Instruct / modeling_deepseek.py
Pinned revision fd1984e2b7a3 ↗ · lines 1152–1218
Read code excerpt
self.mlp = (
DeepseekV3MoE(config)
if (
config.n_routed_experts is not None
and layer_idx >= config.first_k_dense_replace
and layer_idx % config.moe_layer_freq == 0
)
else DeepseekV3MLP(config)
)
self.input_layernorm = DeepseekV3RMSNorm(
config.hidden_size, eps=config.rms_norm_eps
)
self.post_attention_layernorm = DeepseekV3RMSNorm(
config.hidden_size, eps=config.rms_norm_eps
)… lines 1167–1197 omitted …
residual = hidden_states
hidden_states = self.input_layernorm(hidden_states)
# Self Attention
hidden_states, self_attn_weights, present_key_value = self.self_attn(
hidden_states=hidden_states,
attention_mask=attention_mask,
position_ids=position_ids,
past_key_value=past_key_value,
output_attentions=output_attentions,
use_cache=use_cache,
**kwargs,
)
hidden_states = residual + hidden_states
# Fully Connected
residual = hidden_states
hidden_states = self.post_attention_layernorm(hidden_states)
hidden_states = self.mlp(hidden_states)
hidden_states = residual + hidden_states