Embedded state
×tensorinputs_embeds
Learned token vectors entering the first block.
- shape
- [B batch, T positions, 7168 d]
Sources
Contextual evidence from Token embedding
The token embedding is a learned table of vocabulary × width rows, looked up by token ID.
embedding · moonshotai/Kimi-K2-Instruct / modeling_deepseek.py
Pinned revision fd1984e2b7a3 ↗ · lines 1360–1441
Read code excerpt
self.embed_tokens = nn.Embedding(
config.vocab_size, config.hidden_size, self.padding_idx
)… lines 1363–1439 omitted …
if inputs_embeds is None:
inputs_embeds = self.embed_tokens(input_ids)Contextual evidence from Token embedding
The released configuration: width, layers, dense layers, heads, routed and shared experts, routing (sigmoid scores, one expert group, top-8 of 384, route scale 2.827), MLA ranks and head widths, vocabulary, YaRN RoPE.
config · moonshotai/Kimi-K2-Instruct / config.json
Pinned revision fd1984e2b7a3 ↗ · lines 1–68
Read code excerpt
{
"architectures": [
"DeepseekV3ForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"auto_map": {
"AutoConfig": "configuration_deepseek.DeepseekV3Config",
"AutoModel": "modeling_deepseek.DeepseekV3Model",
"AutoModelForCausalLM": "modeling_deepseek.DeepseekV3ForCausalLM"
},
"aux_loss_alpha": 0.001,
"bos_token_id": 163584,
"eos_token_id": 163585,
"first_k_dense_replace": 1,
"hidden_act": "silu",
"hidden_size": 7168,
"initializer_range": 0.02,
"intermediate_size": 18432,
"kv_lora_rank": 512,
"max_position_embeddings": 131072,
"model_type": "kimi_k2",
"moe_intermediate_size": 2048,
"moe_layer_freq": 1,
"n_group": 1,
"n_routed_experts": 384,
"n_shared_experts": 1,
"norm_topk_prob": true,
"num_attention_heads": 64,
"num_experts_per_tok": 8,
"num_hidden_layers": 61,
"num_key_value_heads": 64,
"num_nextn_predict_layers": 0,
"pretraining_tp": 1,
"q_lora_rank": 1536,
"qk_nope_head_dim": 128,
"qk_rope_head_dim": 64,
"quantization_config": {
"activation_scheme": "dynamic",
"fmt": "e4m3",
"quant_method": "fp8",
"weight_block_size": [
128,
128
]
},
"rms_norm_eps": 1e-06,
"rope_theta": 50000.0,
"routed_scaling_factor": 2.827,
"rope_scaling": {
"beta_fast": 1.0,
"beta_slow": 1.0,
"factor": 32.0,
"mscale": 1.0,
"mscale_all_dim": 1.0,
"original_max_position_embeddings": 4096,
"type": "yarn"
},
"scoring_func": "sigmoid",
"seq_aux": true,
"tie_word_embeddings": false,
"topk_group": 1,
"topk_method": "noaux_tc",
"torch_dtype": "bfloat16",
"transformers_version": "4.48.3",
"use_cache": true,
"v_head_dim": 128,
"vocab_size": 163840Contextual evidence from Transformer blocks
The model embeds tokens, numbers positions from the cached length, builds a 4D causal mask, applies all blocks, normalizes and projects to vocabulary logits with an untied head, computing every position’s logits in float32.
transformer · moonshotai/Kimi-K2-Instruct / modeling_deepseek.py
Pinned revision fd1984e2b7a3 ↗ · lines 1360–1616
Read code excerpt
self.embed_tokens = nn.Embedding(
config.vocab_size, config.hidden_size, self.padding_idx
)
self.layers = nn.ModuleList(
[
DeepseekV3DecoderLayer(config, layer_idx)
for layer_idx in range(config.num_hidden_layers)
]
)
self._use_flash_attention_2 = config._attn_implementation == "flash_attention_2"
self.norm = DeepseekV3RMSNorm(config.hidden_size, eps=config.rms_norm_eps)… lines 1371–1422 omitted …
past_key_values_length = 0
if use_cache:
use_legacy_cache = not isinstance(past_key_values, Cache)
if use_legacy_cache:
past_key_values = DynamicCache.from_legacy_cache(past_key_values)
past_key_values_length = past_key_values.get_usable_length(seq_length)
if position_ids is None:
device = input_ids.device if input_ids is not None else inputs_embeds.device
position_ids = torch.arange(
past_key_values_length,
seq_length + past_key_values_length,
dtype=torch.long,
device=device,
)
position_ids = position_ids.unsqueeze(0)… lines 1439–1442 omitted …
if self._use_flash_attention_2:
# 2d mask is passed through the layers
attention_mask = (
attention_mask
if (attention_mask is not None and 0 in attention_mask)
else None
)
else:
# 4d mask is passed through the layers
attention_mask = _prepare_4d_causal_attention_mask(
attention_mask,
(batch_size, seq_length),
inputs_embeds,
past_key_values_length,
)… lines 1458–1521 omitted …
self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)… lines 1523–1613 omitted …
hidden_states = outputs[0]
logits = self.lm_head(hidden_states)
logits = logits.float()Contextual evidence from Transformer blocks
A block applies input RMSNorm, attention and a residual addition, then post-attention RMSNorm, the FFN and a second residual addition; the first first_k_dense_replace blocks use the dense MLP, the rest MoE.
block · moonshotai/Kimi-K2-Instruct / modeling_deepseek.py
Pinned revision fd1984e2b7a3 ↗ · lines 1152–1218
Read code excerpt
self.mlp = (
DeepseekV3MoE(config)
if (
config.n_routed_experts is not None
and layer_idx >= config.first_k_dense_replace
and layer_idx % config.moe_layer_freq == 0
)
else DeepseekV3MLP(config)
)
self.input_layernorm = DeepseekV3RMSNorm(
config.hidden_size, eps=config.rms_norm_eps
)
self.post_attention_layernorm = DeepseekV3RMSNorm(
config.hidden_size, eps=config.rms_norm_eps
)… lines 1167–1197 omitted …
residual = hidden_states
hidden_states = self.input_layernorm(hidden_states)
# Self Attention
hidden_states, self_attn_weights, present_key_value = self.self_attn(
hidden_states=hidden_states,
attention_mask=attention_mask,
position_ids=position_ids,
past_key_value=past_key_value,
output_attentions=output_attentions,
use_cache=use_cache,
**kwargs,
)
hidden_states = residual + hidden_states
# Fully Connected
residual = hidden_states
hidden_states = self.post_attention_layernorm(hidden_states)
hidden_states = self.mlp(hidden_states)
hidden_states = residual + hidden_states