DeepSeek-V3.2-Exp · Token embedding

Token embedding

learnedself.embed(tokens)

Look up a learned feature vector for each token ID.

{\htmlData{light=embedded embedding}{x_0}}=E[{\htmlData{light=token-ids tokenize}{I}}]

in: $I$ [B batch, T positions]
out: $x_0$ [B batch, T positions, 7168 d]
parameters: 926,679,040

Parameters

926,679,040 parameters per instance.

Exact per-instance accounting
Component	Calculation	Parameters
Token embedding	$129{,}280 \times 7{,}168$	926,679,040
Per-instance total		926,679,040

Derived from the configured table dimensions; checkpoint tensors have not been independently counted.

Codemodel.py 108, 127 · config_671B_v3.2.json 1–26

model.py 108

        self.weight = nn.Parameter(torch.empty(self.part_vocab_size, self.dim))

model.py 127

        y = F.embedding(x, self.weight)

config_671B_v3.2.json 1–26

{
    "vocab_size": 129280,
    "dim": 7168,
    "inter_dim": 18432,
    "moe_inter_dim": 2048,
    "n_layers": 61,
    "n_dense_layers": 3,
    "n_heads": 128,
    "n_routed_experts": 256,
    "n_shared_experts": 1,
    "n_activated_experts": 8,
    "n_expert_groups": 8,
    "n_limited_groups": 4,
    "route_scale": 2.5,
    "score_func": "sigmoid",
    "q_lora_rank": 1536,
    "kv_lora_rank": 512,
    "qk_nope_head_dim": 128,
    "qk_rope_head_dim": 64,
    "v_head_dim": 128,
    "dtype": "fp8",
    "scale_fmt": "ue8m0",
    "index_n_heads": 64,
    "index_head_dim": 128,
    "index_topk": 2048
}

Sources2

Direct evidence

The token embedding is a learned table of vocabulary × width rows, looked up by token ID.

embedding · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 108–127

Read code excerpt

        self.weight = nn.Parameter(torch.empty(self.part_vocab_size, self.dim))

… lines 109–126 omitted …

        y = F.embedding(x, self.weight)

Direct evidence

The released 671B configuration: width, layers, dense layers, heads, experts and routing, MLA ranks and head widths, indexer heads and top-k.

config · deepseek-ai/DeepSeek-V3.2-Exp / inference/config_671B_v3.2.json
Pinned revision 87e509a2e5a1 ↗ · lines 1–26

Read code excerpt

{
    "vocab_size": 129280,
    "dim": 7168,
    "inter_dim": 18432,
    "moe_inter_dim": 2048,
    "n_layers": 61,
    "n_dense_layers": 3,
    "n_heads": 128,
    "n_routed_experts": 256,
    "n_shared_experts": 1,
    "n_activated_experts": 8,
    "n_expert_groups": 8,
    "n_limited_groups": 4,
    "route_scale": 2.5,
    "score_func": "sigmoid",
    "q_lora_rank": 1536,
    "kv_lora_rank": 512,
    "qk_nope_head_dim": 128,
    "qk_rope_head_dim": 64,
    "v_head_dim": 128,
    "dtype": "fp8",
    "scale_fmt": "ue8m0",
    "index_n_heads": 64,
    "index_head_dim": 128,
    "index_topk": 2048
}