DeepSeek-V3.2-Exp · Vocabulary projection

Vocabulary projection

learnedself.head(h[:, -1].float())

Project each normalized feature vector into vocabulary logits.

{\htmlData{light=logits projection}{L}}={\htmlData{light=final-norm}{z}}W_{\mathrm{out}}

in: $z$ [B batch, T positions, 7168 d]
out: $L$ [B batch, T positions, 129280 vocabulary]
parameters: 926,679,040

Held in float32 and applied to float32 input; the linear weights in the blocks are FP8 (model.py 876, 885–886). The reference projects only the last position (908).

Parameters

926,679,040 parameters per instance.

Exact per-instance accounting
Component	Calculation	Parameters
Weight matrix	$7{,}168 \times 129{,}280$	926,679,040
Per-instance total		926,679,040

The reference module holds the weight output by input, [out, in].

Derived from the configured matrix dimensions; checkpoint tensors have not been independently counted.

Explanation

Each entry scores one vocabulary token as the next token for its position. The softmax and the choice of token happen outside the model.

Codemodel.py 880–887, 901–908 · config_671B_v3.2.json 1–26

model.py 880–887

        self.embed = ParallelEmbedding(args.vocab_size, args.dim)
        self.layers = torch.nn.ModuleList()
        for layer_id in range(args.n_layers):
            self.layers.append(Block(layer_id, args))
        self.norm = RMSNorm(args.dim)
        # lm_head in the checkpoint is stored in bf16, while the parameter here is stored in fp32 for easier computation of logits later.
        self.head = ColumnParallelLinear(args.dim, args.vocab_size, dtype=torch.float32)
        self.register_buffer("freqs_cis", precompute_freqs_cis(args), persistent=False)

model.py 901–908

        seqlen = tokens.size(1)
        freqs_cis = self.freqs_cis[start_pos:start_pos+seqlen]
        mask = torch.full((seqlen, seqlen), float("-inf"), device=tokens.device).triu_(1) if seqlen > 1 else None
        h, residual = self.embed(tokens), None
        for layer in self.layers:
            h, residual = layer(h, residual, start_pos, freqs_cis, mask)
        h, _ = self.norm(h, residual)
        logits = self.head(h[:, -1].float())

config_671B_v3.2.json 1–26

{
    "vocab_size": 129280,
    "dim": 7168,
    "inter_dim": 18432,
    "moe_inter_dim": 2048,
    "n_layers": 61,
    "n_dense_layers": 3,
    "n_heads": 128,
    "n_routed_experts": 256,
    "n_shared_experts": 1,
    "n_activated_experts": 8,
    "n_expert_groups": 8,
    "n_limited_groups": 4,
    "route_scale": 2.5,
    "score_func": "sigmoid",
    "q_lora_rank": 1536,
    "kv_lora_rank": 512,
    "qk_nope_head_dim": 128,
    "qk_rope_head_dim": 64,
    "v_head_dim": 128,
    "dtype": "fp8",
    "scale_fmt": "ue8m0",
    "index_n_heads": 64,
    "index_head_dim": 128,
    "index_topk": 2048
}

Sources2

Direct evidence

The model embeds tokens, applies all blocks with a causal mask, normalizes and projects to vocabulary logits with an untied head. The reference generator projects only the last position; training projects every position.

transformer · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 880–908

Read code excerpt

        self.embed = ParallelEmbedding(args.vocab_size, args.dim)
        self.layers = torch.nn.ModuleList()
        for layer_id in range(args.n_layers):
            self.layers.append(Block(layer_id, args))
        self.norm = RMSNorm(args.dim)
        # lm_head in the checkpoint is stored in bf16, while the parameter here is stored in fp32 for easier computation of logits later.
        self.head = ColumnParallelLinear(args.dim, args.vocab_size, dtype=torch.float32)
        self.register_buffer("freqs_cis", precompute_freqs_cis(args), persistent=False)

… lines 888–900 omitted …

        seqlen = tokens.size(1)
        freqs_cis = self.freqs_cis[start_pos:start_pos+seqlen]
        mask = torch.full((seqlen, seqlen), float("-inf"), device=tokens.device).triu_(1) if seqlen > 1 else None
        h, residual = self.embed(tokens), None
        for layer in self.layers:
            h, residual = layer(h, residual, start_pos, freqs_cis, mask)
        h, _ = self.norm(h, residual)
        logits = self.head(h[:, -1].float())

Direct evidence

The released 671B configuration: width, layers, dense layers, heads, experts and routing, MLA ranks and head widths, indexer heads and top-k.

config · deepseek-ai/DeepSeek-V3.2-Exp / inference/config_671B_v3.2.json
Pinned revision 87e509a2e5a1 ↗ · lines 1–26

Read code excerpt

{
    "vocab_size": 129280,
    "dim": 7168,
    "inter_dim": 18432,
    "moe_inter_dim": 2048,
    "n_layers": 61,
    "n_dense_layers": 3,
    "n_heads": 128,
    "n_routed_experts": 256,
    "n_shared_experts": 1,
    "n_activated_experts": 8,
    "n_expert_groups": 8,
    "n_limited_groups": 4,
    "route_scale": 2.5,
    "score_func": "sigmoid",
    "q_lora_rank": 1536,
    "kv_lora_rank": 512,
    "qk_nope_head_dim": 128,
    "qk_rope_head_dim": 64,
    "v_head_dim": 128,
    "dtype": "fp8",
    "scale_fmt": "ue8m0",
    "index_n_heads": 64,
    "index_head_dim": 128,
    "index_topk": 2048
}