DeepSeek-V3.2-Exp

All models, with previews →

DeepSeek · Sep 2025Latent attention and experts

Reference code deepseek-ai/DeepSeek-V3.2-Exp @ 87e509a · checkpoint, tokenizer and tensors not verified · How to read

Token IDs

×

tensortokens

Integer vocabulary indices used to look up token embeddings.

shape
II [B batch, T positions]
Sources

Contextual evidence from Tokenizer

The reference generator loads the checkpoint’s tokenizer with Hugging Face AutoTokenizer and encodes each prompt through its chat template. The tokenizer’s vocabulary and rules ship with the checkpoint; these sources do not include them.

tokenizer · deepseek-ai/DeepSeek-V3.2-Exp / inference/generate.py
Pinned revision 87e509a2e5a1 ↗ · lines 117–150

Read code excerpt
    tokenizer = AutoTokenizer.from_pretrained(ckpt_path)

… lines 118–149 omitted …

        prompt_tokens = [tokenizer.apply_chat_template([{"role": "user", "content": prompt}], add_generation_prompt=True) for prompt in prompts]

Contextual evidence from Tokenizer

The released 671B configuration: width, layers, dense layers, heads, experts and routing, MLA ranks and head widths, indexer heads and top-k.

config · deepseek-ai/DeepSeek-V3.2-Exp / inference/config_671B_v3.2.json
Pinned revision 87e509a2e5a1 ↗ · lines 1–26

Read code excerpt
{
    "vocab_size": 129280,
    "dim": 7168,
    "inter_dim": 18432,
    "moe_inter_dim": 2048,
    "n_layers": 61,
    "n_dense_layers": 3,
    "n_heads": 128,
    "n_routed_experts": 256,
    "n_shared_experts": 1,
    "n_activated_experts": 8,
    "n_expert_groups": 8,
    "n_limited_groups": 4,
    "route_scale": 2.5,
    "score_func": "sigmoid",
    "q_lora_rank": 1536,
    "kv_lora_rank": 512,
    "qk_nope_head_dim": 128,
    "qk_rope_head_dim": 64,
    "v_head_dim": 128,
    "dtype": "fp8",
    "scale_fmt": "ue8m0",
    "index_n_heads": 64,
    "index_head_dim": 128,
    "index_topk": 2048
}

Contextual evidence from Token embedding

The token embedding is a learned table of vocabulary × width rows, looked up by token ID.

embedding · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 108–127

Read code excerpt
        self.weight = nn.Parameter(torch.empty(self.part_vocab_size, self.dim))

… lines 109–126 omitted …

        y = F.embedding(x, self.weight)

Contextual evidence from Position IDs

The model embeds tokens, applies all blocks with a causal mask, normalizes and projects to vocabulary logits with an untied head. The reference generator projects only the last position; training projects every position.

transformer · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 880–908

Read code excerpt
        self.embed = ParallelEmbedding(args.vocab_size, args.dim)
        self.layers = torch.nn.ModuleList()
        for layer_id in range(args.n_layers):
            self.layers.append(Block(layer_id, args))
        self.norm = RMSNorm(args.dim)
        # lm_head in the checkpoint is stored in bf16, while the parameter here is stored in fp32 for easier computation of logits later.
        self.head = ColumnParallelLinear(args.dim, args.vocab_size, dtype=torch.float32)
        self.register_buffer("freqs_cis", precompute_freqs_cis(args), persistent=False)

… lines 888–900 omitted …

        seqlen = tokens.size(1)
        freqs_cis = self.freqs_cis[start_pos:start_pos+seqlen]
        mask = torch.full((seqlen, seqlen), float("-inf"), device=tokens.device).triu_(1) if seqlen > 1 else None
        h, residual = self.embed(tokens), None
        for layer in self.layers:
            h, residual = layer(h, residual, start_pos, freqs_cis, mask)
        h, _ = self.norm(h, residual)
        logits = self.head(h[:, -1].float())