DeepSeek-V3.2-Exp

All models, with previews →

DeepSeek · Sep 2025Latent attention and experts

Reference code deepseek-ai/DeepSeek-V3.2-Exp @ 87e509a · checkpoint, tokenizer and tensors not verified · How to read

in
Raw text
out
LL [B, T, 129280]
parameters
671,877,944,064 in all
Parameters

671,877,944,064 parameters in DeepSeek-V3.2-Exp

Exact model ownership reconciliation
ComponentCalculationParameters
Token embedding129,280×7,168129{,}280 \times 7{,}168926,679,040
Transformer blocks670,024,578,816 across its components670,024,578,816
Final RMSNorm7,1687{,}1687,168
Vocabulary projection7,168×129,2807{,}168 \times 129{,}280926,679,040
Model total671,877,944,064

Derived exact parameter total: 671,877,944,064. Sources contains the pinned configuration and implementation used for this derivation; checkpoint tensors are not independently counted.

Derived from the pinned reference configuration and inference code. Checkpoint tensors have not been independently counted; FP8 block scales and the multi-token-prediction module are not counted.

In numbers

Fact sheet

Blocks
61: the first 3 dense, then 58 MoEThe reference builds no multi-token-prediction module, and the atlas draws none.config_671B_v3.2.json 6–7model.py 880–887
Attention
128 heads; keys 128 + 64 rotary features, values 128; latents: query 1,536, key–value 512Lightning indexer: 64 heads of 128 features, keeping 2,048 positions per query.config_671B_v3.2.json 8config_671B_v3.2.json 16–20config_671B_v3.2.json 23–25
Feed-forward
256 routed + 1 shared experts; 8 routed chosen per token; 2,048 hidden features eachThe first 3 blocks have a dense SwiGLU of 18,432 hidden features instead.config_671B_v3.2.json 4–5config_671B_v3.2.json 9–11
Vocabulary
129,280config_671B_v3.2.json 2
Allocated length
16,384 positionsThe reference’s max_seq_len: it sizes the caches and the rotation table, and generation stops there, prompt included. The release configuration leaves this default in place.model.py 56model.py 335model.py 541–542model.py 453–454generate.py 52–53
Parameters
671.9 B in all · 38.4 B used per token (5.7%)Both counts include every block’s indexer, and the embedding table and output head whole; without those two tables, 36.6 B per token.config_671B_v3.2.json 9–11model.py 790–803
KV cache
76.5 KiB per position, all layersThe 512-feature latent and 64-feature rotary key in bfloat16, and the 128-feature indexer key in FP8 with its float32 scale. Written in Prefill and Decode; the Full sequence view draws none.model.py 541–542generate.py 109model.py 572–573model.py 453–454

Training

How DeepSeek-V3.2-Exp was made, stage by stage: from DeepSeek-V3’s pre-training through V3.1 to the sparse stages

Weights · whole model

DeepSeek-V3.2-Exp · FP8629.4 GiB
Tied readout · FP8625.9 GiB

671,877,944,064 parameters: FP8 matrices, block scales left out; the embedding table and the router’s matrices in bfloat16; the output head, the norms, the router’s bias and the indexer’s head weights in float32. The separate output head holds 926,679,040 of them (0.14%), 3.5 GiB in float32; tied to the embedding table, the weights would take 625.9 GiB.

config_671B_v3.2.json 2–3config_671B_v3.2.json 21model.py 876model.py 183–187model.py 108generate.py 109model.py 885–886model.py 284model.py 317–318model.py 674–675model.py 449model.py 880
Sources

Direct evidence

The released 671B configuration: width, layers, dense layers, heads, experts and routing, MLA ranks and head widths, indexer heads and top-k.

config · deepseek-ai/DeepSeek-V3.2-Exp / inference/config_671B_v3.2.json
Pinned revision 87e509a2e5a1 ↗ · lines 1–26

Read code excerpt
{
    "vocab_size": 129280,
    "dim": 7168,
    "inter_dim": 18432,
    "moe_inter_dim": 2048,
    "n_layers": 61,
    "n_dense_layers": 3,
    "n_heads": 128,
    "n_routed_experts": 256,
    "n_shared_experts": 1,
    "n_activated_experts": 8,
    "n_expert_groups": 8,
    "n_limited_groups": 4,
    "route_scale": 2.5,
    "score_func": "sigmoid",
    "q_lora_rank": 1536,
    "kv_lora_rank": 512,
    "qk_nope_head_dim": 128,
    "qk_rope_head_dim": 64,
    "v_head_dim": 128,
    "dtype": "fp8",
    "scale_fmt": "ue8m0",
    "index_n_heads": 64,
    "index_head_dim": 128,
    "index_topk": 2048
}

Direct evidence

Reference defaults the configuration does not override: maximum sequence length, YaRN RoPE settings and the softmax scale correction.

model-args · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 56–86

Read code excerpt
    max_seq_len: int = 4096 * 4

… lines 57–80 omitted …

    original_seq_len: int = 4096
    rope_theta: float = 10000.0
    rope_factor: float = 40
    beta_fast: int = 32
    beta_slow: int = 1
    mscale: float = 1.

Direct evidence

The model embeds tokens, applies all blocks with a causal mask, normalizes and projects to vocabulary logits with an untied head. The reference generator projects only the last position; training projects every position.

transformer · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 880–908

Read code excerpt
        self.embed = ParallelEmbedding(args.vocab_size, args.dim)
        self.layers = torch.nn.ModuleList()
        for layer_id in range(args.n_layers):
            self.layers.append(Block(layer_id, args))
        self.norm = RMSNorm(args.dim)
        # lm_head in the checkpoint is stored in bf16, while the parameter here is stored in fp32 for easier computation of logits later.
        self.head = ColumnParallelLinear(args.dim, args.vocab_size, dtype=torch.float32)
        self.register_buffer("freqs_cis", precompute_freqs_cis(args), persistent=False)

… lines 888–900 omitted …

        seqlen = tokens.size(1)
        freqs_cis = self.freqs_cis[start_pos:start_pos+seqlen]
        mask = torch.full((seqlen, seqlen), float("-inf"), device=tokens.device).triu_(1) if seqlen > 1 else None
        h, residual = self.embed(tokens), None
        for layer in self.layers:
            h, residual = layer(h, residual, start_pos, freqs_cis, mask)
        h, _ = self.norm(h, residual)
        logits = self.head(h[:, -1].float())