DeepSeek-V3.2-Exp · Construct causal mask

Construct causal mask

no parameterstorch.full((seqlen, seqlen), float("-inf"), device=tokens.device).triu_(1)

Identify the key positions permitted for every query position.

{\htmlData{light=causal-mask model-mask}{{\htmlClass{vl-stream-context}{M_{t,s}}}}}=\begin{cases}0&s\le t\\-\infty&s>t\end{cases}

in: $p$ [T positions]
out: $M$ [T queries, T keys]

Built only when T > 1; a single decoded token gets no mask (model.py 903).

Explanation

Row $t$ is the query position, column $s$ the key position.

Codemodel.py 880–887, 901–908

model.py 880–887

        self.embed = ParallelEmbedding(args.vocab_size, args.dim)
        self.layers = torch.nn.ModuleList()
        for layer_id in range(args.n_layers):
            self.layers.append(Block(layer_id, args))
        self.norm = RMSNorm(args.dim)
        # lm_head in the checkpoint is stored in bf16, while the parameter here is stored in fp32 for easier computation of logits later.
        self.head = ColumnParallelLinear(args.dim, args.vocab_size, dtype=torch.float32)
        self.register_buffer("freqs_cis", precompute_freqs_cis(args), persistent=False)

model.py 901–908

        seqlen = tokens.size(1)
        freqs_cis = self.freqs_cis[start_pos:start_pos+seqlen]
        mask = torch.full((seqlen, seqlen), float("-inf"), device=tokens.device).triu_(1) if seqlen > 1 else None
        h, residual = self.embed(tokens), None
        for layer in self.layers:
            h, residual = layer(h, residual, start_pos, freqs_cis, mask)
        h, _ = self.norm(h, residual)
        logits = self.head(h[:, -1].float())

Sources1

Direct evidence

The model embeds tokens, applies all blocks with a causal mask, normalizes and projects to vocabulary logits with an untied head. The reference generator projects only the last position; training projects every position.

transformer · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 880–908

Read code excerpt

        self.embed = ParallelEmbedding(args.vocab_size, args.dim)
        self.layers = torch.nn.ModuleList()
        for layer_id in range(args.n_layers):
            self.layers.append(Block(layer_id, args))
        self.norm = RMSNorm(args.dim)
        # lm_head in the checkpoint is stored in bf16, while the parameter here is stored in fp32 for easier computation of logits later.
        self.head = ColumnParallelLinear(args.dim, args.vocab_size, dtype=torch.float32)
        self.register_buffer("freqs_cis", precompute_freqs_cis(args), persistent=False)

… lines 888–900 omitted …

        seqlen = tokens.size(1)
        freqs_cis = self.freqs_cis[start_pos:start_pos+seqlen]
        mask = torch.full((seqlen, seqlen), float("-inf"), device=tokens.device).triu_(1) if seqlen > 1 else None
        h, residual = self.embed(tokens), None
        for layer in self.layers:
            h, residual = layer(h, residual, start_pos, freqs_cis, mask)
        h, _ = self.norm(h, residual)
        logits = self.head(h[:, -1].float())