61 transformer blocks › 60 MoE blocks › Latent attention
Causal mask
×supplied · no parametersattention_mask
Current and earlier key positions for every query.
- shape
- [1 queries, 1001 keys]
The mask is still added (modeling_deepseek.py 825–831). A causal mask for the one new query, built by Transformers from the call at 1448–1457 and not pinned here, permits every cached position when nothing is padded, so it changes nothing.
Codemodeling_deepseek.py 825–831
assert attention_mask is not None
if attention_mask is not None:
if attention_mask.size() != (bsz, 1, q_len, kv_seq_len):
raise ValueError(
f"Attention mask should be of size {(bsz, 1, q_len, kv_seq_len)}, but is {attention_mask.size()}"
)
attn_weights = attn_weights + attention_maskSources1
Contextual evidence
The eager path requires a mask of shape (batch, 1, query length, cached length) and adds it to the scores.
attention-mask · moonshotai/Kimi-K2-Instruct / modeling_deepseek.py
Pinned revision fd1984e2b7a3 ↗ · lines 825–831
Read code excerpt
assert attention_mask is not None
if attention_mask is not None:
if attention_mask.size() != (bsz, 1, q_len, kv_seq_len):
raise ValueError(
f"Attention mask should be of size {(bsz, 1, q_len, kv_seq_len)}, but is {attention_mask.size()}"
)
attn_weights = attn_weights + attention_mask