Position IDs
×no parametersposition_ids
Identify the order and length of the current sequence.
- in
- [B batch, T positions]
- out
- [T positions]
Explanation
Positions count from the cached length (modeling_deepseek.py 1430–1438), 0 for the full-sequence call.
Codemodeling_deepseek.py 1360–1616
self.embed_tokens = nn.Embedding(
config.vocab_size, config.hidden_size, self.padding_idx
)
self.layers = nn.ModuleList(
[
DeepseekV3DecoderLayer(config, layer_idx)
for layer_idx in range(config.num_hidden_layers)
]
)
self._use_flash_attention_2 = config._attn_implementation == "flash_attention_2"
self.norm = DeepseekV3RMSNorm(config.hidden_size, eps=config.rms_norm_eps) past_key_values_length = 0
if use_cache:
use_legacy_cache = not isinstance(past_key_values, Cache)
if use_legacy_cache:
past_key_values = DynamicCache.from_legacy_cache(past_key_values)
past_key_values_length = past_key_values.get_usable_length(seq_length)
if position_ids is None:
device = input_ids.device if input_ids is not None else inputs_embeds.device
position_ids = torch.arange(
past_key_values_length,
seq_length + past_key_values_length,
dtype=torch.long,
device=device,
)
position_ids = position_ids.unsqueeze(0) if self._use_flash_attention_2:
# 2d mask is passed through the layers
attention_mask = (
attention_mask
if (attention_mask is not None and 0 in attention_mask)
else None
)
else:
# 4d mask is passed through the layers
attention_mask = _prepare_4d_causal_attention_mask(
attention_mask,
(batch_size, seq_length),
inputs_embeds,
past_key_values_length,
) self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False) hidden_states = outputs[0]
logits = self.lm_head(hidden_states)
logits = logits.float()Sources1
Direct evidence
The model embeds tokens, numbers positions from the cached length, builds a 4D causal mask, applies all blocks, normalizes and projects to vocabulary logits with an untied head, computing every position’s logits in float32.
transformer · moonshotai/Kimi-K2-Instruct / modeling_deepseek.py
Pinned revision fd1984e2b7a3 ↗ · lines 1360–1616
Read code excerpt
self.embed_tokens = nn.Embedding(
config.vocab_size, config.hidden_size, self.padding_idx
)
self.layers = nn.ModuleList(
[
DeepseekV3DecoderLayer(config, layer_idx)
for layer_idx in range(config.num_hidden_layers)
]
)
self._use_flash_attention_2 = config._attn_implementation == "flash_attention_2"
self.norm = DeepseekV3RMSNorm(config.hidden_size, eps=config.rms_norm_eps)… lines 1371–1422 omitted …
past_key_values_length = 0
if use_cache:
use_legacy_cache = not isinstance(past_key_values, Cache)
if use_legacy_cache:
past_key_values = DynamicCache.from_legacy_cache(past_key_values)
past_key_values_length = past_key_values.get_usable_length(seq_length)
if position_ids is None:
device = input_ids.device if input_ids is not None else inputs_embeds.device
position_ids = torch.arange(
past_key_values_length,
seq_length + past_key_values_length,
dtype=torch.long,
device=device,
)
position_ids = position_ids.unsqueeze(0)… lines 1439–1442 omitted …
if self._use_flash_attention_2:
# 2d mask is passed through the layers
attention_mask = (
attention_mask
if (attention_mask is not None and 0 in attention_mask)
else None
)
else:
# 4d mask is passed through the layers
attention_mask = _prepare_4d_causal_attention_mask(
attention_mask,
(batch_size, seq_length),
inputs_embeds,
past_key_values_length,
)… lines 1458–1521 omitted …
self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)… lines 1523–1613 omitted …
hidden_states = outputs[0]
logits = self.lm_head(hidden_states)
logits = logits.float()