GLM-5

Z.ai · Feb 2026Latent attention and experts

Lightning indexer

Checkpoint configuration and implementation zai-org/GLM-5 @ c183ef8, huggingface/transformers @ 5e4d630 · parameter count checked against the checkpoint’s tensor headers · How to read

Full sequence Prefill Decode

78 transformer blocks › 75 MoE blocks › Sparse latent attention › Lightning indexer

Index key cache

state · kept between callspast_key_values.update_indexer(k, self.layer_idx)

The indexer key of every position so far, kept between forward calls.

shape: $\tilde k^{I}$ k_I [B batch, T cached positions, 128 index d_h]

Prefill hands the prompt’s keys to the cache and scores against them. 128 features per position.

Codemodeling_glm_moe_dsa.py 236–239

modeling_glm_moe_dsa.py 236–239

        if past_key_values is not None:
            k = past_key_values.update_indexer(k, self.layer_idx)

        scores = torch.matmul(q.float(), k.transpose(-1, -2).float().unsqueeze(1)) * self.softmax_scale

Sources1

Contextual evidence

The indexer hands its key to the cache and scores its queries against every cached key so far.

indexer-cache · huggingface/transformers / modeling_glm_moe_dsa.py
Pinned revision 5e4d6304de55 ↗ · lines 236–239

Read code excerpt

        if past_key_values is not None:
            k = past_key_values.update_indexer(k, self.layer_idx)

        scores = torch.matmul(q.float(), k.transpose(-1, -2).float().unsqueeze(1)) * self.softmax_scale