GLM-5

Z.ai · Feb 2026Latent attention and experts

Query heads

Checkpoint configuration and implementation zai-org/GLM-5 @ c183ef8, huggingface/transformers @ 5e4d630 · parameter count checked against the checkpoint’s tensor headers · How to read

Full sequence Prefill Decode

c^Q[B, T, 2048]R[T, 32]Query content up-projection

{\htmlClass{vl-stream-q}{q^{C}}}=\hat c^{{\htmlClass{vl-stream-q}{Q}}}W_{UQ}

q^C[B, T, 12288]Split content heads Q^C[B, 64, T, 192]Query rotary up-projection

{\htmlClass{vl-stream-q}{q^{\mathrm{pe}}}}=\hat c^{{\htmlClass{vl-stream-q}{Q}}}W_{QR}

q^pe[B, T, 4096]Split rotary heads Q^pe[B, 64, T, 64]Rotate query part

{\htmlClass{vl-stream-q}{Q^{R}}}=\operatorname{RoPE}({\htmlClass{vl-stream-q}{Q^{\mathrm{pe}}}})

Q^R[B, 64, T, 64]Join query parts

{\htmlClass{vl-stream-q}{Q}}=[{\htmlClass{vl-stream-q}{Q^{C}}};\,{\htmlClass{vl-stream-q}{Q^{R}}}]

Q[B, 64, T, 256]

78 transformer blocks › 75 MoE blocks › Sparse latent attention › Query heads

Split rotary heads

no parametersself.q_b_proj(q_resid).view(query_shape).transpose(1, 2)

Partition the rotary features into 64 heads.

[B,\,T,\,4096]\;\to\;[B,\,64,\,T,\,64]

in: $q^{\mathrm{pe}}$ [B batch, T positions, 4096 heads × rotary]
out: $Q^{\mathrm{pe}}$ [B batch, 64 heads, T positions, 64 rotary]

Codemodeling_glm_moe_dsa.py 394–396

modeling_glm_moe_dsa.py 394–396

        q_resid = self.q_a_layernorm(self.q_a_proj(hidden_states))
        q_states = self.q_b_proj(q_resid).view(query_shape).transpose(1, 2)
        q_pass, q_rot = torch.split(q_states, [self.qk_nope_head_dim, self.qk_rope_head_dim], dim=-1)

Sources1

Direct evidence

Queries are produced from the normalized query latent and split into a non-rotated and a rotated part.

mla-queries · huggingface/transformers / modeling_glm_moe_dsa.py
Pinned revision 5e4d6304de55 ↗ · lines 394–396

Read code excerpt

        q_resid = self.q_a_layernorm(self.q_a_proj(hidden_states))
        q_states = self.q_b_proj(q_resid).view(query_shape).transpose(1, 2)
        q_pass, q_rot = torch.split(q_states, [self.qk_nope_head_dim, self.qk_rope_head_dim], dim=-1)