GLM-5

Z.ai · Feb 2026Latent attention and experts

Query heads

Checkpoint configuration and implementation zai-org/GLM-5 @ c183ef8, huggingface/transformers @ 5e4d630 · parameter count checked against the checkpoint’s tensor headers · How to read

Full sequence Prefill DecodeP1,000

0 127 1,000 4,096 202,751

c^Q[B, 1, 2048]R[1, 32]Query content up-projection

{\htmlClass{vl-stream-q}{q^{C}}}=\hat c^{{\htmlClass{vl-stream-q}{Q}}}W_{UQ}

q^C[B, 1, 12288]Split content heads Q^C[B, 64, 1, 192]Query rotary up-projection

{\htmlClass{vl-stream-q}{q^{\mathrm{pe}}}}=\hat c^{{\htmlClass{vl-stream-q}{Q}}}W_{QR}

q^pe[B, 1, 4096]Split rotary heads Q^pe[B, 64, 1, 64]Rotate query part

{\htmlClass{vl-stream-q}{Q^{R}}}=\operatorname{RoPE}({\htmlClass{vl-stream-q}{Q^{\mathrm{pe}}}})

Q^R[B, 64, 1, 64]Join query parts

{\htmlClass{vl-stream-q}{Q}}=[{\htmlClass{vl-stream-q}{Q^{C}}};\,{\htmlClass{vl-stream-q}{Q^{R}}}]

Q[B, 64, 1, 256]

78 transformer blocks › 75 MoE blocks › Sparse latent attention › Query heads

Split content heads

no parametersself.q_b_proj(q_resid).view(query_shape).transpose(1, 2)

Partition the content features into 64 heads.

[B,\,T,\,12288]\;\to\;[B,\,64,\,T,\,192]

in: $q^{C}$ [B batch, 1 positions, 12288 heads × content]
out: $Q^{C}$ [B batch, 64 heads, 1 positions, 192 content]

Codemodeling_glm_moe_dsa.py 394–396

modeling_glm_moe_dsa.py 394–396

        q_resid = self.q_a_layernorm(self.q_a_proj(hidden_states))
        q_states = self.q_b_proj(q_resid).view(query_shape).transpose(1, 2)
        q_pass, q_rot = torch.split(q_states, [self.qk_nope_head_dim, self.qk_rope_head_dim], dim=-1)

Sources1

Direct evidence

Queries are produced from the normalized query latent and split into a non-rotated and a rotated part.

mla-queries · huggingface/transformers / modeling_glm_moe_dsa.py
Pinned revision 5e4d6304de55 ↗ · lines 394–396

Read code excerpt

        q_resid = self.q_a_layernorm(self.q_a_proj(hidden_states))
        q_states = self.q_b_proj(q_resid).view(query_shape).transpose(1, 2)
        q_pass, q_rot = torch.split(q_states, [self.qk_nope_head_dim, self.qk_rope_head_dim], dim=-1)