GLM-5

All models, with their blocks →

Size
744B
Published
Z.ai · Feb 2026
Line
Latent attention and experts
View
Architecture
Source

zai‑org/GLM‑5 @ c183ef8, huggingface/transformers @ 5e4d630 · parameter count checked against the checkpoint’s tensor headersHow to read

Mix and project

Full sequencePrefillDecode

78 transformer blocks › 75 MoE blocks › Sparse latent attention › Sparse attention › Mix and project

Merge heads

×

no parametersattn_output.reshape(batch_size, seq_length, -1)

Concatenate the heads at each position.

[B, 64, T, 256][B,\,64,\,T,\,256]
→  [B, T, 64, 256]\to\;[B,\,T,\,64,\,256]
→  [B, T, 16384]\to\;[B,\,T,\,16384]
in
OO [B batch, 64 heads, 1 position, 256 value]
out
oo [B batch, 1 position, 16384 heads × value]
Explanation

Heads are concatenated in order: head ii fills features 256i,…,256i+255{256i},\dots,{256i+255}.

Codemodeling_glm_moe_dsa.py 460–462
modeling_glm_moe_dsa.py 460–462
        attn_output = attn_output.reshape(batch_size, seq_length, -1).contiguous()
        attn_output = self.o_proj(attn_output)
        return attn_output, attn_weights, topk_indices
Sources1

Direct evidence

Head outputs are flattened and projected back to the model width.

mla-output · huggingface/transformers / modeling_glm_moe_dsa.py
Pinned revision 5e4d6304de55 ↗ · lines 460–462

Read code excerpt
        attn_output = attn_output.reshape(batch_size, seq_length, -1).contiguous()
        attn_output = self.o_proj(attn_output)
        return attn_output, attn_weights, topk_indices