GLM-5

All models, with previews →

Z.ai · Feb 2026Latent attention and experts

Shared expert

Checkpoint configuration and implementation zai-org/GLM-5 @ c183ef8, huggingface/transformers @ 5e4d630 · parameter count checked against the checkpoint’s tensor headers · How to read

78 transformer blocks › 75 MoE blocks › Mixture of experts › Shared expert

Down projection

×

learnedself.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))

Map 2,048 gated features back to 6,144 features.

yshared=hW2{\htmlData{light=output down}{y^{\mathrm{shared}}}}={\htmlData{light=h product}{h}}W_{2}
in
hh [B batch, T positions, 2048 shared d_ff]
out
ysharedy^{\mathrm{shared}} [B batch, T positions, 6144 d]
parameters
12,582,912 · × 75 MoE blocks = 943,718,400
Parameters

12,582,912 parameters per instance.

Exact per-instance accounting
ComponentCalculationParameters
Weight matrix2,048×6,1442{,}048 \times 6{,}14412,582,912
Per-instance total12,582,912

This projection has no bias parameter.

The reference module holds the weight output by input, [out, in].

Across 75 independent MoE blocks: 12,582,912×75=943,718,40012{,}582{,}912 \times 75 = 943{,}718{,}400 parameters.

Derived from the configured matrix dimensions; the model total matches the checkpoint’s tensors.

Codemodeling_glm_moe_dsa.py 465–478 · config.json 5–7
modeling_glm_moe_dsa.py 465–478
class GlmMoeDsaMLP(nn.Module):
    def __init__(self, config, intermediate_size=None):
        super().__init__()
        self.config = config
        self.hidden_size = config.hidden_size
        self.intermediate_size = config.intermediate_size if intermediate_size is None else intermediate_size
        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=False)
        self.act_fn = ACT2FN[config.hidden_act]

    def forward(self, x):
        down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
        return down_proj
config.json 5–7
  "attention_bias": false,
  "attention_dropout": 0.0,
  "dtype": "bfloat16",
Sources2

Direct evidence

The dense feed-forward network is a SwiGLU MLP, down(silu(gate x) ⊙ up x), computed in the input’s precision.

mlp · huggingface/transformers / modeling_glm_moe_dsa.py
Pinned revision 5e4d6304de55 ↗ · lines 465–478

Read code excerpt
class GlmMoeDsaMLP(nn.Module):
    def __init__(self, config, intermediate_size=None):
        super().__init__()
        self.config = config
        self.hidden_size = config.hidden_size
        self.intermediate_size = config.intermediate_size if intermediate_size is None else intermediate_size
        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=False)
        self.act_fn = ACT2FN[config.hidden_act]

    def forward(self, x):
        down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
        return down_proj

Direct evidence

Attention projections have no bias (attention_bias is false) and the checkpoint stores its weights in bfloat16.

linear · zai-org/GLM-5 / config.json
Pinned revision c183ef8c61fa ↗ · lines 5–7

Read code excerpt
  "attention_bias": false,
  "attention_dropout": 0.0,
  "dtype": "bfloat16",