GLM-5

All models, with previews →

Z.ai · Feb 2026Latent attention and experts

Shared expert

Checkpoint configuration and implementation zai-org/GLM-5 @ c183ef8, huggingface/transformers @ 5e4d630 · parameter count checked against the checkpoint’s tensor headers · How to read

78 transformer blocks › 75 MoE blocks › Mixture of experts › Shared expert

Gate projection

×

learnedself.gate_proj(x)

Map 6,144 input features to 2,048 gate features.

g=xffnW1{\htmlData{light=g gate}{g}}={\htmlData{light=x}{x_{\mathrm{ffn}}}}W_{1}
in
xffnx_{\mathrm{ffn}} [B batch, T positions, 6144 d]
out
gg [B batch, T positions, 2048 shared d_ff]
parameters
12,582,912 · × 75 MoE blocks = 943,718,400
Parameters

12,582,912 parameters per instance.

Exact per-instance accounting
ComponentCalculationParameters
Weight matrix6,144×2,0486{,}144 \times 2{,}04812,582,912
Per-instance total12,582,912

This projection has no bias parameter.

The reference module holds the weight output by input, [out, in].

Across 75 independent MoE blocks: 12,582,912×75=943,718,40012{,}582{,}912 \times 75 = 943{,}718{,}400 parameters.

Derived from the configured matrix dimensions; the model total matches the checkpoint’s tensors.

Codemodeling_glm_moe_dsa.py 465–478 · config.json 5–7
modeling_glm_moe_dsa.py 465–478
class GlmMoeDsaMLP(nn.Module):
    def __init__(self, config, intermediate_size=None):
        super().__init__()
        self.config = config
        self.hidden_size = config.hidden_size
        self.intermediate_size = config.intermediate_size if intermediate_size is None else intermediate_size
        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=False)
        self.act_fn = ACT2FN[config.hidden_act]

    def forward(self, x):
        down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
        return down_proj
config.json 5–7
  "attention_bias": false,
  "attention_dropout": 0.0,
  "dtype": "bfloat16",
Sources2

Direct evidence

The dense feed-forward network is a SwiGLU MLP, down(silu(gate x) ⊙ up x), computed in the input’s precision.

mlp · huggingface/transformers / modeling_glm_moe_dsa.py
Pinned revision 5e4d6304de55 ↗ · lines 465–478

Read code excerpt
class GlmMoeDsaMLP(nn.Module):
    def __init__(self, config, intermediate_size=None):
        super().__init__()
        self.config = config
        self.hidden_size = config.hidden_size
        self.intermediate_size = config.intermediate_size if intermediate_size is None else intermediate_size
        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=False)
        self.act_fn = ACT2FN[config.hidden_act]

    def forward(self, x):
        down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
        return down_proj

Direct evidence

Attention projections have no bias (attention_bias is false) and the checkpoint stores its weights in bfloat16.

linear · zai-org/GLM-5 / config.json
Pinned revision c183ef8c61fa ↗ · lines 5–7

Read code excerpt
  "attention_bias": false,
  "attention_dropout": 0.0,
  "dtype": "bfloat16",