Kimi K3

All models, with their blocks →

Size
2.8T
Published
Moonshot AI · Release date not pinned
Line
New parts in the block · after Kimi K2
View
Source

moonshotai/Kimi‑K3 @ f831ab6, fla‑org/flash‑linear‑attention @ 07ca1e4 · parameter count checked against the checkpoint’s tensor headers · text model onlyHow to read

Gated MLA

Full sequencePrefillDecode

93 layers › 24 MLA layers › Gated MLA

Merge heads

×

no parametersattn_output.reshape(batch_size, seq_length, -1) · 468–469

Concatenate the 96 heads into 12,288 features.

[B, 96, T, 128][B,\,96,\,T,\,128]
→  [B, T, 96, 128]\to\;[B,\,T,\,96,\,128]
→  [B, T, 12288]\to\;[B,\,T,\,12288]
in
O~\tilde O [B batch, 96 heads, T positions, 128 value d_h]
out
o~\tilde o [B batch, T positions, 12288 heads × value d_h]
Explanation

Heads are concatenated in order: head ii fills features 128i,…,128i+127{128i},\dots,{128i+127}.

Codemodeling_kimi_linear.py 468–474
modeling_kimi_linear.py 468–474
        attn_output = attn_output.reshape(
            batch_size, seq_length, -1).contiguous()
        if self.use_output_gate:
            g = self.g_proj(hidden_states).sigmoid()
            attn_output = attn_output * g
        attn_output = self.o_proj(attn_output)
        return attn_output
Sources1

Direct evidence

Head outputs are flattened, multiplied by the sigmoid of a full-rank projection of the layer’s input, and projected back to the model width.

mla-output · moonshotai/Kimi-K3 / modeling_kimi_linear.py
Pinned revision f831ab668142 ↗ · lines 468–474

Read code excerpt
        attn_output = attn_output.reshape(
            batch_size, seq_length, -1).contiguous()
        if self.use_output_gate:
            g = self.g_proj(hidden_states).sigmoid()
            attn_output = attn_output * g
        attn_output = self.o_proj(attn_output)
        return attn_output