Kimi K3

Size

2.8T

Published

Moonshot AI · Release date not pinned

Line

New parts in the block · after Kimi K2

View

Source

Kimi K3 technical report @ 3cb39df · Kimi K2.5 report (arXiv 2602.02276v2), for the RL step · no training code is publishedHow to read

MOPD reward

x Queryx x e[B]Student writes

y\sim\pi_\theta(\cdot\mid e,x)

y[B, T]Teacher (d, e)

p_T=\pi^{(d,e)}_{\mathrm{teacher}}(y_t\mid x,y_{<t})

p_T[B, T]Student

p_S=\pi_\theta(y_t\mid e,x,y_{<t})

p_S[B, T]Clipped log-ratio

r_t=\operatorname{clip}\big(\operatorname{sg}(\log(p_T/p_S)),\pm {\htmlClass{vl-stream-context}{R_{\max}}}\big)

r[B, T]hatched band: not updated in this step · solid band: what this stage trains · dashed frame: concluded, not stated by the reportx: one query, named beside each model that reads it · the update that uses r is outside this plate

MOPD reward

Teacher (d, e)

frozen · not trained

The expert of the query’s domain and of the sampled effort, one of nine, gives each token of the response its probability.

{\htmlData{light=teacher-prob teacher}{p_T}}=\pi^{(d,e)}_{\mathrm{teacher}}({\htmlData{light=response rollout}{y_t}}\mid {\htmlData{light=query}{x}},y_{<t})

in: $x$ [B batch] $y$ [B batch, T positions]
out: $p_T$ [B batch, T positions]

Drawn as not updated: the teacher is one of the nine experts of the RL stage, and the report names only $\pi_\theta$ as the student. That the teachers stay fixed is read from that; the report does not state it. $p_T$ is this plate’s name.

Sources2

Direct evidence

§4.1.3, Multi-Teacher On-Policy Distillation: for a domain d and a sampled reasoning effort e ∈ {low, high, max}, optimization is guided by the teacher of that pair among the nine experts. The per-token reward on y_t is the stop-gradient log-ratio of the teacher’s probability to the student’s, clipped to ±R_max (eq. 15). The dense reward integrates into the RL framework, partial rollouts included.

mopd · MoonshotAI/Kimi-K3 / k3_tech_report.pdf
Pinned revision 3cb39dfd32e5 ↗ · pp. 13–14

Direct evidence

§4.1.2: three domains crossed with three reasoning efforts in {low, high, max} yield nine expert models.

experts · MoonshotAI/Kimi-K3 / k3_tech_report.pdf
Pinned revision 3cb39dfd32e5 ↗ · p. 12