Kimi K3

Size

2.8T

Published

Moonshot AI · Release date not pinned

Line

New parts in the block · after Kimi K2

View

Source

Kimi K3 technical report @ 3cb39df · Kimi K2.5 report (arXiv 2602.02276v2), for the RL step · no training code is publishedHow to read

MOPD reward

x Queryx x e[B]Student writes

y\sim\pi_\theta(\cdot\mid e,x)

y[B, T]Teacher (d, e)

p_T=\pi^{(d,e)}_{\mathrm{teacher}}(y_t\mid x,y_{<t})

p_T[B, T]Student

p_S=\pi_\theta(y_t\mid e,x,y_{<t})

p_S[B, T]Clipped log-ratio

r_t=\operatorname{clip}\big(\operatorname{sg}(\log(p_T/p_S)),\pm {\htmlClass{vl-stream-context}{R_{\max}}}\big)

r[B, T]hatched band: not updated in this step · solid band: what this stage trains · dashed frame: concluded, not stated by the reportx: one query, named beside each model that reads it · the update that uses r is outside this plate

MOPD reward

Response

tensor

The tokens $y_1,\dots,y_T$ the reward is computed on, each after its prefix $y_{<t}$ .

shape: $y$ [B batch, T positions]

Sources1

Contextual evidence

§4.1.3, Multi-Teacher On-Policy Distillation: for a domain d and a sampled reasoning effort e ∈ {low, high, max}, optimization is guided by the teacher of that pair among the nine experts. The per-token reward on y_t is the stop-gradient log-ratio of the teacher’s probability to the student’s, clipped to ±R_max (eq. 15). The dense reward integrates into the RL framework, partial rollouts included.

mopd · MoonshotAI/Kimi-K3 / k3_tech_report.pdf
Pinned revision 3cb39dfd32e5 ↗ · pp. 13–14