SFT
×Stage · Kimi K3
Supervised fine-tuning on synthesized agentic trajectories gives a cold-start policy for RL; quantization-aware training starts here.
- Purpose
- A cold-start policy for the RL stage
- Data
- synthesized agentic trajectories, verified and annotated
- Number format
- MXFP4 expert weights, MXFP8 activations
Not disclosed Data size · Epochs · Learning rate · Optimizer
Why
SFT gives the RL stage a high-quality policy to start from. Kimi K3 report §4.1.1, p. 12
Quantization-aware training covers SFT and RL so that the model adapts to the precision loss of quantization. Kimi K3 report §4.1.4, p. 14
Architecture at this stageFrom here on the MoE experts’ weights are trained in MXFP4, with MXFP8 activations.
All settings4
- Purpose
- A cold-start policy for the RL stage
- Order
- After the cooldownderivedNo sentence of the report orders the cooldown and SFT: the cooldown is read as the end of pre-training, since it is described under Pre-Training.
- Data
- Trajectories synthesized by domain-specialized models of the earlier Kimi series, then multi-stage verification and human annotation; serialized with the XTML chat template
- Number format
- Quantization-aware training: MoE expert weights in MXFP4, activations in MXFP8; the other parts in higher precision
Sources2
- k3_tech_report.pdf, p. 12Purpose, Order, Data, Number format
- k3_tech_report.pdf, p. 14Number format