- out
- [1]
Explanation
The draft layer is the multi-token-prediction layer trained in pre-training, which mirrors the structure of a backbone block. The draft model of EAGLE-3 is a single decoder layer, and the report says its structure matches the MTP layer’s.
In training the draft is unrolled for seven steps. At the first it reads the target’s features; after it, the target’s features of the newest position do not exist yet, and the draft reads its own earlier outputs, as it will at inference.
The data, the number of steps and the learning rate are not disclosed, nor which checkpoint the frozen target is. The report also does not say how the draft layer’s output becomes a distribution over the vocabulary, which of the target’s distributions the draft’s is compared with at the steps after the first, or how the seven steps’ losses are combined.
The step runs in the post-training’s quantization: MoE expert weights in MXFP4 and their input activations in MXFP8, the other modules in higher precision. Speculative decoding, which the draft model serves, is used in RL rollouts.
Sources3
Direct evidence
§4.1.4, Draft Model Fine-Tuning: the pre-trained MTP layer is fine-tuned into an EAGLE-3-style draft model, with the target model frozen and only the draft layer and its feature-fusion projection updated; the draft is unrolled for seven steps and, beyond the first, consumes its own outputs. The input concatenates the outputs of the 1st, 4th and final AttnRes blocks and projects them by a bias-free matrix W_E3 initialized [0 0 I]. The loss is L_LK = −log Σ_x min(p(x), q(x)) (eq. 16), at temperature 1, with no cross-entropy term.
draft · MoonshotAI/Kimi-K3 / k3_tech_report.pdf
Pinned revision 3cb39dfd32e5 ↗ · p. 14
Direct evidence
Table 1: hidden dimension 7,168; vocabulary size 160K; 1 MTP layer.
sizes · MoonshotAI/Kimi-K3 / k3_tech_report.pdf
Pinned revision 3cb39dfd32e5 ↗ · p. 11
Direct evidence
§5.3.1: speculative decoding “further accelerates request turnover” in RL rollouts.
rollout · MoonshotAI/Kimi-K3 / k3_tech_report.pdf
Pinned revision 3cb39dfd32e5 ↗ · p. 21