RL
×Stage · DeepSeek-V3
GRPO with rule-based and model-based rewards.
- Algorithm
- GRPO: a group of outputs per question from the old policy, the advantage normalised within the group, a clipped ratio and a KL penalty to a reference model
- Rewards
- Rule-based where answers can be checked (formats, compilers and tests); a reward model trained from the V3 SFT checkpoints elsewhere
- Prompts
- Coding, mathematics, writing, role-playing and question answering
- Self-feedback
- DeepSeek-V3 with voting judges open-ended answers; the constitutional AI approach for broader contexts
- Compute
- 5K H800 GPU hours for SFT and RL together
Not disclosed Group size, ε, β · Steps · Learning rate
Sources4
- DeepSeek-V3 report, p. 30Algorithm, Prompts
- DeepSeek-V3 report, p. 29Rewards
- DeepSeek-V3 report, pp. 33–34Self-feedback
- DeepSeek-V3 report, p. 5Compute
Key
- stage
- tokens, log scale
- a total no source states, computed from steps × batch × length
- checkpoint
- checkpoint, weights published
- a join the sources leave unexplained
- stages of a released model, folded into one row
- derived
- computed from reported values; the arithmetic is shown
- not disclosed
- no pinned source states it; nothing is filled in