RL
×Stage · DeepSeek-V4.1-Flash
Large-scale asynchronous reinforcement learning in synthesized agent tasks, with the reasoning effort as a condition and a length penalty that depends on it.
- Kind
- asynchronous, in synthesized agent tasks
- Effort
- a scalar from 1 to 100 in the prompt
- Length penalty
- falls exponentially with the effort
- Successive runs
- merged checkpoints start the next run
Not disclosed Context of this stage · What is trained · Group size · The trained effort levels · k₀, λ, C_max, L_norm · Clipping, KL term · Learning rate · Tokens
Why
The report says the gains come from the scale, diversity and verifiability of synthesized tasks and environments, under a fixed and unremarkable optimization procedure. DeepSeek-V4.1 report §5.1, p. 25
The training step →What one step of this stage computes, as a plate.
Architecture at this stageStated for post-training as a whole, with no stage named: the Hierarchical Sparse Indexer is switched on, the main KV cache is trained in FP4, and DSpark trains beside the backbone with its gradient cut.
All settings14
- Kind
- Large-scale asynchronous RL in synthesized tasks, over several agent scaffolds
- Algorithm
- GRPO is named only through its group; rewards are mean-centered within a subgroup to give group-relative advantages; the report writes no objective
- Effort
- A scalar effort b from 1 to 100 in the system prompt; responses sampled for the same prompt and effort form a subgroup (eq. 8)
- Length penalty
- Added to the reward: −min(C_max, k(b)·ℓ / L_norm), ℓ the reasoning tokens, with k(b) falling exponentially in b (eq. 9, 10)
- Tasks
- A task is a problem, an environment and a verification system; environments for general agents and coding agents are synthesized and checked by agents
- Successive runs
- Checkpoints of runs across scaffolds or configurations are merged to start the next run
- Other reward signals
- A crashed environment is a failed trajectory with a “repercussion” signal; Agent Team mode has a reward of its own
- Asynchronous training
- A prompt is dispatched once as many samples complete as its group holds; recorded expert routing is replayed; tokens too stale are masked from the loss
- Figures
- Figure 7 labels its runs 512K and, on one benchmark from about 1,900 cumulative steps, 1M; three set-ups are plotted to about 2,000, 2,500 and 4,500 cumulative RL steps, none identified as the released model’s
- DSpark
- Trained alongside the backbone through post-training, with no gradient from its objective into the backbone
- Hierarchical Sparse Indexer
- “Training-aware and introduced in post-training”: later indexers score a candidate pool an earlier one selected
- Number format
- Quantization-aware training for the main KV cache in FP4, introduced in post-training; the sliding-window KV cache stays FP8
- Replay
- The decoder’s bounded replay of the sliding window is simulated in post-training
- Long sequences
- “A substantial share of training occurring at ultra-long sequence lengths in both the pre-training and post-training stages”, of sequences up to one million tokens
Sources12
- DeepSeek_V41_Tech_Report.pdf, p. 26Kind
- DeepSeek_V41_Tech_Report.pdf, p. 29Algorithm, Effort, Length penalty, Other reward signals
- DeepSeek_V41_Tech_Report.pdf, p. 30Algorithm
- DeepSeek_V41_Tech_Report.pdf, pp. 25–26Tasks
- DeepSeek_V41_Tech_Report.pdf, p. 27Successive runs
- DeepSeek_V41_Tech_Report.pdf, p. 36Other reward signals
- DeepSeek_V41_Tech_Report.pdf, pp. 30–31Asynchronous training
- DeepSeek_V41_Tech_Report.pdf, pp. 27–28Figures
- DeepSeek_V41_Tech_Report.pdf, p. 14DSpark, Number format
- DeepSeek_V41_Tech_Report.pdf, p. 12Hierarchical Sparse Indexer
- DeepSeek_V41_Tech_Report.pdf, p. 20Replay
- DeepSeek_V41_Tech_Report.pdf, p. 17Long sequences