Vision encoder: contrastive
×Stage · DeepSeek-ViT
The vision encoder, trained from scratch, first learns from image–text pairs with a contrastive loss at low resolution.
- Data
- about 47B image–text pairs
- Objective
- The sigmoid contrastive loss of SigLIP
- Resolution
- at most 224 × 224
Not disclosed Tokens · Steps · Batch · Learning rate · Optimizer
Why
Higher resolution in this phase costs much more and adds little to the final model, since the next stage handles high resolution. DeepSeek-V4.1 report §4.2.2, p. 23
Architecture at this stageThe vision encoder; the text side of the contrastive pairs is not described.
All settings4
- Objective
- The sigmoid contrastive loss of SigLIP
- Data
- About 47B image–text pairs from alt-text data
- Resolution
- At most 224 × 224, larger images downscaled with their aspect ratio kept
- Trainable
- The vision encoder, trained from scratch, is “first optimized with a contrastive objective”; the text side of the pairs, which also takes gradients, is not described
Sources3
- DeepSeek_V41_Tech_Report.pdf, p. 23Objective, Data, Resolution
- DeepSeek_V41_Tech_Report.pdf, p. 8Trainable
- DeepSeek_V41_Tech_Report.pdf, p. 16Trainable