Models
Nine models, 2019 to 2026, drawn from their published code and papers. Grouped by what their blocks are made of; open one to read it part by part.
Dense decoders
Each block is one attention and one feed-forward network, and every token passes through all of a block’s weights.
3 modelsFollows Llama 3: the same blocks, with a 128K context
Latent attention and experts
Keys and values pass through a compressed latent (MLA); after the first few blocks the feed-forward network is many experts, a few chosen for each token.
3 modelsNew parts in the block
Experts as before, with parts the earlier lines lack: linear attention and residuals that attend over earlier layers (Kimi K3); a lookup memory and four residual streams (DeepSeek V4.1).
2 models- DeepSeek V4.1552B + 196B Engramsliding windowcompressed sparse attentionexpertsEngram memoryfour residual streamsRMSNormRoPESwiGLU× 40 layers1M context
After DeepSeek-V3.2-Exp: DeepSeek’s earlier model in the atlas
After Kimi K2: Moonshot AI’s earlier model in the atlas
Diffusion language models
Text made by denoising a whole sequence of latents, all positions at once, step by step.
1 model- LDLMIn construction306M trainedPerceiver attentionRMSNormLayerNormRoPElearned positionsSwiGLUGELUdiffusion time× 12 denoiser layers1K context
No models match your search.