GPT-2

All models, with previews →

OpenAI · Feb–Nov 2019Dense decoder

Transformer block

Reference code openai/gpt-2 @ 9b63575, huggingface/transformers @ 71c6f69, configuration openai-community/gpt2 @ 607a30d · checkpoint tensors not independently counted · How to read

xn[B, T, 768]LayerNorm 1μ,  σ2\mu,\;\sigma^2gain · bias
AttentionΔattn=Attention⁡(Z1)\Delta_{\mathrm{attn}}=\operatorname{Attention}(Z_1)Open →
Dropoutpresidp_{\mathrm{resid}}+x′n[B, T, 768]LayerNorm 2μ,  σ2\mu,\;\sigma^2gain · bias
MLPΔmlp=MLP⁡(Z2)\Delta_{\mathrm{mlp}}=\operatorname{MLP}(Z_2)Open →
Dropoutpresidp_{\mathrm{resid}}+xn+1[B, T, 768]residual bypassresidual bypass