GPT-2

All models, with previews →

OpenAI · Feb–Nov 2019Dense decoder

Position-wise feed-forward network

Reference code openai/gpt-2 @ 9b63575, huggingface/transformers @ 71c6f69, configuration openai-community/gpt2 @ 607a30d · checkpoint tensors not independently counted · How to read

Transformer block › MLP

Approximate GELU

×

no parametersgelu(…), applied to conv1d(x, 'c_fc', n_state)

A smooth feature-wise activation, computed through its tanh approximation.

g=h2[1+tanh⁡ ⁣(2π(h+0.044715h3))]{\htmlData{light=activated gelu}{g}}=\frac{{\htmlData{light=pre-activation c_fc}{h}}}{2}\left[1+\tanh\!\left(\sqrt{\frac{2}{\pi}}\left({\htmlData{light=pre-activation c_fc}{h}}+0.044715{\htmlData{light=pre-activation c_fc}{h^3}}\right)\right)\right]
in
hh [B batch, T positions, 3072 d_ff]
out
gg [B batch, T positions, 3072 d_ff]
Explanation

The function is near zero for strongly negative inputs, dips to about −0.17 near −0.75, and approaches the identity for large positive ones.

-3-2-10123y = xGELU
The dashed line is y = x. Function evaluated from −4 to 4.
Codemodel.py 25–26
model.py 25–26
def gelu(x):
    return 0.5*x*(1+tf.tanh(np.sqrt(2/np.pi)*(x+0.044715*tf.pow(x, 3))))
Sources1

Direct evidence

The released graph defines the tanh approximation used by GPT-2.

openai · openai/gpt-2 / src/model.py
Pinned revision 9b63575ef427 ↗ · lines 25–26

Read code excerpt
def gelu(x):
    return 0.5*x*(1+tf.tanh(np.sqrt(2/np.pi)*(x+0.044715*tf.pow(x, 3))))