Raw text
×tensortext
Text supplied for tokenization.
Sources
Contextual evidence from Byte-level BPE tokenizer
The pinned configuration supplies the family vocabulary size.
configuration · openai-community/gpt2 / config.json
Pinned revision 607a30d783df ↗ · $
Contextual evidence from Byte-level BPE tokenizer
Original Python encoder shipped with the released checkpoints.
openai · openai/gpt-2 / src/encoder.py
Pinned revision 9b63575ef427 ↗ · lines 96–101
Read code excerpt
def encode(self, text):
bpe_tokens = []
for token in re.findall(self.pat, text):
token = ''.join(self.byte_encoder[b] for b in token.encode('utf-8'))
bpe_tokens.extend(self.encoder[bpe_token] for bpe_token in self.bpe(token).split(' '))
return bpe_tokensContextual evidence from Byte-level BPE tokenizer
The maintained tokenizer configures byte-level pre-tokenization and decoding with the released vocabulary and merges.
transformers · huggingface/transformers / src/transformers/models/gpt2/tokenization_gpt2.py
Pinned revision 71c6f699ac9b ↗ · lines 107–120
Read code excerpt
self._vocab = vocab if vocab is not None else {}
self._merges = merges or []
self._tokenizer = Tokenizer(
BPE(
vocab=self._vocab,
merges=self._merges,
dropout=None,
continuing_subword_prefix="",
end_of_word_suffix="",
fuse_unk=False,
)
)
self._tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=add_prefix_space)
self._tokenizer.decoder = decoders.ByteLevel()