Raw text
×tensormessages
Text supplied for tokenization.
Codetokenizer_config.json 149–156
"tokenizer_class": "TikTokenTokenizer",
"unk_token": "[UNK]",
"auto_map": {
"AutoTokenizer": [
"tokenization_kimi.TikTokenTokenizer",
null
]
}Sources1
Contextual evidence
The checkpoint’s tokenizer is the repository’s own TikTokenTokenizer class, loaded through AutoTokenizer.
tokenizer · moonshotai/Kimi-K3 / tokenizer_config.json
Pinned revision f831ab668142 ↗ · lines 149–156
Read code excerpt
"tokenizer_class": "TikTokenTokenizer",
"unk_token": "[UNK]",
"auto_map": {
"AutoTokenizer": [
"tokenization_kimi.TikTokenTokenizer",
null
]
}