Checkpoint tokenizer
×no parametersTikTokenTokenizer
The tiktoken vocabulary, tokenizer class and chat template shipped with the checkpoint.
Unverified: the tiktoken vocabulary file and its token-to-ID mapping, which the pinned sources do not include.
Explanation
The checkpoint’s own TikTokenTokenizer class loads the vocabulary; tokenizer_config.json names it (156–160).
Codetokenizer_config.json 156–164 · chat_template.jinja 20–52
"tokenizer_class": "TikTokenTokenizer",
"unk_token": "[UNK]",
"auto_map": {
"AutoTokenizer": [
"tokenization_kimi.TikTokenTokenizer",
null
]
}
}{% for message in messages %}
{%- if loop.first and messages[0]['role'] != 'system' -%}
<|im_system|>system<|im_middle|>You are Kimi, an AI assistant created by Moonshot AI.<|im_end|>
{% endif %}
{%- set role_name = message.get('name') or message['role'] -%}
{%- if message['role'] == 'user' -%}
<|im_user|>{{role_name}}<|im_middle|>
{%- elif message['role'] == 'assistant' -%}
<|im_assistant|>{{role_name}}<|im_middle|>
{%- else -%}
<|im_system|>{{role_name}}<|im_middle|>
{%- endif -%}
{%- if message['role'] == 'assistant' and message.get('tool_calls') -%}
{{render_content(message)}}<|tool_calls_section_begin|>
{%- for tool_call in message['tool_calls'] -%}
{%- set formatted_id = tool_call['id'] -%}
<|tool_call_begin|>{{ formatted_id }}<|tool_call_argument_begin|>{% if tool_call['function']['arguments'] is string %}{{ tool_call['function']['arguments'] }}{% else %}{{ tool_call['function']['arguments'] | tojson }}{% endif %}<|tool_call_end|>
{%- endfor -%}
<|tool_calls_section_end|>
{%- elif message['role'] == 'tool' -%}
{%- set tool_call_id = message.tool_call_id -%}
## Return of {{ tool_call_id }}
{{render_content(message)}}
{%- elif message['content'] is not none -%}
{{render_content(message)}}
{%- endif -%}
<|im_end|>
{%- endfor -%}
{%- if add_generation_prompt -%}
<|im_assistant|>assistant<|im_middle|>
{%- endif -%}Sources2
Contextual evidence
The checkpoint’s tokenizer is the repository’s own TikTokenTokenizer class, loaded through AutoTokenizer.
tokenizer · moonshotai/Kimi-K2-Instruct / tokenizer_config.json
Pinned revision fd1984e2b7a3 ↗ · lines 156–164
Read code excerpt
"tokenizer_class": "TikTokenTokenizer",
"unk_token": "[UNK]",
"auto_map": {
"AutoTokenizer": [
"tokenization_kimi.TikTokenTokenizer",
null
]
}
}Contextual evidence
The chat template wraps each message in role markers (<|im_user|>, <|im_assistant|>, <|im_system|>) ending in <|im_end|>, adding a default system message when none is given.
chat-template · moonshotai/Kimi-K2-Instruct / chat_template.jinja
Pinned revision fd1984e2b7a3 ↗ · lines 20–52
Read code excerpt
{% for message in messages %}
{%- if loop.first and messages[0]['role'] != 'system' -%}
<|im_system|>system<|im_middle|>You are Kimi, an AI assistant created by Moonshot AI.<|im_end|>
{% endif %}
{%- set role_name = message.get('name') or message['role'] -%}
{%- if message['role'] == 'user' -%}
<|im_user|>{{role_name}}<|im_middle|>
{%- elif message['role'] == 'assistant' -%}
<|im_assistant|>{{role_name}}<|im_middle|>
{%- else -%}
<|im_system|>{{role_name}}<|im_middle|>
{%- endif -%}
{%- if message['role'] == 'assistant' and message.get('tool_calls') -%}
{{render_content(message)}}<|tool_calls_section_begin|>
{%- for tool_call in message['tool_calls'] -%}
{%- set formatted_id = tool_call['id'] -%}
<|tool_call_begin|>{{ formatted_id }}<|tool_call_argument_begin|>{% if tool_call['function']['arguments'] is string %}{{ tool_call['function']['arguments'] }}{% else %}{{ tool_call['function']['arguments'] | tojson }}{% endif %}<|tool_call_end|>
{%- endfor -%}
<|tool_calls_section_end|>
{%- elif message['role'] == 'tool' -%}
{%- set tool_call_id = message.tool_call_id -%}
## Return of {{ tool_call_id }}
{{render_content(message)}}
{%- elif message['content'] is not none -%}
{{render_content(message)}}
{%- endif -%}
<|im_end|>
{%- endfor -%}
{%- if add_generation_prompt -%}
<|im_assistant|>assistant<|im_middle|>
{%- endif -%}