For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /max/get-started.md).
Python class
TextTokenizer
TextTokenizer
class max.pipelines.TextTokenizer(model_path, pipeline_config, *, revision=None, max_length=None, trust_remote_code=False, enable_llama_whitespace_fix=False, chat_template=None, **unused_kwargs)
Bases: PipelineTokenizer[TextContext, ndarray[tuple[Any, …], dtype[integer[Any]]], TextGenerationRequest]
Encapsulates creation of TextContext and specific token encode/decode logic.
-
Parameters:
-
- model_path (str) – Path to the model/tokenizer
- revision (str | None) – Git revision/branch to use
- max_length (int | None) – Maximum sequence length
- trust_remote_code (bool) – Whether to trust remote code from the model
- enable_llama_whitespace_fix (bool) – Enable whitespace fix for Llama tokenizers
- pipeline_config (PipelineConfig) – Optional pipeline configuration
- chat_template (str | None) – Optional custom chat template string to override the one shipped with the Hugging Face model config. This allows customizing the prompt formatting for different use cases.
apply_chat_template()
apply_chat_template(messages, tools, **chat_template_options)
Applies the delegate chat template to messages (and optional tools).
-
Parameters:
-
- messages (list[TextGenerationRequestMessage])
- tools (list[TextGenerationRequestTool] | None)
- chat_template_options (Any)
-
Return type:
create_eos_tracker()
async create_eos_tracker(request)
Builds an EOSTracker from the request sampling params and tokenizer default EOS token IDs.
-
Parameters:
-
request (TextGenerationRequest)
-
Return type:
decode()
async decode(encoded, **kwargs)
Transforms a provided encoded token array back into readable text.
encode()
async encode(prompt, add_special_tokens=True)
Transforms the provided prompt into a token array.
eos
property eos: int
Returns the end-of-sequence token ID from the delegate.
expects_content_wrapping
property expects_content_wrapping: bool
Returns whether this tokenizer expects content wrapping.
new_context()
async new_context(request)
Create a new TextContext object, leveraging necessary information from TextGenerationRequest.
-
Parameters:
-
request (TextGenerationRequest)
-
Return type:
tokenizer_vocab_size
property tokenizer_vocab_size: int
Vocabulary size of the HuggingFace tokenizer delegate.
Was this page helpful?
Thank you! We'll create more content like this.
Thank you for helping us improve!