For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /max/get-started.md).
Python class
TextAndVisionTokenizer
TextAndVisionTokenizer
class max.pipelines.TextAndVisionTokenizer(model_path, pipeline_config, *, revision=None, max_length=None, trust_remote_code=False, **unused_kwargs)
Bases: PipelineTokenizer[TextAndVisionContext, ndarray[tuple[Any, …], dtype[integer[Any]]], TextGenerationRequest]
Encapsulates creation of TextAndVisionContext and specific token encode/decode logic.
-
Parameters:
-
- model_path (str)
- pipeline_config (PipelineConfig)
- revision (str | None)
- max_length (int | None)
- trust_remote_code (bool)
apply_chat_template()
apply_chat_template(messages, tools=None, **chat_template_options)
Applies the processor’s chat template to the messages.
-
Parameters:
-
- messages (list[TextGenerationRequestMessage]) – List of messages for the chat template.
- tools (list[TextGenerationRequestTool] | None) – Optional tools available for the model to invoke.
- **chat_template_options (Any) – Template options to forward to the Jinja
template. Merged with
add_generation_prompt=Truedefault.
-
Returns:
-
The templated chat message as a string.
-
Return type:
create_eos_tracker()
async create_eos_tracker(request)
Builds an EOSTracker from the request sampling params and tokenizer default EOS token IDs.
-
Parameters:
-
request (TextGenerationRequest)
-
Return type:
decode()
async decode(encoded, **kwargs)
Transforms a provided encoded token array back into readable text.
encode()
async encode(prompt, add_special_tokens=True)
Transforms the provided prompt into a token array.
eos
property eos: int
Returns the end-of-sequence token ID from the delegate.
expects_content_wrapping
property expects_content_wrapping: bool
Returns whether this tokenizer expects content wrapping.
new_context()
async new_context(request)
Create a new TextAndVisionContext object, leveraging necessary information from TextGenerationRequest.
-
Parameters:
-
request (TextGenerationRequest)
-
Return type:
tokenizer_vocab_size
property tokenizer_vocab_size: int
Vocabulary size of the HuggingFace tokenizer delegate.
Was this page helpful?
Thank you! We'll create more content like this.
Thank you for helping us improve!