Python class
TextAndVisionTokenizer
TextAndVisionTokenizer
class max.pipelines.TextAndVisionTokenizer(model_path, pipeline_config, *, revision=None, max_length=None, trust_remote_code=False, **unused_kwargs)
Bases: PipelineTokenizer[TextAndVisionContext, ndarray[tuple[Any, …], dtype[integer[Any]]], TextGenerationRequest]
Encapsulates creation of TextAndVisionContext and specific token encode/decode logic.
-
Parameters:
-
- model_path (str)
- pipeline_config (PipelineConfig)
- revision (str | None)
- max_length (int | None)
- trust_remote_code (bool)
apply_chat_template()
apply_chat_template(messages, tools=None)
Applies the processor’s chat template to the messages.
-
Parameters:
-
- messages (list[TextGenerationRequestMessage])
- tools (list[TextGenerationRequestTool] | None)
-
Return type:
create_eos_tracker()
async create_eos_tracker(request)
Builds an EOSTracker from the request sampling params and tokenizer default EOS token IDs.
-
Parameters:
-
request (TextGenerationRequest)
-
Return type:
-
EOSTracker
decode()
async decode(encoded, **kwargs)
Transforms a provided encoded token array back into readable text.
encode()
async encode(prompt, add_special_tokens=True)
Transforms the provided prompt into a token array.
eos
property eos: int
Returns the end-of-sequence token ID from the delegate.
expects_content_wrapping
property expects_content_wrapping: bool
Returns whether this tokenizer expects content wrapping.
new_context()
async new_context(request)
Create a new TextAndVisionContext object, leveraging necessary information from TextGenerationRequest.
-
Parameters:
-
request (TextGenerationRequest)
-
Return type:
Was this page helpful?
Thank you! We'll create more content like this.
Thank you for helping us improve!