Python module

tokenizer

Implementations of provided tokenizers.

IdentityPipelineTokenizer

class max.pipelines.lib.tokenizer.IdentityPipelineTokenizer(*args, **kwargs)

decode()

async decode(context, encoded, **kwargs)

Decodes response tokens to text.

Parameters:

• context (TokenGeneratorContext) – Current generation context.
• encoded (TokenizerEncoded) – Encoded response tokens.

Returns:

Un-encoded response text.

Return type:

str

encode()

async encode(prompt, add_special_tokens=False)

Encodes text prompts as tokens.

Parameters:

• prompt (str) – Un-encoded prompt text.
• add_special_tokens (bool)

Raises:

ValueError – If the prompt exceeds the configured maximum length.

Return type:

str

eos

property eos: int

The end of sequence token for this tokenizer.

expects_content_wrapping

property expects_content_wrapping: bool

If true, this tokenizer expects messages to have a content property.

Text messages are formatted as:

{ "type": "text", "content": "text content" }

{ "type": "text", "content": "text content" }

instead of the OpenAI spec:

{ "type": "text", "text": "text content" }

{ "type": "text", "text": "text content" }

NOTE: Multimodal messages omit the content property. Both image_urls and image content parts are converted to:

{ "type": "image" }

{ "type": "image" }

Their content is provided as byte arrays through the top-level property on the request object, i.e., PipelineTokenizerRequest.images.

PreTrainedPipelineTokenizer

class max.pipelines.lib.tokenizer.PreTrainedPipelineTokenizer(delegate)

Parameters:

delegate (Union[PreTrainedTokenizer, PreTrainedTokenizerFast])

apply_chat_template()

apply_chat_template(messages)

Parameters:

messages (list[TokenGeneratorRequestMessage])

Return type:

str

decode()

async decode(context, encoded, **kwargs)

Decodes response tokens to text.

Parameters:

• context (TokenGeneratorContext) – Current generation context.
• encoded (TokenizerEncoded) – Encoded response tokens.

Returns:

Un-encoded response text.

Return type:

str

encode()

async encode(prompt, add_special_tokens=False)

Encodes text prompts as tokens.

Parameters:

• prompt (str) – Un-encoded prompt text.
• add_special_tokens (bool)

Raises:

ValueError – If the prompt exceeds the configured maximum length.

Return type:

ndarray

eos

property eos: int

The end of sequence token for this tokenizer.

expects_content_wrapping

property expects_content_wrapping: bool

If true, this tokenizer expects messages to have a content property.

Text messages are formatted as:

{ "type": "text", "content": "text content" }

{ "type": "text", "content": "text content" }

instead of the OpenAI spec:

{ "type": "text", "text": "text content" }

{ "type": "text", "text": "text content" }

NOTE: Multimodal messages omit the content property. Both image_urls and image content parts are converted to:

{ "type": "image" }

{ "type": "image" }

Their content is provided as byte arrays through the top-level property on the request object, i.e., PipelineTokenizerRequest.images.

TextAndVisionTokenizer

class max.pipelines.lib.tokenizer.TextAndVisionTokenizer(model_path, *, revision=None, max_length=None, max_new_tokens=None, trust_remote_code=False, **unused_kwargs)

Encapsulates creation of TextContext and specific token encode/decode logic.

Parameters:

• model_path (str)
• revision (str | None)
• max_length (int | None)
• max_new_tokens (int | None)
• trust_remote_code (bool)

apply_chat_template()

apply_chat_template(messages)

Parameters:

messages (list[TokenGeneratorRequestMessage])

Return type:

str

decode()

async decode(context, encoded, **kwargs)

Transformer a provided encoded token array, back into readable text.

Parameters:

• context (TextAndVisionContext)
• encoded (ndarray)

Return type:

str

encode()

async encode(prompt, add_special_tokens=True)

Transform the provided prompt into a token array.

Parameters:

• prompt (str | Sequence[int])
• add_special_tokens (bool)

Return type:

ndarray

eos

property eos: int

The end of sequence token for this tokenizer.

expects_content_wrapping

property expects_content_wrapping: bool

If true, this tokenizer expects messages to have a content property.

Text messages are formatted as:

{ "type": "text", "content": "text content" }

{ "type": "text", "content": "text content" }

instead of the OpenAI spec:

{ "type": "text", "text": "text content" }

{ "type": "text", "text": "text content" }

NOTE: Multimodal messages omit the content property. Both image_urls and image content parts are converted to:

{ "type": "image" }

{ "type": "image" }

Their content is provided as byte arrays through the top-level property on the request object, i.e., PipelineTokenizerRequest.images.

new_context()

async new_context(request)

Create a new TextAndVisionContext object, leveraging necessary information like cache_seq_id and prompt from TokenGeneratorRequest.

Parameters:

request (TokenGeneratorRequest)

Return type:

TextAndVisionContext

TextTokenizer

class max.pipelines.lib.tokenizer.TextTokenizer(model_path, *, revision=None, max_length=None, max_new_tokens=None, trust_remote_code=False, enable_llama_whitespace_fix=False, **unused_kwargs)

Encapsulates creation of TextContext and specific token encode/decode logic.

Parameters:

• model_path (str)
• revision (str | None)
• max_length (int | None)
• max_new_tokens (int | None)
• trust_remote_code (bool)
• enable_llama_whitespace_fix (bool)

apply_chat_template()

apply_chat_template(messages, tools, chat_template_options=None)

Parameters:

• messages (list[TokenGeneratorRequestMessage])
• tools (list[TokenGeneratorRequestTool] | None)
• chat_template_options (dict[str, Any] | None)

Return type:

str

decode()

async decode(context, encoded, **kwargs)

Transformer a provided encoded token array, back into readable text.

Parameters:

• context (TextContext)
• encoded (ndarray)

Return type:

str

encode()

async encode(prompt, add_special_tokens=True)

Transform the provided prompt into a token array.

Parameters:

• prompt (str | Sequence[int])
• add_special_tokens (bool)

Return type:

ndarray

eos

property eos: int

The end of sequence token for this tokenizer.

expects_content_wrapping

property expects_content_wrapping: bool

If true, this tokenizer expects messages to have a content property.

Text messages are formatted as:

{ "type": "text", "content": "text content" }

{ "type": "text", "content": "text content" }

instead of the OpenAI spec:

{ "type": "text", "text": "text content" }

{ "type": "text", "text": "text content" }

NOTE: Multimodal messages omit the content property. Both image_urls and image content parts are converted to:

{ "type": "image" }

{ "type": "image" }

Their content is provided as byte arrays through the top-level property on the request object, i.e., PipelineTokenizerRequest.images.

new_context()

async new_context(request)

Create a new TextContext object, leveraging necessary information like cache_seq_id and prompt from TokenGeneratorRequest.

Parameters:

request (TokenGeneratorRequest)

Return type:

TextContext

max_tokens_to_generate()

max.pipelines.lib.tokenizer.max_tokens_to_generate(prompt_size, max_length, max_new_tokens=None)

Returns the max number of new tokens to generate.

Parameters:

• prompt_size (int)
• max_length (int | None)
• max_new_tokens (int | None)

Return type:

int | None

run_with_default_executor()

async max.pipelines.lib.tokenizer.run_with_default_executor(fn, *args)

Was this page helpful?

Thank you! We'll create more content like this.

Thank you for helping us improve!

😔 What went wrong?

Some code doesn’t work

It includes inaccurate information

It's missing information I need

It was difficult to understand

Other

Submit