Python class

PipelineTokenizer

`PipelineTokenizer`

class max.interfaces.PipelineTokenizer(*args, **kwargs)

source

Bases: Protocol[UnboundContextType, TokenizerEncoded, RequestType]

Interface for LLM tokenizers.

`decode()`

async decode(encoded, **kwargs)

source

Decodes response tokens to text.

Parameters:

encoded (TokenizerEncoded) – Encoded response tokens.
**kwargs – Additional decoder options (for example, skip_special_tokens).

Returns:

Un-encoded response text.

Return type:

str

`encode()`

async encode(prompt, add_special_tokens)

source

Encodes text prompts as tokens.

Parameters:

prompt (str) – Un-encoded prompt text.
add_special_tokens (bool) – Whether to add special tokens (for example, BOS).

Raises:

ValueError – If the prompt exceeds the configured maximum length.

Return type:

TokenizerEncoded

`eos`

property eos: int

source

The end of sequence token for this tokenizer.

`expects_content_wrapping`

property expects_content_wrapping: bool

source

If True, this tokenizer expects messages to be wrapped as a dict.

Text messages are formatted as:

{
  "role": "user",
  "content": [{ "type": "text", "text": "text content" }]
}

instead of:

{ "role": "user", "content": "text_content" }

NOTE: Multimodal messages omit the content property. Both image_urls and image content parts are converted to:

{ "type": "image" }

Their content is provided as byte arrays through the top-level property on the request object, that is, RequestType.images.

`new_context()`

async new_context(request)

source

Creates a new context from a request object.

This is sent to the worker process once and then cached locally.

Parameters:: request (RequestType) – Incoming request.
Returns:: Initialized context.
Return type:: UnboundContextType

PipelineTokenizer​

decode()​

encode()​

eos​

expects_content_wrapping​

new_context()​