IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /max/get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /max/get-started.md).

Python class

TextGenerationRequest

TextGenerationRequest​

class max.pipelines.modeling.types.pipeline_variants.TextGenerationRequest(request_id, model_name, prompt=None, messages=<factory>, images=<factory>, videos=<factory>, decoded_images=<factory>, tools=None, response_format=None, timestamp_ns=0, request_path='/', logprobs=0, echo=False, chat_template_options=None, sampling_params=<factory>, target_endpoint=None, dkv_cache_hint=None, cache_salt=None)

source

Bases: object

An immutable request for text token generation from a pipeline.

Parameters:

cache_salt​

cache_salt: str | None = None

source

Optional per-request salt that isolates this prompt’s prefix-cache

entries from other requests sharing the same tokens. Combined with the cluster-level kv_cache_hash_seed via XOR inside BlockManager to derive the root parent hash. Has effect only when kv_cache_hash_algo is sha256 or sha256_64; under ahash64 the salt is dropped with a one-time warning.

Capped at 512 chars at the OpenAI schema layer.

chat_template_options​

chat_template_options: dict[str, Any] | None = None

source

Optional dictionary of options to pass when applying the chat template.

decoded_images​

decoded_images: list[PILImage]

source

1 to images, decoded once at request admission (the API server validates images by fully decoding them, so the decoded result is carried here to avoid a second decode in the tokenizer). API-process-only: this is never serialized across the worker boundary, so it must stay populated only for the in-process tokenization step. Empty when images were not pre-decoded (offline/test callers); tokenizers fall back to decoding images in that case.

Type:

Decoded PIL.Image objects corresponding 1

dkv_cache_hint​

dkv_cache_hint: dict[str, Any] | None = None

source

Cache hint from the Orchestrator for distributed KV cache.

When present, the serving layer converts this into TextContext.external_block_metadata so the DKVConnector can fetch cached blocks before the forward pass.

echo​

echo: bool = False

source

If set to True, the response will include the original prompt along with the generated output. This can be useful for debugging or when you want to see how the input relates to the output.

images​

images: list[bytes]

source

A list of image byte arrays that can be included as part of the request. This field is optional and may be used for multimodal inputs where images are relevant to the prompt or task.

images_for_processing()​

images_for_processing()

source

Return the images for tokenizer preprocessing, decoded once.

Prefers the pre-decoded decoded_images (decoded and validated once at the API server) and falls back to the raw images bytes for offline and test callers. Tokenizers consume images through this so the decode-once policy lives in one place rather than being repeated at every per-model decode site.

Return type:

list[bytes | PILImage]

logprobs​

logprobs: int = 0

source

The number of top log probabilities to return for each generated token. A value of 0 means that log probabilities will not be returned. Useful for analyzing model confidence in its predictions.

messages​

messages: list[TextGenerationRequestMessage]

source

A list of messages for chat-based interactions. This is used in chat completion APIs, where each message represents a turn in the conversation. If provided, the model will generate responses based on these messages.

model_name​

model_name: str

source

The name of the model to be used for generating tokens. This should match the available models on the server and determines the behavior and capabilities of the response generation.

number_of_images​

property number_of_images: int

source

Returns the total number of image-type contents across all provided messages.

Returns:

Total count of image-type contents found in messages.

number_of_videos​

property number_of_videos: int

source

Returns the total number of video-type contents across all provided messages.

Returns:

Total count of video-type contents found in messages.

prompt​

prompt: str | Sequence[int] | None = None

source

The prompt to be processed by the model. This field supports legacy completion APIs and can accept either a string or a sequence of integers representing token IDs. If not provided, the model may generate output based on the messages field.

request_id​

request_id: RequestID

source

A unique identifier for the request.

request_path​

request_path: str = '/'

source

The endpoint path for the request. This is typically used for routing and logging requests within the server infrastructure.

response_format​

response_format: TextGenerationResponseFormat | None = None

source

Specifies the desired format for the model’s output. When set, it enables structured generation, which adheres to the json_schema provided.

sampling_params​

sampling_params: SamplingParams

source

Token sampling configuration parameters for the request.

target_endpoint​

target_endpoint: str | None = None

source

Optional target endpoint identifier for routing the request to a specific service or model instance. This should be used in disaggregate serving scenarios, when you want to dynamically route to a specific instance. If not specified, the request will be routed to the default endpoint.

timestamp_ns​

timestamp_ns: int = 0

source

The time (in nanoseconds) when the request was received by the server. This can be useful for performance monitoring and logging purposes.

tools​

tools: list[TextGenerationRequestTool] | None = None

source

A list of tools that can be invoked during the generation process. This allows the model to utilize external functionalities or APIs to enhance its responses.

videos​

videos: list[bytes]

source

A list of video byte arrays that can be included as part of the request. Each video is decoded into frames during preprocessing.

Models

Cookbook

Puzzles

Blog

Community

Contact