For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /max/get-started.md).
Python class
TextGenerationPipeline
TextGenerationPipeline
class max.pipelines.TextGenerationPipeline(pipeline_config, pipeline_model, weight_adapters, tokenizer, memory_plan)
Bases: TextGenerationPipelineInterface[TextGenerationContextType], Generic[TextGenerationContextType]
Generalized token generator pipeline.
Initialize a text generation pipeline instance.
This sets up devices, the inference session, tokenizer, KV-cache manager, sampling kernel, and loads model weights and adapters.
-
Parameters:
-
- pipeline_config (PipelineConfig) – Configuration for the pipeline and runtime behavior.
- pipeline_model (type[PipelineModel[TextGenerationContextType]]) – Concrete model implementation to use for execution.
- weight_adapters (dict[WeightsFormat, WeightsAdapter]) – Mapping from weights format to adapter implementation.
- tokenizer (PipelineTokenizer[TextGenerationContextType, npt.NDArray[np.integer[Any]], TextGenerationRequest]) – Tokenizer implementation used to build contexts and decode.
- memory_plan (_MemoryPlan) – Memory plan from the registry containing max_batch_size and other resolved memory parameters.
-
Raises:
-
ValueError – If
quantization_encodingis not configured inpipeline_config.modelor if structured output is requested without a valid tokenizer delegate.
batch_vision_metrics()
batch_vision_metrics()
Returns vision encoder metrics for the most recent batch.
Returns None for text-only models and for batches that did no
vision encoding (e.g. decode steps). The metrics come from the
pipeline-owned VisionEncoderCache, if this pipeline has one.
-
Return type:
-
VisionEncoderMetrics | None
execute()
execute(inputs)
Processes the batch and returns decoded tokens.
Executes the graph for a single decode step, samples the next token, then decodes and returns the generated tokens.
-
Parameters:
-
inputs (TextGenerationInputs[TextGenerationContextType])
-
Return type:
initialize_bitmask()
initialize_bitmask(batch)
Allocates a per-request token bitmask for structured decoding.
kv_manager
property kv_manager: PagedKVCacheManager
Returns the KV cache manager for this pipeline.
max_batch_size
property max_batch_size: int
Maximum number of requests that can be processed in a single batch.
pipeline_config
property pipeline_config: PipelineConfig
Return the pipeline configuration.
prepare_batch()
prepare_batch(batches)
Prepare model inputs and ancillary state for execution.
This flattens replica batches, optionally initializes constrained decoding bitmasks, ensures KV-cache reservations, and builds initial model inputs.
release()
release(request_id)
Release model-specific resources for a completed request.
Primary and extra KV cache lifecycle is managed by the batch constructor. This method drops the request’s vision-encoder-cache references and any model-specific state.
-
Parameters:
-
request_id (RequestID)
-
Return type:
-
None
tokenizer
property tokenizer: PipelineTokenizer[TextGenerationContextType, ndarray[tuple[Any, ...], dtype[integer[Any]]], TextGenerationRequest]
Return the tokenizer used for building contexts and decoding.
update_for_structured_output()
update_for_structured_output(context, bitmask, index)
Update context and logits bitmask for structured output.
If a json_schema is present and no matcher is set, this compiles a
grammar matcher and installs it on the context. It may also jump ahead in
generation and fills the per-request token bitmask used to constrain the
next-token distribution.
-
Parameters:
-
Raises:
-
InputError – If a JSON schema is provided but structured output is not enabled via sampling configuration.
-
Return type:
-
None
Was this page helpful?
Thank you! We'll create more content like this.
Thank you for helping us improve!