For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

ModelInputs

ModelInputs

class max.pipelines.ModelInputs(*, kv_cache_inputs=None, lora=None, lora_buffers=(), vision_embeddings=<factory>, vision_scatter_indices=<factory>, hidden_states=None)

source

Bases: object

Base class for model inputs.

Use this class to encapsulate inputs for your model; you may store any number of dataclass fields.

The following example demonstrates how to create a custom inputs class:

from dataclasses import dataclass

from max.driver import Buffer
from max.dtype import DType
from max.pipelines.lib.interfaces.pipeline_model import ModelInputs

@dataclass
class ReplitInputs(ModelInputs):
    tokens: Buffer
    input_row_offsets: Buffer

tokens = Buffer.zeros((1, 2, 3), DType.int64)
input_row_offsets = Buffer.zeros((1, 1, 1), DType.int64)

# Initialize inputs
inputs = ReplitInputs(tokens=tokens, input_row_offsets=input_row_offsets)

# Access tensors
assert inputs.tokens is tokens
assert inputs.input_row_offsets is input_row_offsets

Parameters:

• kv_cache_inputs (KVCacheInputsInterface[Buffer, Buffer] | None)
• lora (LoRAInputs | None)
• lora_buffers (tuple[Buffer, ...])
• vision_embeddings (list[Buffer])
• vision_scatter_indices (list[Buffer])
• hidden_states (Buffer | list[Buffer] | None)

buffers

property buffers: tuple[Buffer, ...]

source

Returns positional Buffer inputs for model ABI calls.

hidden_states

hidden_states: Buffer | list[Buffer] | None = None

source

Hidden states for a variable number of tokens per sequence.

For data parallel models, this can be a list of Buffers where each Buffer contains hidden states for the sequences assigned to that device.

kv_cache_inputs

kv_cache_inputs: KVCacheInputsInterface[Buffer, Buffer] | None = None

source

KV cache graph inputs holding every (DP replica x TP shard) device’s inputs: a KVCacheInputs leaf, or a MultiKVCacheInputs tree for multi-cache models. flatten() yields the full positional input list.

lora

lora: LoRAInputs | None = None

source

Per-batch LoRA adapter buffers, or None when LoRA is disabled.

lora_buffers

lora_buffers: tuple[Buffer, ...] = ()

source

ModuleV3 LoRA graph inputs (routing triple + per-slot adapter stacks) in LoRAManagerV3.symbolic_inputs order, set by the ModuleV3 batch processor; empty when ModuleV3 LoRA is off. The arch’s buffers property splices these onto the positional ABI tail.

update()

update(**kwargs)

source

Updates attributes from keyword arguments (only existing, non-None).

Return type:

None

vision_embeddings

vision_embeddings: list[Buffer]

source

Per-device vision-merge embedding inputs for the language graph, set by the pipeline’s vision seam (finalize_vision_inputs) on every prepared batch of a vision-capable model: the assembled embeddings when this step encoded images, the model’s cached zero-row empties otherwise. Stays empty for text-only architectures.

vision_scatter_indices

vision_scatter_indices: list[Buffer]

source

Per-device merge (scatter) indices for vision_embeddings, with the same lifecycle.