For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python module

max.pipelines.architectures.inkling

InklingConfig

class max.pipelines.architectures.inkling.InklingConfig(*, devices, dtype, kv_params, max_seq_len, text_config, quant_config=None, use_subgraphs=True)

source

Bases: ArchConfigWithKVCache

Top-level Inkling config wrapping the text backbone.

Parameters:

• devices (list[DeviceRef])
• dtype (DType)
• kv_params (MultiKVCacheParams)
• max_seq_len (int)
• text_config (InklingTextConfig)
• quant_config (QuantConfig | None)
• use_subgraphs (bool)

DEFAULT_ENCODING

DEFAULT_ENCODING: ClassVar[max.pipelines.lib.config.SupportedEncoding] = 'bfloat16'

source

SUPPORTED_ENCODINGS

SUPPORTED_ENCODINGS: ClassVar[set[max.pipelines.lib.config.SupportedEncoding]] = {'bfloat16', 'float4_e2m1fnx2'}

source

calculate_max_seq_len()

static calculate_max_seq_len(pipeline_config, huggingface_config, model_config=None)

source

Parameters:

• pipeline_config (PipelineConfig)
• huggingface_config (AutoConfig)
• model_config (MAXModelConfig | None)

Return type:

int

construct_kv_params()

static construct_kv_params(huggingface_config, pipeline_config, devices, kv_cache_config, cache_dtype)

source

One cache per attention flavor; they differ in KV head count.

Parameters:

• huggingface_config (AutoConfig)
• pipeline_config (PipelineConfig)
• devices (list[DeviceRef])
• kv_cache_config (KVCacheConfig)
• cache_dtype (DType)

Return type:

MultiKVCacheParams

devices

devices: list[DeviceRef]

source

dtype

dtype: DType

source

finalize()

finalize(huggingface_config, state_dict)

source

Sets quant_config from which routed tensors ship block scales; the released NVFP4 checkpoint leaves one MoE layer bfloat16.

Parameters:

• huggingface_config (AutoConfig)
• state_dict (Mapping[str, WeightData])

Return type:

None

get_kv_params()

get_kv_params()

source

KV cache parameters to use when running the model.

Return type:

MultiKVCacheParams

get_max_seq_len()

get_max_seq_len()

source

Returns the default maximum sequence length for the model.

Subclasses should determine whether this value can be overridden by setting the --max-length (pipeline_config.model.max_length) flag.

Return type:

int

initialize()

classmethod initialize(pipeline_config, model_config=None)

source

Initialize the config from a PipelineConfig.

Parameters:

• pipeline_config (PipelineConfig) – The pipeline configuration.
• model_config (MAXModelConfig | None) – The model configuration to read from. When None (the default), pipeline_config.model is used. Pass an explicit config (e.g. pipeline_config.draft_model) to initialize the arch config for a different model.

Return type:

Self

kv_params

kv_params: MultiKVCacheParams

source

max_seq_len

max_seq_len: int

source

quant_config

quant_config: QuantConfig | None = None

source

Set by finalize() when the routed experts are packed FP4.

text_config

text_config: InklingTextConfig

source

use_subgraphs

use_subgraphs: bool = True

source

InklingInputs

class max.pipelines.architectures.inkling.InklingInputs(tokens, input_row_offsets, positions, return_n_logits, signal_buffers, slot_idx, conv_pools, *, kv_cache_inputs=None, lora_buffers=(), vision_embeddings=<factory>, vision_scatter_indices=<factory>, hidden_states=None)

source

Bases: ModelInputs

Ragged token inputs plus the convolution-state pool addressing.

Parameters:

• tokens (Buffer)
• input_row_offsets (Buffer)
• positions (Buffer)
• return_n_logits (Buffer)
• signal_buffers (list[Buffer])
• slot_idx (list[Buffer])
• conv_pools (list[Buffer])
• kv_cache_inputs (KVCacheInputsInterface[Buffer, Buffer] | None)
• lora_buffers (tuple[Buffer, ...])
• vision_embeddings (list[Buffer])
• vision_scatter_indices (list[Buffer])
• hidden_states (Buffer | list[Buffer] | None)

buffers

property buffers: tuple[Buffer, ...]

source

Returns positional Buffer inputs for model ABI calls.

conv_pools

conv_pools: list[Buffer]

source

Per device, one pool per convolution site per layer, mutated in place.

input_row_offsets

input_row_offsets: Buffer

source

positions

positions: Buffer

source

return_n_logits

return_n_logits: Buffer

source

signal_buffers

signal_buffers: list[Buffer]

source

slot_idx

slot_idx: list[Buffer]

source

tokens

tokens: Buffer

source

InklingModel

class max.pipelines.architectures.inkling.InklingModel(pipeline_config, session, devices, kv_cache_config, weights, adapter=None, return_logits=ReturnLogits.LAST_TOKEN, return_hidden_states=ReturnHiddenStates.NONE, max_batch_size=1)

source

Bases: LogProbabilitiesMixin, GraphPipelineModelWithKVCache[TextContext], SupportsSSMStateWarmup

Pipeline model for Inkling’s text decoder.

Parameters:

• pipeline_config (PipelineConfig)
• session (InferenceSession)
• devices (list[Device])
• kv_cache_config (KVCacheConfig)
• weights (Weights)
• adapter (WeightsAdapter | None)
• return_logits (ReturnLogits)
• return_hidden_states (ReturnHiddenStates)
• max_batch_size (int)

batch_processor_cls

batch_processor_cls

source

alias of InklingBatchProcessor

emits_folded_sampled_tokens

property emits_folded_sampled_tokens: bool

source

Whether the forward graph appends a folded greedy-token output.

Architectures that fold the sampler (argmax) into the forward graph emit the sampled-token buffer as a trailing graph output and override this to return True. Callers must peel that trailing output into ModelOutputs.sampled_tokens only when this is True; otherwise the fold_sampler_into_graph runtime flag is a no-op for the architecture.

execute()

execute(model_inputs)

source

Executes the graph with the given inputs.

Parameters:

model_inputs (ModelInputs) – The model inputs to execute, containing tensors and any other required data for model execution.

Returns:

ModelOutputs containing the pipeline’s output tensors.

Return type:

ModelOutputs

This is an abstract method that must be implemented by concrete PipelineModels to define their specific execution logic.

model

model: Model

source

model_config_cls

model_config_cls

source

alias of InklingConfig

release()

release(request_id)

source

Drops the request’s convolution state, freeing its slot.

Parameters:

request_id (RequestID)

Return type:

None

release_warmup_state()

release_warmup_state(request_ids)

source

Frees the slots a graph-capture warmup probe claimed.

Without this the second probe finds no free slot and serving never starts; claim zeros a slot when a real request takes it.

Parameters:

request_ids (list[RequestID])

Return type:

None

state_dict

state_dict: dict[str, Any]

source