Python module

max.pipelines.architectures.idefics3

Idefics3 vision-language architecture for multimodal text generation.

Idefics3Config

class max.pipelines.architectures.idefics3.Idefics3Config(*, devices, scale_factor, image_token_id, vision_config, text_config)

source

Bases: ArchConfigWithKVCache

Configuration for Idefics3 models.

Parameters:

• devices (list[DeviceRef])
• scale_factor (int)
• image_token_id (int)
• vision_config (Idefics3VisionConfig)
• text_config (Llama3Config)

calculate_max_seq_len()

static calculate_max_seq_len(pipeline_config, huggingface_config)

source

Calculate maximum sequence length for Idefics3.

Parameters:

• pipeline_config (PipelineConfig)
• huggingface_config (AutoConfig)

Return type:

int

construct_kv_params()

static construct_kv_params(huggingface_config, pipeline_config, devices, kv_cache_config, cache_dtype)

source

Get KV cache parameters for the language model.

Parameters:

• huggingface_config (AutoConfig)
• pipeline_config (PipelineConfig)
• devices (list[DeviceRef])
• kv_cache_config (KVCacheConfig)
• cache_dtype (DType)

Return type:

KVCacheParams

devices

devices: list[DeviceRef]

source

Devices that the Idefics3 model is parallelized over.

finalize()

finalize(huggingface_config, llm_state_dict, return_logits, norm_method='rms_norm')

source

Finalize the Idefics3Config instance with state_dict dependent fields.

Parameters:

• huggingface_config (AutoConfig) – HuggingFace model configuration.
• llm_state_dict (dict[str, WeightData]) – Language model weights dictionary.
• dtype – Data type for model parameters.
• return_logits (ReturnLogits) – Return logits configuration.
• norm_method (Literal['rms_norm', 'layer_norm']) – Normalization method.

Return type:

None

get_kv_params()

get_kv_params()

source

Returns the KV cache parameters from the embedded text config.

Return type:

KVCacheParams

get_max_seq_len()

get_max_seq_len()

source

Returns the maximum sequence length from the embedded text config.

Return type:

int

get_num_layers()

static get_num_layers(huggingface_config)

source

Get number of layers in the language model.

Parameters:

huggingface_config (AutoConfig)

Return type:

int

image_seq_len

property image_seq_len: int

source

Calculate the number of image tokens after connector processing.

image_token_id

image_token_id: int

source

Token ID used to represent image tokens in the text sequence.

initialize()

classmethod initialize(pipeline_config, model_config=None)

source

Initializes an Idefics3Config instance from pipeline configuration.

Parameters:

• pipeline_config (PipelineConfig) – The MAX Engine pipeline configuration.
• model_config (MAXModelConfig | None)

Returns:

An Idefics3Config instance with fields initialized from config.

Return type:

Self

scale_factor

scale_factor: int

source

Scale factor for pixel shuffle operation in the connector.

text_config

text_config: Llama3Config

source

Text model configuration (Llama3-based).

vision_config

vision_config: Idefics3VisionConfig

source

Vision encoder configuration (SigLIP-based).

Idefics3Model

class max.pipelines.architectures.idefics3.Idefics3Model(pipeline_config, session, devices, kv_cache_config, weights, adapter=None, return_logits=ReturnLogits.LAST_TOKEN)

source

Bases: PipelineModelWithKVCache[TextAndVisionContext]

An Idefics3 pipeline model for multimodal text generation.

Parameters:

• pipeline_config (PipelineConfig)
• session (InferenceSession)
• devices (list[Device])
• kv_cache_config (KVCacheConfig)
• weights (Weights)
• adapter (WeightsAdapter | None)
• return_logits (ReturnLogits)

calculate_max_seq_len()

static calculate_max_seq_len(pipeline_config, huggingface_config)

source

Calculates the maximum sequence length for the Idefics3 model.

Parameters:

• pipeline_config (PipelineConfig)
• huggingface_config (AutoConfig)

Return type:

int

execute()

execute(model_inputs)

source

Executes the Idefics3 model with the prepared inputs.

Parameters:

model_inputs (ModelInputs)

Return type:

ModelOutputs

get_kv_params()

classmethod get_kv_params(huggingface_config, pipeline_config, devices, kv_cache_config, cache_dtype)

source

Gets the parameters required to configure the KV cache for Idefics3.

Parameters:

• huggingface_config (AutoConfig)
• pipeline_config (PipelineConfig)
• devices (list[DeviceRef])
• kv_cache_config (KVCacheConfig)
• cache_dtype (DType)

Return type:

KVCacheParams

language_model

language_model: Model

source

The compiled language model for text generation.

load_model()

load_model(session)

source

Loads the compiled Idefics3 models into the MAX Engine session.

Returns:

A tuple of (vision_model, language_model).

Parameters:

session (InferenceSession)

Return type:

tuple[Model, Model]

prepare_initial_token_inputs()

prepare_initial_token_inputs(replica_batches, kv_cache_inputs=None, return_n_logits=1)

source

Prepares the initial inputs for the first execution pass of the Idefics3 model.

Parameters:

• replica_batches (Sequence[Sequence[TextAndVisionContext]])
• kv_cache_inputs (KVCacheInputs[Buffer, Buffer] | None)
• return_n_logits (int)

Return type:

ModelInputs

prepare_next_token_inputs()

prepare_next_token_inputs(next_tokens, prev_model_inputs)

source

Prepares the secondary inputs to be passed to execute().

While prepare_initial_token_inputs is responsible for managing the initial inputs. This function is responsible for updating the inputs, for each step in a multi-step execution pattern.

Parameters:

• next_tokens (Buffer)
• prev_model_inputs (ModelInputs)

Return type:

Idefics3Inputs

vision_model

vision_model: Model

source

The compiled vision model for processing images.

Was this page helpful?

Thank you! We'll create more content like this.

Thank you for helping us improve!

😔 What went wrong?

Some code doesn’t work

It includes inaccurate information

It's missing information I need

It was difficult to understand

Other

Submit