Skip to main content
Max PlatformDocsCompanyBlogModCon
Max PlatformDocsCompanyBlogModCon
Contact salesSign in

struct

InferenceSession

Holds the context for MAX Engine in which you can load and run models.

For example, you can load a model like this:

var session = engine.InferenceSession()
var model = session.load("bert-base-uncased")

Implemented traits​

AnyType, Copyable, Movable

Methods​

__init__​

__init__(inout self: Self, options: SessionOptions = SessionOptions())

Creates a new inference session.

Args:

  • ​options (SessionOptions): Session options to configure how session is created.

load​

load(self: Self, path: Path, *, custom_ops_paths: List[Path] = List(), input_specs: Optional[List[InputSpec]] = #kgen.none) -> Model

Compile and initialize a model in MAX Engine, with the given model path and config.

Note: PyTorch models must be in TorchScript format.

If you're loading a TorchScript model, you must specify the input_specs argument with a list of InputSpec objects that specify the model's input specs (which may have dynamic shapes). For details, see how to specify input specs.

Args:

  • ​path (Path): Location of model in filesystem. You may pass a string here because the Path object supports implicit casting from a string.
  • ​custom_ops_paths (List[Path]): List of paths to Mojo custom op packages, to replace Modular kernels in models with user-defined kernels.
  • ​input_specs (Optional[List[InputSpec]]): Provide shapes and dtypes for model inputs. Required for TorchScript models, optional for other input formats.

Returns:

Initialized model ready for inference.

load(self: Self, graph: Graph, *, custom_ops_paths: List[Path] = List(), input_specs: Optional[List[InputSpec]] = #kgen.none) -> Model

Compile and initialize a model in MAX Engine, with the given Graph and config.

Args:

  • ​graph (Graph): MAX Graph.
  • ​custom_ops_paths (List[Path]): List of paths to Mojo custom op packages, to replace Modular kernels in models with user-defined kernels.
  • ​input_specs (Optional[List[InputSpec]]): Provide shapes and dtypes for model inputs. Required for TorchScript models, optional for other input formats.

Returns:

Initialized model ready for inference.

get_as_engine_tensor_spec​

get_as_engine_tensor_spec(self: Self, name: String, spec: TensorSpec) -> EngineTensorSpec

Gets a TensorSpec compatible with MAX Engine.

Args:

  • ​name (String): Name of the Tensor.
  • ​spec (TensorSpec): Tensor specification in Mojo TensorSpec format.

Returns:

EngineTensorSpec to be used with MAX Engine APIs.

get_as_engine_tensor_spec(self: Self, name: String, shape: Optional[List[Optional[SIMD[int64, 1]]]], dtype: DType) -> EngineTensorSpec

Gets a TensorSpec compatible with MAX Engine.

Args:

  • ​name (String): Name of the Tensor.
  • ​shape (Optional[List[Optional[SIMD[int64, 1]]]]): Shape of the Tensor. Dynamic Dimensions can be represented with None and for Dynamic Rank Tensor use None as value for shape.
  • ​dtype (DType): DataType of the Tensor.

Returns:

EngineTensorSpec to be used with MAX Engine APIs.

new_tensor_map​

new_tensor_map(self: Self) -> TensorMap

Gets a new TensorMap. This can be used to pass inputs to model.

Returns:

A new instance of TensorMap.

new_borrowed_tensor_value​

new_borrowed_tensor_value[type: DType](self: Self, tensor: Tensor[type]) -> Value

Create a new Value representing data borrowed from given tensor.

The user must ensure the tensor stays live through the lifetime of the value.

Parameters:

  • ​type (DType): Data type of the tensor to turn into a Value.

Args:

  • ​tensor (Tensor[type]): Tensor to borrow into a value.

Returns:

A value borrowing the tensor.

new_bool_value​

new_bool_value(self: Self, value: Bool) -> Value

Create a new Value representing a Bool.

Args:

  • ​value (Bool): Boolean to wrap into a value.

Returns:

Value representing the given boolean.

new_list_value​

new_list_value(self: Self) -> Value

Create a new Value representing an empty list.

Returns:

A new value containing an empty list.