Modular Inference Engine

The world’s fastest unified inference engine, supercharging any model from TensorFlow or PyTorch on a wide range of hardware.

The Modular Inference Engine can help simplify your workflow and reduce your inference latency so you can scale your AI products.

We’ve incorporated best-in-class compiler and runtime technologies to create the world’s fastest unified inference engine. It supercharges all models from TensorFlow and PyTorch, and runs on a wide variety of hardware backends.

Requires no changes to your model.
Delivers high performance on a wide range of hardware.
Just load your model, execute, and watch latency drop.

Below, you can preview our APIs for Python and C, and our C++ API is coming soon!

Talk to us on Discord

Python API docs

C API docs

Get started in C

A walkthrough of the C API, showing how to load and run a trained model.

Inference Engine C API

The Inference Engine C API reference.

Server integration docs

Server integration overview

An introduction to using the Inference Engine as a backend for Triton and TF Serving.

Triton serving demo

A Jupyter notebook that compares the performance of Triton with TF to Triton with Modular.