Start a chat endpoint

Modular Team

max

llama3

openai

The MAX framework simplifies the process to serve open source models with the same API interface as OpenAI. This allows you to replace commercial models with alternatives from the MAX Builds site with minimal code changes.

This tutorial shows you how to serve Llama 3.1 locally with the max CLI and interact with it through REST and Python APIs. You'll learn to configure the server and make requests using the OpenAI client libraries as a drop-in replacement.

System requirements:

Mac

Linux

WSL

Set up your environment

Create a Python project to install our APIs and CLI tools:

pixi

1. If you don't have it, install pixi:

   curl -fsSL https://pixi.sh/install.sh | sh

   curl -fsSL https://pixi.sh/install.sh | sh

   Then restart your terminal for the changes to take effect.

2. Create a project:

   pixi init chat-tutorial \
     -c https://conda.modular.com/max-nightly/ -c conda-forge \
     && cd chat-tutorial

   pixi init chat-tutorial \
     -c https://conda.modular.com/max-nightly/ -c conda-forge \
     && cd chat-tutorial
3. Install the modular conda package:
   Nightly

   pixi add modular

   pixi add modular

   Stable

   pixi add "modular=25.4"

   pixi add "modular=25.4"
4. Start the virtual environment:

   pixi shell

   pixi shell

uv

1. If you don't have it, install uv:

   curl -LsSf https://astral.sh/uv/install.sh | sh

   curl -LsSf https://astral.sh/uv/install.sh | sh

   Then restart your terminal to make uv accessible.

2. Create a project:

   uv init chat-tutorial && cd chat-tutorial

   uv init chat-tutorial && cd chat-tutorial
3. Create and start a virtual environment:

   uv venv && source .venv/bin/activate

   uv venv && source .venv/bin/activate
4. Install the modular Python package:
   Nightly

   uv pip install modular \
     --extra-index-url https://download.pytorch.org/whl/cpu \
     --index-url https://dl.modular.com/public/nightly/python/simple/ \
     --index-strategy unsafe-best-match --prerelease allow

   uv pip install modular \
     --extra-index-url https://download.pytorch.org/whl/cpu \
     --index-url https://dl.modular.com/public/nightly/python/simple/ \
     --index-strategy unsafe-best-match --prerelease allow

   Stable

   uv pip install modular \
     --extra-index-url https://download.pytorch.org/whl/cpu \
     --extra-index-url https://modular.gateway.scarf.sh/simple/ \
     --index-strategy unsafe-best-match

   uv pip install modular \
     --extra-index-url https://download.pytorch.org/whl/cpu \
     --extra-index-url https://modular.gateway.scarf.sh/simple/ \
     --index-strategy unsafe-best-match

pip

1. Create a project folder:

   mkdir chat-tutorial && cd chat-tutorial

   mkdir chat-tutorial && cd chat-tutorial
2. Create and activate a virtual environment:

   python3 -m venv .venv/chat-tutorial \
     && source .venv/chat-tutorial/bin/activate

   python3 -m venv .venv/chat-tutorial \
     && source .venv/chat-tutorial/bin/activate
3. Install the modular Python package:
   Nightly

   pip install --pre modular \
     --extra-index-url https://download.pytorch.org/whl/cpu \
     --index-url https://dl.modular.com/public/nightly/python/simple/

   pip install --pre modular \
     --extra-index-url https://download.pytorch.org/whl/cpu \
     --index-url https://dl.modular.com/public/nightly/python/simple/

   Stable

   pip install modular \
     --extra-index-url https://download.pytorch.org/whl/cpu \
     --extra-index-url https://modular.gateway.scarf.sh/simple/

   pip install modular \
     --extra-index-url https://download.pytorch.org/whl/cpu \
     --extra-index-url https://modular.gateway.scarf.sh/simple/

conda

1. If you don't have it, install conda. A common choice is with brew:

   brew install miniconda

   brew install miniconda
2. Initialize conda for shell interaction:

   conda init

   conda init

   If you're on a Mac, instead use:

   conda init zsh

   conda init zsh

   Then restart your terminal for the changes to take effect.

3. Create a project:

   conda create -n chat-tutorial

   conda create -n chat-tutorial
4. Start the virtual environment:

   conda activate chat-tutorial

   conda activate chat-tutorial
5. Install the modular conda package:
   Nightly

   conda install -c conda-forge -c https://conda.modular.com/max-nightly/ modular

   conda install -c conda-forge -c https://conda.modular.com/max-nightly/ modular

   Stable

   conda install -c conda-forge -c https://conda.modular.com/max/ modular

   conda install -c conda-forge -c https://conda.modular.com/max/ modular

Serve your model

Use the max serve command to start a local model server with the Llama 3.1 model:

max serve \
  --model-path modularai/Llama-3.1-8B-Instruct-GGUF

max serve \
  --model-path modularai/Llama-3.1-8B-Instruct-GGUF

This will create a server running the Llama-3.1-8B-Instruct-GGUF large language model on http://localhost:8000/v1/chat/completions, an OpenAI compatible endpoint.

While this example uses the Llama 3.1 model, you can replace it with any of the models listed in the MAX Builds site.

When searching for a model using the MAX Builds site, ensure that the model type can fit into memory of your machine. You can filter and sort models by hardware type, and size of the model. For more information and to learn how to use the MAX Builds site, see MAX Builds in 60 seconds.

The endpoint is ready when you see this message printed in your terminal:

Server ready on http://0.0.0.0:8000 (Press CTRL+C to quit)

Server ready on http://0.0.0.0:8000 (Press CTRL+C to quit)

For a complete list of max CLI commands and options, refer to the MAX CLI reference.

Interact with your model

MAX supports OpenAI's REST APIs and you can interact with the model using either the OpenAI Python SDK or curl:

Python

You can use OpenAI's Python client to interact with the model.

To interact with MAX's OpenAI-compatible endpoints, install the OpenAI Python SDK:

pixi

pixi add openai

pixi add openai

uv

uv add openai

uv add openai

pip

pip install openai

pip install openai

conda

conda install openai

conda install openai

Then, create a client and make a request to the model:

generate-text.py

from openai import OpenAI

client = OpenAI(
    base_url = 'http://0.0.0.0:8000/v1',
    api_key='EMPTY', # required by the API, but not used by MAX
)

response = client.chat.completions.create(
  model="modularai/Llama-3.1-8B-Instruct-GGUF",
  messages=[
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Who won the world series in 2020?"},
    {"role": "assistant", "content": "The LA Dodgers won in 2020."},
    {"role": "user", "content": "Where was it played?"}
  ]
)
print(response.choices[0].message.content)

from openai import OpenAI

client = OpenAI(
    base_url = 'http://0.0.0.0:8000/v1',
    api_key='EMPTY', # required by the API, but not used by MAX
)

response = client.chat.completions.create(
  model="modularai/Llama-3.1-8B-Instruct-GGUF",
  messages=[
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Who won the world series in 2020?"},
    {"role": "assistant", "content": "The LA Dodgers won in 2020."},
    {"role": "user", "content": "Where was it played?"}
  ]
)
print(response.choices[0].message.content)

In this example, you're using the OpenAI Python client to interact with the MAX endpoint running on local host 8000. The client object is initialized with the base URL http://0.0.0.0:8000/v1 and the API key is ignored.

When you run this code, the model should respond with information about the 2020 World Series location:

python generate-text.py

python generate-text.py

The 2020 World Series was played at Globe Life Field in Arlington, Texas. It was a neutral site due to the COVID-19 pandemic.

The 2020 World Series was played at Globe Life Field in Arlington, Texas. It was a neutral site due to the COVID-19 pandemic.

cURL

The following curl command sends a chat request to the model's chat completions endpoint:

curl http://0.0.0.0:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "modularai/Llama-3.1-8B-Instruct-GGUF",
        "messages": [
            {
            "role": "system",
            "content": "You are a helpful assistant."
            },
            {
            "role": "user",
            "content": "Hello, how are you?"
            }
        ],
        "max_tokens": 100
    }'

curl http://0.0.0.0:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "modularai/Llama-3.1-8B-Instruct-GGUF",
        "messages": [
            {
            "role": "system",
            "content": "You are a helpful assistant."
            },
            {
            "role": "user",
            "content": "Hello, how are you?"
            }
        ],
        "max_tokens": 100
    }'

You should receive a response similar to this:

{
  "id": "18b0abd2d2fd463ea43efe2c147bcac0",
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": " I'm doing well, thank you for asking. How can I assist you today?",
        "refusal": "",
        "tool_calls": null,
        "role": "assistant",
        "function_call": null
      },
      "logprobs": {
        "content": [],
        "refusal": []
      }
    }
  ],
  "created": 1743543698,
  "model": "modularai/Llama-3.1-8B-Instruct-GGUF",
  "service_tier": null,
  "system_fingerprint": null,
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 17,
    "prompt_tokens": null,
    "total_tokens": 17
  }
}

{
  "id": "18b0abd2d2fd463ea43efe2c147bcac0",
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": " I'm doing well, thank you for asking. How can I assist you today?",
        "refusal": "",
        "tool_calls": null,
        "role": "assistant",
        "function_call": null
      },
      "logprobs": {
        "content": [],
        "refusal": []
      }
    }
  ],
  "created": 1743543698,
  "model": "modularai/Llama-3.1-8B-Instruct-GGUF",
  "service_tier": null,
  "system_fingerprint": null,
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 17,
    "prompt_tokens": null,
    "total_tokens": 17
  }
}

For complete details on all available API endpoints and options, see the MAX Serve API documentation.

Next steps

Now that you have successfully set up MAX with an OpenAI-compatible chat endpoint, checkout out these other tutorials:

Did this tutorial work for you?

Thank you! We'll create more content like this.

Thank you for helping us improve!

😔 What went wrong?

Some code doesn’t work

It's missing a step; I don't know what to do

I don't understand the purpose

Other

Submit