Text generation

Generate text from a prompt or conversation with the chat completions API. This API allows you to send text, images, and videos with your request. Learn about different ways to use the API throughout this page. For information on all request and response fields, see the chat completions API reference. For a list of models and their supported features and modalities, see Supported models.

This API supports the OpenAI Chat Completions interface.To try a Modular endpoint with an existing OpenAI SDK integration, set base_url to https://api.modular.com/v1 and api_key to your Modular API key.

Generate a text response

Generate text from a prompt or conversation history.

Python
curl

Send a chat completion request using the OpenAI Python SDK:

generate-text.py

from openai import OpenAI

client = OpenAI(
    base_url="https://api.modular.com/v1",
    api_key="<your-api-key>",  # Load your key
)

response = client.chat.completions.create(
    model="google/gemma-4-31b-it",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Who won the world series in 2020?"},
        {"role": "assistant", "content": "The LA Dodgers won in 2020."},
        {"role": "user", "content": "Where was it played?"}
    ]
)
print(response.choices[0].message.content)

The response should be similar to:

The 2020 World Series was played at Globe Life Field in Arlington, Texas. It was a neutral site due to the COVID-19 pandemic.

Send the same request using curl:

curl -X POST https://api.modular.com/v1/chat/completions \
    -H "Authorization: Bearer <your-api-key>" \
    -H "Content-Type: application/json" \
    -d '{
        "model": "google/gemma-4-31b-it",
        "messages": [
            {
            "role": "system",
            "content": "You are a helpful assistant."
            },
            {
            "role": "user",
            "content": "Hello, how are you?"
            }
        ],
        "max_tokens": 100
    }'

The response should be similar to:

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "I'm doing well, thank you for asking. How can I assist you today?",
        "role": "assistant"
      }
    }
  ],
  "model": "google/gemma-4-31b-it",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 17,
    "prompt_tokens": null,
    "total_tokens": 17
  }
}

Stream a text response

Set stream=True to receive tokens as they are generated instead of waiting for the full response.

Python
curl

Set stream=True and iterate over the response chunks as they arrive:

stream-text.py

from openai import OpenAI

client = OpenAI(
    base_url="https://api.modular.com/v1",
    api_key="<your-api-key>",  # Load your key
)

stream = client.chat.completions.create(
    model="google/gemma-4-31b-it",
    messages=[{"role": "user", "content": "Write a short poem about the sea."}],
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content is not None:
        print(chunk.choices[0].delta.content, end="", flush=True)
print()

Pass "stream": true in the request body to receive server-sent events:

curl -X POST https://api.modular.com/v1/chat/completions \
    -H "Authorization: Bearer <your-api-key>" \
    -H "Content-Type: application/json" \
    -d '{
        "model": "google/gemma-4-31b-it",
        "messages": [{"role": "user", "content": "Write a short poem about the sea."}],
        "stream": true
    }'

Each streamed chunk is a server-sent event with a data: prefix:

data: {"choices":[{"delta":{"content":"The"},"index":0}],"object":"chat.completion.chunk"}
data: {"choices":[{"delta":{"content":" sea"},"index":0}],"object":"chat.completion.chunk"}
...
data: [DONE]

Analyze an image

Provide an image and a text prompt to generate a description of or extract information from visual content. Include the image in the messages array using an image_url content item. The url field accepts either a publicly accessible URL or a base64-encoded data URI (for example, data:image/jpeg;base64,...). The following example asks the model to describe the contents of an image.

Python
curl

Include an image_url content block alongside your text prompt:

generate-image-description.py

from openai import OpenAI

client = OpenAI(
    base_url="https://api.modular.com/v1",
    api_key="<your-api-key>",  # Load your key
)

response = client.chat.completions.create(
    model="google/gemma-4-31b-it",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "What is in this image?"
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg"
                    }
                }
            ]
        }
    ],
    max_tokens=300
)

print(response.choices[0].message.content)

The response should be similar to:

Here's a breakdown of what's in the image:

*   **Peter Rabbit:** The main focus is a realistic-looking depiction of Peter
Rabbit, the character from Beatrix Potter's stories...

Pass the image URL and text prompt in the content array of the request body:

curl -X POST https://api.modular.com/v1/chat/completions \
-H "Authorization: Bearer <your-api-key>" \
-H "Content-Type: application/json" \
-d '{
  "model": "google/gemma-4-31b-it",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "What is in this image?"
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg"
          }
        }
      ]
    }
  ],
  "max_tokens": 300
}' | grep -o '"content":"[^"]*"' | sed 's/"content":"//g' | sed 's/"//g' | tr -d '\n' | sed 's/\\n/\n/g'

The response should be similar to:

Here's a breakdown of what's in the image:

*   **Peter Rabbit:** The main focus is a realistic, anthropomorphic
(human-like) rabbit character...

Analyze a video

Provide a video and a text prompt to generate a natural-language description or analysis of the video’s visual content. Include the video in the messages array using a video_url content item. The url field accepts either a publicly accessible URL or a base64-encoded data URI (for example, data:video/mp4;base64,...). The following example asks the model to describe the contents of a video.

Python
curl

Pass a video URL and a text question to get a natural-language description of the video:

generate-video-description.py

from openai import OpenAI

client = OpenAI(
    base_url="https://api.modular.com/v1",
    api_key="<your-api-key>",  # Load your key
)

completion = client.chat.completions.create(
    model="google/gemma-4-31b-it",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Describe what is happening in this video"
                },
                {
                    "type": "video_url",
                    "video_url": {
                        "url": "https://avtshare01.rz.tu-ilmenau.de/avt-vqdb-uhd-1/test_1/segments/bigbuck_bunny_8bit_15000kbps_1080p_60.0fps_h264.mp4"
                    }
                }
            ]
        }
    ],
    max_tokens=300
)

print(completion.choices[0].message.content)

The response should be similar to:

The video is an animated short film featuring a large, fluffy rabbit in a
colorful meadow. The rabbit wanders through the environment, encountering
butterflies and small birds. The animation has a warm, lighthearted tone with
vibrant natural scenery...

Pass the video URL and text prompt in the content array of the request body:

curl -X POST https://api.modular.com/v1/chat/completions \
-H "Authorization: Bearer <your-api-key>" \
-H "Content-Type: application/json" \
-d '{
  "model": "google/gemma-4-31b-it",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "Describe what is happening in this video"
        },
        {
          "type": "video_url",
          "video_url": {
            "url": "https://avtshare01.rz.tu-ilmenau.de/avt-vqdb-uhd-1/test_1/segments/bigbuck_bunny_8bit_15000kbps_1080p_60.0fps_h264.mp4"
          }
        }
      ]
    }
  ],
  "max_tokens": 300
}' | grep -o '"content":"[^"]*"' | sed 's/"content":"//g' | sed 's/"//g' | tr -d '\n' | sed 's/\\n/\n/g'

The response should be similar to:

The video is an animated short film featuring a large, fluffy rabbit in a
colorful meadow. The rabbit wanders through the environment, encountering
butterflies and small birds. The animation has a warm, lighthearted tone with
vibrant natural scenery...

Get started

Inference

Administration

Generate a text response

Stream a text response

Analyze an image

Analyze a video

​Generate a text response

​Stream a text response

​Analyze an image

​Analyze a video

Generate a text response

Stream a text response

Analyze an image

Analyze a video