> ## Documentation Index
> Fetch the complete documentation index at: https://docs.modular.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Create chat completion

> Generate a response from a conversation.

Use this API to send text, images, and/or videos and receive generated text back.
All supported API capabilities, such as function calling, are documented here.
The input and output modality support depends on the model you're using for inference.

This API supports the OpenAI Chat Completions interface. Use a Modular endpoint as the `base_url` and use a Modular API key when using the OpenAI SDK.


## OpenAPI

````yaml /reference/openapi-inference.yaml post /v1/chat/completions
openapi: 3.1.0
info:
  title: Modular Cloud Inference API
  version: 0.1.0
  description: >-
    REST API for Modular Cloud inference. All endpoints are served at
    `https://api.modular.com`.


    Run inference against hosted models. Different model types use different
    endpoints. See the [supported models](/models) page to check which endpoint
    each model uses.
servers:
  - url: https://api.modular.com
security:
  - BearerAuth: []
tags:
  - name: Inference
    description: Run inference against hosted models.
paths:
  /v1/chat/completions:
    post:
      tags:
        - Inference
      summary: Create chat completion
      description: Generate a response from a conversation.
      operationId: createChatCompletion
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/ChatCompletionRequest'
            examples:
              text-to-text:
                summary: Text input
                value:
                  model: google/gemma-4-26b-a4b-it
                  messages:
                    - role: user
                      content: What is the capital of France?
              image-to-text:
                summary: Image input
                value:
                  model: google/gemma-4-26b-a4b-it
                  messages:
                    - role: user
                      content:
                        - type: image_url
                          image_url:
                            url: https://example.com/image.jpg
                        - type: text
                          text: Describe this image.
              video-to-text:
                summary: Video input
                value:
                  model: google/gemma-4-26b-a4b-it
                  messages:
                    - role: user
                      content:
                        - type: video_url
                          video_url:
                            url: https://example.com/video.mp4
                        - type: text
                          text: Describe what happens in this video.
              function-calling:
                summary: Function calling
                value:
                  model: google/gemma-4-31b-it
                  messages:
                    - role: user
                      content: What's the weather like in San Francisco today?
                  tools:
                    - type: function
                      function:
                        name: get_weather
                        description: Get current temperature for a given location.
                        parameters:
                          type: object
                          properties:
                            location:
                              type: string
                              description: City and country e.g. Bogotá, Colombia
                          required:
                            - location
                          additionalProperties: false
                        strict: true
                  tool_choice: auto
      responses:
        '200':
          description: Chat completion response.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ChatCompletionResponse'
              examples:
                text:
                  summary: Text response
                  value:
                    id: chatcmpl-abc123
                    object: chat.completion
                    model: google/gemma-4-26b-a4b-it
                    choices:
                      - index: 0
                        message:
                          role: assistant
                          content: The capital of France is Paris.
                        finish_reason: stop
                    usage:
                      prompt_tokens: 12
                      completion_tokens: 9
                      total_tokens: 21
                tool-calls:
                  summary: Tool call response
                  value:
                    id: chatcmpl-abc123
                    object: chat.completion
                    model: google/gemma-4-31b-it
                    choices:
                      - index: 0
                        message:
                          role: assistant
                          content: ''
                          tool_calls:
                            - id: call_ac73df14fe184349
                              type: function
                              function:
                                name: get_weather
                                arguments: '{"location": "San Francisco, USA"}'
                        finish_reason: tool_calls
                    usage:
                      prompt_tokens: 45
                      completion_tokens: 18
                      total_tokens: 63
      x-codeSamples:
        - lang: Python
          label: Text → text
          source: |-
            from openai import OpenAI

            client = OpenAI(
                base_url="https://api.modular.com/v1",
                api_key="<your-api-key>",  # Load your key
            )

            response = client.chat.completions.create(
                model="google/gemma-4-26b-a4b-it",
                messages=[{"role": "user", "content": "What is the capital of France?"}],
            )

            print(response.choices[0].message.content)
        - lang: Python
          label: Function calling
          source: |-
            from openai import OpenAI

            client = OpenAI(
                base_url="https://api.modular.com/v1",
                api_key="<your-api-key>",  # Load your key
            )

            tools = [{
                "type": "function",
                "function": {
                    "name": "get_weather",
                    "description": "Get current temperature for a given location.",
                    "parameters": {
                        "type": "object",
                        "properties": {
                            "location": {"type": "string", "description": "City and country e.g. Bogotá, Colombia"}
                        },
                        "required": ["location"],
                        "additionalProperties": False
                    },
                    "strict": True
                }
            }]

            response = client.chat.completions.create(
                model="google/gemma-4-31b-it",
                messages=[{"role": "user", "content": "What's the weather like in San Francisco today?"}],
                tools=tools,
            )

            print(response.choices[0].message.tool_calls)
        - lang: Python
          label: Image → text
          source: |-
            from openai import OpenAI

            client = OpenAI(
                base_url="https://api.modular.com/v1",
                api_key="<your-api-key>",  # Load your key
            )

            response = client.chat.completions.create(
                model="google/gemma-4-26b-a4b-it",
                messages=[
                    {
                        "role": "user",
                        "content": [
                            {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}},
                            {"type": "text", "text": "Describe this image."},
                        ],
                    }
                ],
            )

            print(response.choices[0].message.content)
        - lang: Python
          label: Video → text
          source: |-
            from openai import OpenAI

            client = OpenAI(
                base_url="https://api.modular.com/v1",
                api_key="<your-api-key>",  # Load your key
            )

            response = client.chat.completions.create(
                model="google/gemma-4-26b-a4b-it",
                messages=[
                    {
                        "role": "user",
                        "content": [
                            {"type": "video_url", "video_url": {"url": "https://example.com/video.mp4"}},
                            {"type": "text", "text": "Describe what happens in this video."},
                        ],
                    }
                ],
            )

            print(response.choices[0].message.content)
        - lang: Shell
          label: Text → text
          source: |-
            curl -X POST https://api.modular.com/v1/chat/completions \
              -H "Authorization: Bearer <your-api-key>" \
              -H "Content-Type: application/json" \
              -d '{
                "model": "google/gemma-4-26b-a4b-it",
                "messages": [{"role": "user", "content": "What is the capital of France?"}]
              }'
        - lang: Shell
          label: Image → text
          source: |-
            curl -X POST https://api.modular.com/v1/chat/completions \
              -H "Authorization: Bearer <your-api-key>" \
              -H "Content-Type: application/json" \
              -d '{
                "model": "google/gemma-4-26b-a4b-it",
                "messages": [
                  {
                    "role": "user",
                    "content": [
                      {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}},
                      {"type": "text", "text": "Describe this image."}
                    ]
                  }
                ]
              }'
        - lang: Shell
          label: Video → text
          source: |-
            curl -X POST https://api.modular.com/v1/chat/completions \
              -H "Authorization: Bearer <your-api-key>" \
              -H "Content-Type: application/json" \
              -d '{
                "model": "google/gemma-4-26b-a4b-it",
                "messages": [
                  {
                    "role": "user",
                    "content": [
                      {"type": "video_url", "video_url": {"url": "https://example.com/video.mp4"}},
                      {"type": "text", "text": "Describe what happens in this video."}
                    ]
                  }
                ]
              }'
        - lang: Shell
          label: Function calling
          source: |-
            curl -X POST https://api.modular.com/v1/chat/completions \
              -H "Authorization: Bearer <your-api-key>" \
              -H "Content-Type: application/json" \
              -d '{
                "model": "google/gemma-4-31b-it",
                "messages": [
                  {"role": "user", "content": "What is the weather like in Boston today?"}
                ],
                "tools": [
                  {
                    "type": "function",
                    "function": {
                      "name": "get_weather",
                      "description": "Get the current weather in a given location",
                      "parameters": {
                        "type": "object",
                        "properties": {
                          "location": {"type": "string", "description": "The city and state, e.g. Los Angeles, CA"}
                        },
                        "required": ["location"]
                      }
                    }
                  }
                ],
                "tool_choice": "auto"
              }'
components:
  schemas:
    ChatCompletionRequest:
      type: object
      properties:
        model:
          type: string
          description: Model identifier. See the [supported models](/models) page.
          example: google/gemma-4-26b-a4b-it
        messages:
          type: array
          items:
            $ref: '#/components/schemas/ChatMessage'
          description: The conversation history.
        max_tokens:
          type: integer
          description: Maximum number of tokens to generate.
        temperature:
          type: number
          description: >-
            Sampling temperature (0–2). Higher values produce more varied
            output.
          minimum: 0
          maximum: 2
        stream:
          type: boolean
          description: >-
            If true, stream partial tokens as server-sent events. Support for
            `tools` while streaming is model-dependent; set this to false if you
            see incomplete or malformed tool-call output.
        tools:
          type: array
          description: >-
            List of functions the model may call. See the [supported
            models](/models) page for a list of models that support tool use
            (function calling).
          items:
            $ref: '#/components/schemas/ChatCompletionTool'
        tool_choice:
          description: >-
            Controls when and how the model calls a tool.


            - `none`: Disables tool calls.

            - `auto` (default): Lets the model decide whether to call a tool.

            - `required`: Forces the model to call at least one tool.

            - A function object: Forces the model to call the named function.


            The following example requires the model to call the `get_weather`
            function:


            ```json

            "tool_choice": {
              "type": "function",
              "function": {"name": "get_weather"}
            }

            ```
          oneOf:
            - type: string
              enum:
                - none
                - auto
                - required
            - type: object
              properties:
                type:
                  type: string
                  enum:
                    - function
                function:
                  type: object
                  properties:
                    name:
                      type: string
                      description: Name of the function the model must call.
                  required:
                    - name
              required:
                - type
                - function
      required:
        - model
        - messages
    ChatCompletionResponse:
      type: object
      properties:
        id:
          type: string
          description: The unique identifier for the chat completion.
        object:
          type: string
          description: The object type, which is always `chat.completion`.
        model:
          type: string
          description: The model that generated the completion.
        choices:
          type: array
          description: The generated chat completion choices.
          items:
            $ref: '#/components/schemas/ChatCompletionChoice'
        usage:
          $ref: '#/components/schemas/CompletionUsage'
          description: Token usage for the request.
      required:
        - id
        - object
        - model
        - choices
    ChatMessage:
      type: object
      properties:
        role:
          type: string
          enum:
            - system
            - developer
            - user
            - assistant
            - tool
          description: >-
            Identifies the source and purpose of a message so the model can
            distinguish instructions, user input, previous responses, and tool
            results.

            Supported roles depend on the model's chat template.


            - `system`: Provides instructions that guide model behavior
            throughout the conversation.

            - `developer`: Newer alias for `system`.

            - `user`: Provides user input or a request.

            - `assistant`: Provides a previous model response or requested tool
            calls.

            - `tool`: Provides the result of a function call. Use `tool_call_id`
            to associate the result with its call.
        content:
          oneOf:
            - type: string
              description: >-
                Plain text content for text-only messages, or a JSON-encoded
                result string for `tool` messages.
            - type: array
              description: >-
                Multimodal content array for messages that include images or
                video.
              items:
                $ref: '#/components/schemas/ChatMessageContentPart'
            - type: 'null'
              description: >-
                Null when an `assistant` message only contains `tool_calls` and
                no text content.
          description: >-
            Message content. Pass a string for text-only inputs, an array of
            content blocks for image or video inputs, or null for an assistant
            message that only contains tool calls.
        tool_calls:
          type: array
          description: >-
            Tool calls requested by the model. Present on `assistant` messages
            when `finish_reason` is `tool_calls`.
          items:
            $ref: '#/components/schemas/ChatCompletionMessageToolCall'
        tool_call_id:
          type: string
          description: >-
            ID of the tool call this message answers. Required on `tool` role
            messages and must match the `id` of a tool call in the preceding
            assistant message.
      required:
        - role
        - content
    ChatCompletionTool:
      type: object
      properties:
        type:
          type: string
          enum:
            - function
          description: Currently always `function`.
        function:
          $ref: '#/components/schemas/ChatCompletionToolFunction'
      required:
        - type
        - function
    ChatCompletionChoice:
      type: object
      properties:
        index:
          type: integer
          description: The zero-based index of this choice in `choices`.
        message:
          $ref: '#/components/schemas/ChatCompletionResponseMessage'
          description: The message generated by the model.
        finish_reason:
          type: string
          enum:
            - stop
            - length
            - tool_calls
          description: |-
            The reason the model stopped generating output.

            - `stop`: The model reached a natural stopping point.
            - `length`: Generation reached the maximum token limit.
            - `tool_calls`: The model requested one or more tool calls.
      required:
        - index
        - message
        - finish_reason
    CompletionUsage:
      type: object
      properties:
        prompt_tokens:
          type: integer
          description: The number of tokens in the input messages.
        completion_tokens:
          type: integer
          description: The number of tokens in the generated response.
        total_tokens:
          type: integer
          description: The total number of input and generated tokens.
    ChatMessageContentPart:
      type: object
      description: A single content block within a multimodal message.
      properties:
        type:
          type: string
          enum:
            - text
            - image_url
            - video_url
          description: Content block type.
        text:
          type: string
          description: Text content. Required when `type` is `text`.
        image_url:
          type: object
          description: Image source. Required when `type` is `image_url`.
          properties:
            url:
              type: string
              description: >-
                URL or base64 data URI of the image
                (`data:<mime-type>;base64,<data>`).
          required:
            - url
        video_url:
          type: object
          description: Video source. Required when `type` is `video_url`.
          properties:
            url:
              type: string
              description: URL of the video file.
          required:
            - url
      required:
        - type
    ChatCompletionMessageToolCall:
      type: object
      description: A single tool call requested by the model.
      properties:
        id:
          type: string
          description: >-
            Unique identifier for this tool call. Echo it back in the matching
            `tool` message's `tool_call_id`.
        type:
          type: string
          enum:
            - function
          description: Currently always `function`.
        function:
          type: object
          properties:
            name:
              type: string
              description: >-
                Name of the function to call, matching a `tools[].function.name`
                from the request.
            arguments:
              type: string
              description: >-
                JSON-encoded string of arguments to pass to the function. Check
                for malformed JSON, which indicates the model failed to produce
                valid tool-call output.
          required:
            - name
            - arguments
      required:
        - id
        - type
        - function
    ChatCompletionToolFunction:
      type: object
      description: >-
        Definition of a callable function, following the OpenAI function calling
        specification.
      properties:
        name:
          type: string
          description: >-
            Function name the model uses to call it. At most 64 characters;
            default character set is `[a-zA-Z0-9_-]`.
        description:
          type: string
          description: >-
            Description of what the function does, used by the model to decide
            when and how to call it.
        parameters:
          type: object
          description: JSON Schema object describing the function's parameters.
        strict:
          type: boolean
          description: >-
            If true, the model's arguments are constrained to strictly match the
            `parameters` schema.
      required:
        - name
    ChatCompletionResponseMessage:
      type: object
      description: A message generated by the model.
      properties:
        role:
          type: string
          enum:
            - assistant
          description: The message author. Always `assistant`.
        content:
          type: string
          description: >-
            The text generated by the model. An empty string if the model
            requests only tool calls.
        tool_calls:
          type:
            - array
            - 'null'
          description: >-
            The tool calls that the model requests. Null when the model doesn't
            request any tool calls.
          items:
            $ref: '#/components/schemas/ChatCompletionMessageToolCall'
        refusal:
          type: string
          description: >-
            The refusal message that the model generates. An empty string when
            the model doesn't refuse the request.
        reasoning:
          type:
            - string
            - 'null'
          description: >-
            The chain-of-thought text that a reasoning model generates. Null for
            models that don't reason. Null for models that return this text in
            `reasoning_content` instead.
        reasoning_content:
          type:
            - string
            - 'null'
          description: >-
            An alternative field for chain-of-thought text. Null for models that
            don't reason. Null for models that return this text in `reasoning`
            instead.
      required:
        - role
        - content
  securitySchemes:
    BearerAuth:
      type: http
      scheme: bearer
      description: >-
        Modular Cloud API key. Obtain from the [API keys
        page](https://console.modular.com/api_tokens).

````