# API keys Source: https://docs.modular.com/administration/api-keys Create and manage API keys to authenticate requests API keys authenticate requests to model endpoints. You can assign access scopes based on what a key needs to do and set an expiration date. [Create API keys in the console →](https://console.modular.com/api_keys) ## Choose an API key type You can create the following types of API keys: * **Personal keys** are associated with individual users. You can view and copy the key value only during key creation. * **Organization keys** are owned by a service account and are intended for automations. Anyone in the organization can view and copy the key value at any time. ## Choose access scopes Grant each API key only the access that it needs. You can select one or both of the following scopes: * **Model access** grants access to the [inference APIs](/api/inference/create-chat-completion), allowing the key to run inference on any model available to your organization. * **Developer access** grants access to the deployment and management APIs, allowing the key to access organization resources and manage infrastructure. Some operations such as running benchmarks also require your organization to have the corresponding features enabled. An API key's permissions are also limited by the [roles and permissions](/administration/roles-permissions) of its owner. A personal key inherits the permissions of the user who owns it. Service account keys always use the Developer role. ## Create an API key To create an API key: 1. Open the [API keys](https://console.modular.com/api_keys) page. 2. Next to **Personal keys** or **Organization keys**, click **+ Create**. 3. Enter the following information: * **Name**: Enter a short, descriptive name for the key. * **Description**: Optionally, describe how the key is used. * **Model access**: Enable this option for keys that call inference endpoints. * **Developer access**: Enable this option for keys that manage infrastructure programmatically. * **Expires**: Select when the key expires. You can't use the key after this date. 4. Click **Submit**. 5. If you created a personal key, copy the API key and store it in a secure location. You can't access the value of a personal key after you close the dialog. The API key has the following format: ```text theme={null} sk-mod-- ``` Store the key securely, such as in a secret manager. You can't regenerate an API key. To replace a key while preserving its name, delete the existing key and create a new key with the same name. # Billing Source: https://docs.modular.com/administration/billing Manage billing and learn how to update your plan Modular Cloud offers credit or contract-based billing plans. This page explains how to manage billing depending on your billing plan. [Set up billing in the console →](https://console.modular.com/billing) ## Compare plans There are two billing plans: | Plan | Billing method | Access | Rate limit tier | | ---------- | ---------------------------------- | ------------------------------ | --------------- | | Starter | Buy credits | Shared endpoints only | Tiers 1 and 2 | | Enterprise | Contract and usage-based invoicing | Shared and dedicated endpoints | Tiers 3 and 4 | You can't create dedicated endpoints on the Starter plan. To deploy models on dedicated infrastructure, you need an Enterprise plan. [Contact us](https://console.modular.com/signup) to learn more about Enterprise plans. For specific rate limits per tier, see [Rate limits](/administration/rate-limits). ## Use credits On the Starter plan, you must have credits to send inference requests. New Modular Cloud accounts include free credits, so you can start making requests right away. Each request deducts credits for input and output tokens. Token prices vary by model deployment—check prices for each model on the **Models** and **Endpoints** pages in the console. Prices shown are the cost per 1 million tokens used. Dedicated deployments are priced separately based on the configuration you choose. One credit equals one US dollar (USD). There's no separate conversion rate for credits, so your credit balance and all model prices are in the same units. ## Add credits If your credit balance reaches zero, requests that use your API keys return a 401 response. When you need to add credits, follow these steps: 1. Open the console [**Billing** page](https://console.modular.com/billing). 2. In the **Payment method** section, click **Add card**. 3. Complete the checkout form in Stripe. Stripe then returns you to the **Billing** page. 4. In the **Credit balance** section, click **Buy credits**. 5. In the **Amount (USD)** field, enter the amount you want to buy, then click **Buy credits**. Credits usually appear right away. If the payment takes longer to clear, the console confirms the purchase and adds the credits once it settles. ## Monitor billing The Billing page also lets you monitor billing activity for your organization. * **Current billing cycle**: monitor the state of the current billing cycle for all models. * **Past invoices**: review and export past invoices. * **Transaction history**: inspect and export individual transactions. # Manage users Source: https://docs.modular.com/administration/manage-users Add and remove members from your organization Only users with the `Owner` or `Admin` roles can access the **Members** page. [Manage users in the console →](https://console.modular.com/members) ## Add a new member To add a member to your organization: 1. On the [**Members** page](https://console.modular.com/members), click **+ Invite new member**. 2. In the dialog that appears, provide the following: * **First name** and **Last name** * **Email**: The email address the invitation will be sent to. * **Role**: `Admin`, `Developer`, or `User`. See [Roles and permissions](/administration/roles-permissions) for more information. The invited member will receive an email with instructions to join. Users are **automatically added to organizations** based on their login email domain. ## Update user information On the [**Members** page](https://console.modular.com/members), click **Update**. Here, you can edit a user's role. ## Remove a member On the [**Members** page](https://console.modular.com/members), click **Delete**. A user with the `Owner` role cannot remove themselves. Removing a member revokes or reassigns their resources according to your organization's policy. For example, their personal API keys become deactivated. # Rate limits Source: https://docs.modular.com/administration/rate-limits Learn when rate limits apply and how to increase your organization's limits Rate limits determine the maximum number of inference requests your organization can send in a given timeframe. They are measured in requests per minute (RPM) and tokens per minute (TPM). ## How rate limits work Rate limits are enforced at two levels: * **Tier limits** apply to all requests made by your organization across every model. Every request counts toward the same organization-wide budget. * **Model limits** apply independently to each model and are enforced per endpoint. Your request succeeds only if both limits have available capacity. If either limit is exceeded, the request fails with an HTTP 429 response. You'll have to wait before future requests run successfully. If this happens frequently, [contact us](https://console.modular.com/signup) to increase your limits. ## Tier limits The following table shows the rate limits for [Starter plan](/administration/billing) tiers. | Tier | Requests per minute (RPM) | Tokens per minute (TPM) | Qualification | | ------ | ------------------------- | ----------------------- | ------------------------------------------ | | Tier 1 | 60 | 600,000 | Sign up for Modular Cloud | | Tier 2 | 180 | 1,800,000 | Add a payment method for your organization | Enterprise plans include Tier 3 and Tier 4, which raise rate limits above the Starter plan tiers. [Contact us](https://console.modular.com/signup) when you're ready for more capacity. ## Model limits Some models have their own rate limit. The limit is enforced separately for each endpoint serving that model. * For **dedicated endpoints**, the model's rate limit is shared by all users in your organization. * For **shared endpoints**, the model's rate limit is shared by every organization using that endpoint. The following table lists models that have their own rate limit. | Model | Requests per minute (RPM) | Tokens per minute (TPM) | | ------------------------------ | ------------------------- | ----------------------- | | `black-forest-labs/FLUX.2-dev` | 100 | No limit | | `google/gemma-4-26b-a4b-it` | No limit | 1,080,000 | | `google/gemma-4-31b-it` | No limit | 2,160,000 | | `moonshotai/Kimi-K2.6` | No limit | 1,800,000 | | `moonshotai/kimi-k2.7-code` | No limit | 1,800,000 | # Roles and permissions Source: https://docs.modular.com/administration/roles-permissions Review the different user roles and the permissions scoped to them Each member of your organization needs to be assigned a role that grants them some level of permissions. This page describes what roles exist and how to assign users the appropriate roles. [Set up roles in the console →](https://console.modular.com/members) ## Manage user roles Users with the `Owner` or `Admin` role can assign roles from the [Members](https://console.modular.com/members) page. You can assign a role in either of the following ways: * **When inviting a user:** Select a role before sending the invitation. * **After a user has joined:** Click **Update** for the user's entry and choose a new role. Role changes take effect immediately. Updated permissions apply to the user's next request, including requests made as part of an existing session. Every role change is recorded in the [Activities](https://console.modular.com/activities) log, including the user who made the change, the affected member, the previous role, the new role, and the time of the update. ## Roles and permissions matrix | Permission | Owner | Admin | Developer | User | | ------------------------------------------------------------------ | :---: | :---: | :-------: | :--: | | View deployments, endpoints, models, and usage | ✓ | ✓ | ✓ | ✓ | | Run inference on any resource | ✓ | ✓ | ✓ | ✓ | | Create and manage your own API keys | ✓ | ✓ | ✓ | — | | Manage organization API keys and keys owned by other members | ✓ | ✓ | — | — | | Invite, change role, and remove members (except the Owner) | ✓ | ✓ | — | — | | View and manage billing (payment methods, plan, credits, invoices) | ✓ | ✓ | — | — | | Accept Terms of Service and legal agreements | ✓ | — | — | — | | Transfer ownership | ✓ | — | — | — | Each API key also carries its own access scopes, which further limit what the key can do. For more information, see [API keys](/administration/api-keys). ### Owner The Owner, also called the Org Admin, has the highest level of access. Each organization has exactly one Owner. By default, the first person who signs up for an organization becomes its Owner. Only the Owner can accept the Terms of Service and other legal agreements for the organization. The organization can't purchase credits or run production inference until the Owner accepts these agreements. The Owner can also manage billing and transfer ownership to another member. When ownership is transferred, the previous Owner becomes an Admin or receives another selected role. An Owner can't remove or change their own role without first transferring ownership. ### Admin Admins manage the organization's day-to-day operations. They can manage billing, invite and remove members, assign roles, and manage organization API keys along with keys owned by other members. Admins can't remove the Owner or change the Owner's role. Assign this role to team leads who need broad operational access. ### Developer Developers can run inference and create, view, and revoke their own API keys. They can't view or manage organization API keys or keys created by other members. Developers can't access billing or manage members and roles. Assign this role to engineers and machine learning practitioners who build on the platform. ### User Users can view organization resources and usage data. They can also run inference. Users can't create, edit, or revoke API keys, access billing, or manage members and roles. Assign this role to people who use your inference endpoints but don't need broader access. # Create chat completion Source: https://docs.modular.com/api/inference/create-chat-completion /reference/openapi-inference.yaml post /v1/chat/completions Generate a response from a conversation. Use this API to send text, images, and/or videos and receive generated text back. All supported API capabilities, such as function calling, are documented here. The input and output modality support depends on the model you're using for inference. # Create response Source: https://docs.modular.com/api/inference/create-response /reference/openapi-inference.yaml post /v1/responses Generate an image or video. Send a text prompt or image to generate a new image or video. See the [Supported models](/models) page to find which models support these different modalities. # Connect with Cursor Source: https://docs.modular.com/coding-assistants/cursor Set up Cursor to use Modular Cloud models This page explains how to run models from Modular Cloud on [Cursor](https://www.cursor.com/). Custom models work in **Chat**, **Composer**, and **Edit** modes. Tab autocomplete uses Cursor's built-in model and doesn't support custom endpoints. ## Before you begin Make sure you have: * A Modular Cloud [account](https://console.modular.com/signup). * A Modular Cloud [API key](/administration/api-keys). * [Cursor](https://www.cursor.com/) installed. ## Set the base URL To point Cursor at the Modular Cloud API: 1. Open Cursor and go to **Settings > Models**. 2. Expand the **API Keys** section. 3. Paste your Modular Cloud API key in the **OpenAI API Key** field. 4. Enable **Override OpenAI Base URL** and paste the following URL: ```text theme={null} https://api.modular.com/v1 ``` Cursor automatically saves your changes. ## Add a model To add a Modular Cloud model to Cursor: 1. Choose the model that you want from our list of [shared endpoints](/models) or use a dedicated endpoint if applicable. 2. Open **Settings > Models** in Cursor. 3. In the **Models** section, click **View All Models > Add Custom Model**. 4. Enter the model ID: ```text theme={null} minimax/minimax-m3 ``` 5. Click **Add** and make sure the model is enabled. Once that's done, open a **Chat**, **Composer**, or **Edit** panel in Cursor and select the Modular Cloud model from the model picker. # Connect with Hermes Agent Source: https://docs.modular.com/coding-assistants/hermes-agent Set up Hermes Agent to use Modular Cloud models This page explains how to run models from Modular Cloud on your [Hermes Agent](https://hermes-agent.nousresearch.com/). ## Before you begin Make sure you have: * A Modular Cloud [account](https://console.modular.com/signup). * A Modular Cloud [API key](/administration/api-keys). * [Hermes Agent](https://hermes-agent.nousresearch.com/) installed. ## Hermes CLI or TUI To configure Modular Cloud as a provider in Hermes in the terminal: 1. Start Hermes with the command: ```bash theme={null} hermes model ``` The `model` option automatically runs the first time you run Hermes. Use the same steps to complete that initial process. 2. Select **Custom endpoint**. 3. Paste the base URL. Use `https://api.modular.com/v1` for testing. 4. Paste your API key now, or skip if you want to configure it later. 5. Select the **Auto-detect** API mode. 6. Choose a model to become your default model. Hermes automatically integrates all available Modular Cloud models for you to switch to later. 7. Click **Enter** to auto-detect the context length. 8. Enter a display name for the provider, such as `Modular Cloud`. After you establish the connection, run the `/model` command and select the Modular Cloud model to use. ## Hermes Desktop To configure Modular Cloud as a provider in Hermes Desktop: 1. Open the Hermes Desktop app. 2. Click the **gear** icon to open Settings. 3. Open **Providers > Custom Endpoints** from the left navigation bar. 4. In the **Add Endpoint** form, fill out the following: * **Name**: `Modular` * **Provider ID**: `modular` * **Endpoint URL**: The base URL, such as `https://api.modular.com/v1` * **Default Model**: A Modular Cloud model you have access to. Example: `minimax/minimax-m3` * **API Key**: Your Modular Cloud API key 5. Click **Test** to test the connection, then **Save**. This connects you to all the Modular Cloud models you have access to. Open a new chat and pick any model from the selector. ## Update the config file Both the Hermes CLI and Desktop share the same config file. To update settings for both: 1. Open the `~/.hermes/config.yaml` config file. 2. Update the Modular Cloud custom provider section: ```yaml theme={null} model: provider: "custom" model: "minimax/minimax-m3" base_url: "https://api.modular.com/v1" ``` You can also use the shortcut CLI command `hermes config set`. For example: ```bash theme={null} hermes config set api_key "your_api_key" ``` To see the complete list of provider configuration options, see [LLM and Model Providers](https://hermes-agent.nousresearch.com/docs/integrations/providers) in the Hermes Agent documentation. # Connect with OpenCode Source: https://docs.modular.com/coding-assistants/opencode Set up OpenCode to use Modular Cloud models This page explains how to run models from Modular Cloud on [OpenCode](https://opencode.ai/). ## Before you begin Make sure you have: * A [Modular Cloud account](https://console.modular.com/signup). * [OpenCode](https://opencode.ai/) installed. ## Add authentication First, connect Modular Cloud to OpenCode. 1. Create a [Modular Cloud API key](/administration/api-keys#create-an-api-key) and copy its value. 2. Start OpenCode: ```bash theme={null} opencode ``` 3. Run the `/connect` command in OpenCode. 4. At the end of the provider list, select **Other**. 5. Set the **Provider id** to `modular` and submit your API key. ## Update the OpenCode config Next, configure Modular Cloud as a provider in the OpenCode config. 1. Open the `opencode.json` config file in your project directory. 2. Choose the model that you want from our list of [shared endpoints](/models) or use a dedicated endpoint if applicable. 3. Add Modular Cloud as a provider: ```json opencode.json highlight={4-15} theme={null} { "$schema": "https://opencode.ai/config.json", "provider": { "modular": { "npm": "@ai-sdk/openai-compatible", "name": "Modular Cloud", "options": { "baseURL": "https://api.modular.com/v1" }, "models": { "minimax/minimax-m3": { "name": "MiniMax M3" } } } } } ``` Modular Cloud supports the OpenAI [chat completions API](/api/inference/create-chat-completion), so set `npm` to `@ai-sdk/openai-compatible`. Modular Cloud doesn't support the OpenAI responses API. You can also set a default model for OpenCode in the config: ```json opencode.json highlight={3} theme={null} { "$schema": "https://opencode.ai/config.json", "model": "modular/minimax/minimax-m3" } ``` To enable the config changes, restart OpenCode. Then, you can select any configured models when you run the `/models` command. The example above demonstrates a basic provider configuration. To see the complete list of configuration options, see the OpenCode documentation on [custom providers](https://opencode.ai/docs/providers#custom-provider). # Connect with Pi Source: https://docs.modular.com/coding-assistants/pi Set up the Pi agent harness to use Modular Cloud models This page explains how to run models from Modular Cloud on [Pi](https://pi.dev/). ## Before you begin Make sure you have: * A Modular Cloud [account](https://console.modular.com/signup). * A Modular Cloud [API key](/administration/api-keys). * [Pi](https://pi.dev/) installed. ## Configure a Pi provider To register Modular Cloud as a provider in Pi: 1. Open (or create) the `~/.pi/agent/models.json` configuration file. 2. Choose the model that you want from our list of [shared endpoints](/models) or use a dedicated endpoint if applicable. 3. Add Modular Cloud as a provider: ```json models.json theme={null} { "providers": { "modular": { "baseUrl": "https://api.modular.com/v1", "api": "openai-completions", "apiKey": "$MODULAR_API_KEY", "models": [ { "id": "minimax/minimax-m3", "name": "MiniMax M3" } ] } } } ``` Modular Cloud supports the OpenAI [chat completions API](/api/inference/create-chat-completion), so set `api` to `openai-completions`. Modular Cloud doesn't support the OpenAI responses API. Once you save your `models.json` file, run `/model` in Pi and select the Modular Cloud model to use. The example above demonstrates a basic provider and model configuration. To see the complete list of configuration options, see the Pi documentation on [Custom Models](https://pi.dev/docs/latest/models). # Connect with TRAE Source: https://docs.modular.com/coding-assistants/trae Set up TRAE to use Modular Cloud models This page explains how to run models from Modular Cloud on [TRAE](https://www.trae.ai/). ## Before you begin Make sure you have: * A Modular Cloud [account](https://console.modular.com/signup). * A Modular Cloud [API key](/administration/api-keys). * [TRAE](https://www.trae.ai/) installed. ## Add a custom provider To configure Modular Cloud as a provider in TRAE: 1. Choose the model that you want from our list of [shared endpoints](/models) or use a dedicated endpoint if applicable. 2. Open TRAE and go to **Settings > Models**. 3. Select **Add Model > Custom Model**. 4. Fill out the following fields: * **API Format**: OpenAI Chat Completions * **Base URL**: `https://api.modular.com/v1` * **Model ID**: `minimax/minimax-m3` * **API Key**: Your Modular Cloud API key 5. Click **Add Model** to save. Then you can open the model picker in TRAE and select the Modular Cloud model to use. To understand the complete list of model settings, see the TRAE documentation on [Built-in models & custom models](https://docs.trae.ai/ide/models). # Deploy with Modular Cloud Source: https://docs.modular.com/index A simple and powerful inference solution Modular's uniquely heterogeneous infrastructure allows you to find the best cost-to-performance ratio for your AI use case. We handle the hard parts, so you can go faster. Get free credits and try it without writing any code. [Start now for free →](/quickstart) ## Built different Other AI infrastructure is assembled from parts that weren't designed to work together: one tool for serving, one for model optimization, and one (or more) for GPU kernels. Modular is built different. Modular is a unified stack, from the kernels to the cloud. This gives us more flexibility to deploy across NVIDIA, AMD, and other hardware, which gives you more options to get the performance you need at the right cost. It's all built upon two key open source components: MAX and Mojo. An open and extensible framework for building and serving AI. It's the serving and modeling framework that makes our models performant and portable. You can use it yourself to build and deploy your own models. An open source programming language built for the AI era. It's a Pythonic and safe systems language that can write blazing fast code for CPUs, GPUs, and ASICs. It powers all of our kernels, and you can use it to accelerate any project. # Function calling Source: https://docs.modular.com/inference/function-calling Use function calling and tool use with the chat completions API Function calling lets a large language model (LLM) call external functions (also called *tools*) during inference. This means the model can retrieve data from external systems or run other tasks, then use the results in its response. To make functions available to the model, pass a `tools` parameter in the request body of the [chat completions API](/api/inference/create-chat-completion). This only works for models that support function calling. Note that you can't use the [responses API](/api/inference/create-response) for function calling. ## When to use function calling Pass a `tools` parameter in your chat completions request when you want the model to call your code instead of only returning text. Typical uses include: * **Fetching data**: Declare tools that read from APIs or databases (for example, weather, prices, or search). The model requests the data; your code runs the call and returns the result for the model to use in its reply. * **Performing actions**: Declare tools that change state or trigger work (for example, update an app, start a workflow, or call another system). The model chooses the tool and arguments; your code executes the action. ## How function calling works Function calling follows this loop: 1. You declare tools in the [chat completions API](/api/inference/create-chat-completion) request. Each tool describes *what the function does* and *which arguments it accepts*. 2. The model returns a response that includes `tool_calls` when it wants to use one. 3. Your code runs the function. 4. For data-fetching tools, you send the result back in a follow-up request. The following sections demonstrate this loop using a weather-fetching tool example. ### Declare tools The first step is to declare the tools your model can use. The following example declares a `get_weather` tool and references the tool in the request: ```python theme={null} from openai import OpenAI def get_weather(location: str) -> str: print("Get weather:", location) return '{"temperature": 62, "unit": "F"}' client = OpenAI( base_url="https://api.modular.com/v1", api_key="", # Load your key ) tools = [{ "type": "function", "function": { "name": "get_weather", "description": "Get current temperature for a given location.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "City and country e.g. Bogotá, Colombia" } }, "required": [ "location" ], "additionalProperties": False }, "strict": True } }] messages = [ { "role": "user", "content": "What's the weather like in San Francisco today?" } ] completion = client.chat.completions.create( model="minimax/minimax-m3", messages=messages, tools=tools ) ``` Take a closer look at each parameter shown in the `tools` property: * `type`: The type of tool. Currently, we only support `function`. * `function`: The function definition. * `name`: The name that the model uses when calling the function. * `description`: A description that helps the model decide when to call the function. * `parameters`: A JSON Schema for the arguments. * `type`: Must be `object`. * `properties`: A list of all argument names and types. * `required`: A list of arguments the model must supply to the function. ### Control tool use Use the `tool_choice` parameter to control tool use. It accepts the following values: * `none`: Don't call any tool. * `auto` (default): Let the model decide whether to call a tool or reply with a message. * `required`: Require at least one tool call. * A specific function: Force the model to always call a specific function, for example: ```json theme={null} "tool_choice": { "type": "function", "function": {"name": "get_weather"} } ``` ### Handle the model response If the model chooses a tool, the response includes `tool_calls`: ```python theme={null} print(completion.choices[0].message.tool_calls) ``` This will print: ```js theme={null} [ChatCompletionMessageToolCall( id='call_a175692d9ff54554', function=Function( arguments='{ "location": "San Francisco, USA" }', name='get_weather' ), type='function' )] ``` Parse `tool_calls` and run the matching function in your code: ```py theme={null} import json tool_call = completion.choices[0].message.tool_calls[0] args = json.loads(tool_call.function.arguments) result = get_weather(args["location"]) ``` ### Return results to the model For tools that fetch data, send another request with the conversation so far, including: * Prior messages * The assistant message that returned `tool_calls` * A `tool` message for each result Set each `tool` message's `tool_call_id` to the `id` of the matching tool call in that assistant message. ```json theme={null} "messages": [ { "role": "user", "content": "What's the weather in San Francisco?" }, { "role": "assistant", "tool_calls": [ { "id": "call_abc123", "type": "function", "function": { "name": "get_weather", "arguments": "{\"location\": \"San Francisco\"}" } } ] }, { "role": "tool", "tool_call_id": "call_abc123", "content": "{\"temperature\": 62, \"unit\": \"F\"}" } ] ``` Each tool message's `tool_call_id` must match an id from the preceding assistant message. The API returns an HTTP 400 response if a `tool_calls` argument isn't JSON, a `tool_call_id` doesn't match, or you omit any required tool reply. ### Run the example This walks through the same `get_weather` example from above as a single, runnable script. If you haven't already, [create an API key](/administration/api-keys#create-an-api-key) and store it securely. Then, send a request with the `tools` parameter to a model that supports function calling: Install the `openai` package: ```bash theme={null} pip install openai ``` Create a program to send a request specifying the available `get_weather()` function: ```python title="function-calling.py" theme={null} import json from openai import OpenAI def get_weather(location: str) -> str: print("Get weather:", location) return '{"temperature": 62, "unit": "F"}' client = OpenAI( base_url="https://api.modular.com/v1", api_key="", # Load your key ) tools = [{ "type": "function", "function": { "name": "get_weather", "description": "Get current temperature for a given location.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "City and country e.g. Bogotá, Colombia" } }, "required": [ "location" ], "additionalProperties": False }, "strict": True } }] messages = [ { "role": "user", "content": "What's the weather like in San Francisco today?" } ] completion = client.chat.completions.create( model="minimax/minimax-m3", messages=messages, tools=tools ) tool_call = completion.choices[0].message.tool_calls[0] args = json.loads(tool_call.function.arguments) result = get_weather(args["location"]) ``` Run it and the `get_weather()` function should print the argument received: ```sh theme={null} python function-calling.py ``` ```output theme={null} Get weather: San Francisco, USA ``` Use the following `curl` command to send a request specifying the available `get_weather()` function: ```bash theme={null} curl -X POST https://api.modular.com/v1/chat/completions \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "model": "minimax/minimax-m3", "messages": [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is the weather like in Boston today?"} ], "tools": [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. Los Angeles, CA" } }, "required": ["location"] } } } ], "tool_choice": "auto" }' ``` You should receive a response similar to this: ```json theme={null} "tool_calls": [ { "id": "call_ac73df14fe184349", "type": "function", "function": { "name": "get_weather", "arguments": "{\"location\": \"Boston, MA\"}" } } ] ``` ## Troubleshooting If your function calling request fails, try the following: * Confirm that function calling is supported by the model. * Check if streaming is enabled. Depending on the model, streaming may interfere with function calling. If you see incomplete or malformed tool-call output while streaming, set `stream: false` for that request. # Image generation Source: https://docs.modular.com/inference/image Generate and transform images with the responses API To use an image generation model, you need to use the [responses API](/api/inference/create-response). The API accepts text and image inputs, and returns generated images as base64-encoded data. The examples below use the `FLUX.2-klein-4B` model, but you can replace it with any [supported model](/models) that's listed as an "Image" type. ## Requirements * An [API key](/administration/api-keys). The code below assumes you set it in an environment variable: ```bash macOS/Linux theme={null} export MODULAR_API_KEY="your_api_key" ``` ```bash Windows theme={null} $env:MODULAR_API_KEY="your_api_key" ``` * The `openai` Python package. You can install it with this command: ```bash Python (pip) theme={null} pip install openai ``` ```bash Python (uv) theme={null} uv add openai ``` ```bash Python (pixi) theme={null} pixi add openai ``` ```bash TypeScript (npm) theme={null} npm install openai ``` ## Generate an image To generate an image, set `input` to a text description of the image and set generation parameters in `provider_options.image`. ```python Python theme={null} import base64 import os from pathlib import Path from openai import OpenAI client = OpenAI( base_url="https://api.modular.com/v1", api_key=os.environ.get("MODULAR_API_KEY"), ) response = client.responses.create( model="black-forest-labs/FLUX.2-klein-4B", input="A serene mountain landscape at sunset", extra_body={ "provider_options": { "image": { "height": 512, "width": 512, "steps": 4, } } }, ) image_data = response.output[0].content[0].image_data Path("landscape.png").write_bytes(base64.b64decode(image_data)) ``` ```typescript TypeScript theme={null} import { writeFile } from "node:fs/promises"; import process from "node:process"; import OpenAI from "openai"; const client = new OpenAI({ baseURL: "https://api.modular.com/v1", apiKey: process.env.MODULAR_API_KEY, }); async function main() { const response = await client.responses.create({ model: "black-forest-labs/FLUX.2-klein-4B", input: "A serene mountain landscape at sunset", provider_options: { image: { height: 512, width: 512, steps: 4, }, }, }); const image_data = response.output[0].content[0].image_data; await writeFile("landscape.png", Buffer.from(image_data, "base64")); } main(); ``` ```bash cURL theme={null} curl -X POST https://api.modular.com/v1/responses \ -H "Authorization: Bearer $MODULAR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "black-forest-labs/FLUX.2-klein-4B", "input": "A serene mountain landscape at sunset", "provider_options": { "image": { "height": 512, "width": 512, "steps": 4 } } }' \ | jq -r '.output[0].content[0].image_data' \ | base64 -d > landscape.png ``` You should quickly see an image that looks like this: ## Transform an image To transform an image, set `input` to a user message containing: * An `input_image` block with an image URL or base64-encoded data URI. * An `input_text` block describing the transformation. This example encodes the `landscape.png` file created above and transforms it into a watercolor painting: ```python Python theme={null} import base64 import os from pathlib import Path from openai import OpenAI client = OpenAI( base_url="https://api.modular.com/v1", api_key=os.environ.get("MODULAR_API_KEY"), ) input_data = base64.b64encode(Path("landscape.png").read_bytes()).decode("utf-8") response = client.responses.create( model="black-forest-labs/FLUX.2-klein-4B", input=[ { "role": "user", "content": [ { "type": "input_image", "image_url": f"data:image/png;base64,{input_data}", }, { "type": "input_text", "text": "Transform this image into a watercolor painting.", }, ], } ], extra_body={ "provider_options": { "image": { "height": 512, "width": 512, "steps": 4, } } }, ) image_data = response.output[0].content[0].image_data Path("watercolor.png").write_bytes(base64.b64decode(image_data)) ``` ```typescript TypeScript theme={null} import { readFile, writeFile } from "node:fs/promises"; import process from "node:process"; import OpenAI from "openai"; const client = new OpenAI({ baseURL: "https://api.modular.com/v1", apiKey: process.env.MODULAR_API_KEY, }); async function main() { const inputImage = await readFile("landscape.png"); const image_data = inputImage.toString("base64"); const response = await client.responses.create({ model: "black-forest-labs/FLUX.2-klein-4B", input: [ { role: "user", content: [ { type: "input_image", image_url: `data:image/png;base64,${image_data}`, }, { type: "input_text", text: "Transform this image into a watercolor painting.", }, ], }, ], provider_options: { image: { height: 512, width: 512, steps: 4, }, }, }); const output_image_data = response.output[0].content[0].image_data; await writeFile("watercolor.png", Buffer.from(output_image_data, "base64")); } main(); ``` ```bash cURL theme={null} IMAGE_DATA=$(base64 < landscape.png | tr -d '\n') curl -X POST https://api.modular.com/v1/responses \ -H "Authorization: Bearer $MODULAR_API_KEY" \ -H "Content-Type: application/json" \ --data-binary @- < watercolor.png { "model": "black-forest-labs/FLUX.2-klein-4B", "input": [ { "role": "user", "content": [ { "type": "input_image", "image_url": "data:image/png;base64,$IMAGE_DATA" }, { "type": "input_text", "text": "Transform this image into a watercolor painting." } ] } ], "provider_options": { "image": { "height": 512, "width": 512, "steps": 4 } } } EOF ``` You should see the image transformed like this: # Reasoning Source: https://docs.modular.com/inference/reasoning Enable chain-of-thought reasoning with the chat completions API Reasoning lets a large language model (LLM) think through a problem step by step before responding. When reasoning is enabled, the model generates chain-of-thought text and returns that text separately from the answer. Reasoning only works for models that support it. ## The `reasoning` parameter To enable or disable reasoning, pass a `reasoning` parameter in the request body of the [chat completions API](/api/inference/create-chat-completion). Some models reason by default and don't require the `reasoning` parameter. Omit the `reasoning` parameter in a request to see if the model returns reasoning by default or not. The chat completions API also accepts a `thinking` parameter as an alternative to `reasoning`. Use whichever format your client or model expects. ## Send a reasoning request Here's an example that shows how to send a request with reasoning enabled. Be sure to [create an API key](/administration/api-keys#create-an-api-key) before you run it. ```python Python theme={null} import os from openai import OpenAI client = OpenAI( base_url="https://api.modular.com/v1", api_key=os.environ.get("MODULAR_API_KEY"), ) completion = client.chat.completions.create( model="google/gemma-4-31b-it", messages=[ {"role": "user", "content": "How many r's are in the word strawberry?"} ], extra_body={"reasoning": {"enabled": True}} ) print(completion.to_json(indent=2)) ``` ```bash cURL theme={null} curl -X POST https://api.modular.com/v1/chat/completions \ -H "Authorization: Bearer $MODULAR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "google/gemma-4-31b-it", "messages": [ {"role": "user", "content": "How many r'\''s are in the word strawberry?"} ], "reasoning": {"enabled": true} }' ``` You should receive a response similar to this: ```json theme={null} { "choices": [ { "finish_reason": "stop", "index": 0, "message": { "content": "There are 3 \"r\"s in strawberry.", "role": "assistant", "reasoning": "The user is asking for the number of letter 'r's in the word \"strawberry\".\n\n * S-T-R-A-W-B-E-R-R-Y\n * R (1)\n * R (2)\n * R (3)\n\nThere are 3 'r's." } } ], "model": "google/gemma-4-31b-it", "usage": { "completion_tokens": 144, "prompt_tokens": 27, "total_tokens": 171, "completion_tokens_details": { "reasoning_tokens": 132 } } } ``` ## Understand the response When reasoning is enabled, the `reasoning` response field contains the model's chain-of-thought text. When reasoning is disabled or the model doesn't support it, `reasoning` is null. Some models return a `reasoning_content` field instead; this is equivalent to `reasoning` and contains the same information. The `usage` field in the response reports `reasoning_tokens`, which counts the tokens used for chain-of-thought. This way, you know how many additional tokens the model uses when reasoning is enabled. When `reasoning` is disabled or not supported, the request returns `0` on this field. # Text generation Source: https://docs.modular.com/inference/text Generate text from an LLM using text, image, and video inputs To generate text with an LLM, you need to use the [chat completions API](/api/inference/create-chat-completion). This API allows you to send text, images, and videos with your request, using an OpenAI-compatible interface, as shown in the following examples. The examples below use the `minimax-m3` model, but you can replace it with any [supported model](/models) that's listed as an "LLM" type. ## Requirements * An [API key](/administration/api-keys). The code below assumes you set it in an environment variable: ```bash macOS/Linux theme={null} export MODULAR_API_KEY="your_api_key" ``` ```bash Windows theme={null} $env:MODULAR_API_KEY="your_api_key" ``` * The `openai` Python package. You can install it with this command: ```bash Python (pip) theme={null} pip install openai ``` ```bash Python (uv) theme={null} uv add openai ``` ```bash Python (pixi) theme={null} pixi add openai ``` ```bash TypeScript (npm) theme={null} npm install openai ``` ## Generate a text response Here's how you can generate text from a prompt or conversation history. ```python Python theme={null} import os from openai import OpenAI client = OpenAI( base_url="https://api.modular.com/v1", api_key=os.environ.get("MODULAR_API_KEY"), ) response = client.chat.completions.create( model="minimax/minimax-m3", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Who won the world series in 2020?"}, {"role": "assistant", "content": "The LA Dodgers won in 2020."}, {"role": "user", "content": "Where was it played?"} ] ) print(response.choices[0].message.content) ``` ```typescript TypeScript theme={null} import process from "node:process"; import OpenAI from "openai"; const client = new OpenAI({ baseURL: "https://api.modular.com/v1", apiKey: process.env.MODULAR_API_KEY, }); async function main() { const response = await client.chat.completions.create({ model: "minimax/minimax-m3", messages: [ { role: "system", content: "You are a helpful assistant." }, { role: "user", content: "Who won the world series in 2020?" }, { role: "assistant", content: "The LA Dodgers won in 2020." }, { role: "user", content: "Where was it played?" }, ], }); console.log(response.choices[0].message.content); } main(); ``` ```bash cURL theme={null} curl -X POST https://api.modular.com/v1/chat/completions \ -H "Authorization: Bearer $MODULAR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "minimax/minimax-m3", "messages": [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Who won the world series in 2020?"}, {"role": "assistant", "content": "The LA Dodgers won in 2020."}, {"role": "user", "content": "Where was it played?"} ] }' ``` ## Stream a text response Set `stream=True` to receive tokens as they are generated instead of waiting for the full response. ```python Python highlight={12} theme={null} import os from openai import OpenAI client = OpenAI( base_url="https://api.modular.com/v1", api_key=os.environ.get("MODULAR_API_KEY"), ) stream = client.chat.completions.create( model="minimax/minimax-m3", messages=[{"role": "user", "content": "Write a short poem about the sea."}], stream=True, ) for chunk in stream: if not chunk.choices: continue content = chunk.choices[0].delta.content if content is not None: print(content, end="", flush=True) print() ``` ```typescript TypeScript highlight={13} theme={null} import process from "node:process"; import OpenAI from "openai"; const client = new OpenAI({ baseURL: "https://api.modular.com/v1", apiKey: process.env.MODULAR_API_KEY, }); async function main() { const stream = await client.chat.completions.create({ model: "minimax/minimax-m3", messages: [{ role: "user", content: "Write a short poem about the sea." }], stream: true, }); for await (const chunk of stream) { process.stdout.write(chunk.choices[0]?.delta?.content ?? ""); } process.stdout.write("\n"); } main(); ``` ```bash cURL highlight={7} theme={null} curl -X POST https://api.modular.com/v1/chat/completions \ -H "Authorization: Bearer $MODULAR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "minimax/minimax-m3", "messages": [{"role": "user", "content": "Write a short poem about the sea."}], "stream": true }' ``` ## Analyze an image Some LLMs accept images, allowing them to generate a description or analysis of the image visual content. You can use any [supported model](/models) that's both an "LLM" and a "Vision" type. To pass an image, add the `image_url` attribute in the `messages.content` object. The inner `url` attribute accepts either a publicly accessible URL or a base64-encoded data URI (for example, `data:image/jpeg;base64,...`). Notice that `content` is an array, allowing you to pass multiple images at once. ```python Python theme={null} import os from openai import OpenAI client = OpenAI( base_url="https://api.modular.com/v1", api_key=os.environ.get("MODULAR_API_KEY"), ) response = client.chat.completions.create( model="minimax/minimax-m3", messages=[ { "role": "user", "content": [ {"type": "text", "text": "What is in this image?"}, { "type": "image_url", "image_url": { "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg" }, }, ], } ], max_tokens=300, ) print(response.choices[0].message.content) ``` ```typescript TypeScript theme={null} import process from "node:process"; import OpenAI from "openai"; const client = new OpenAI({ baseURL: "https://api.modular.com/v1", apiKey: process.env.MODULAR_API_KEY, }); async function main() { const response = await client.chat.completions.create({ model: "minimax/minimax-m3", messages: [ { role: "user", content: [ { type: "text", text: "What is in this image?", }, { type: "image_url", image_url: { url: "https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg", }, }, ], }, ], max_tokens: 300, }); console.log(response.choices[0].message.content); } main(); ``` ```bash cURL theme={null} curl -X POST https://api.modular.com/v1/chat/completions \ -H "Authorization: Bearer $MODULAR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "minimax/minimax-m3", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "What is in this image?" }, { "type": "image_url", "image_url": { "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg" } } ] } ], "max_tokens": 300 }' | grep -o '"content":"[^"]*"' | sed 's/"content":"//g' | sed 's/"//g' | tr -d '\n' | sed 's/\\n/\n/g' ``` ## Analyze a video Some LLMs accept video input, allowing them to generate a description or analysis of the video's visual content. You can use any [supported model](/models) that's both an "LLM" and a "Vision" type. To pass a video, add the `video_url` attribute in the `messages.content` object. The inner `url` attribute accepts either a publicly accessible URL or a base64-encoded data URI (for example, `data:video/mp4;base64,...`). Notice that `content` is an array, allowing you to pass multiple videos at once, or even a combination of videos and images. ```python Python theme={null} import os from openai import OpenAI client = OpenAI( base_url="https://api.modular.com/v1", api_key=os.environ.get("MODULAR_API_KEY"), ) completion = client.chat.completions.create( model="minimax/minimax-m3", messages=[ { "role": "user", "content": [ {"type": "text", "text": "Describe what is happening in this video"}, { "type": "video_url", "video_url": { "url": "https://avtshare01.rz.tu-ilmenau.de/avt-vqdb-uhd-1/test_1/segments/bigbuck_bunny_8bit_15000kbps_1080p_60.0fps_h264.mp4" }, }, ], } ], max_tokens=300, # minimax-m3 uses adaptive thinking by default, which can consume the entire # token budget when analyzing videos, leaving message.content empty. extra_body={"reasoning": {"enabled": False}}, ) print(completion.choices[0].message.content) ``` ```typescript TypeScript theme={null} import process from "node:process"; import OpenAI from "openai"; const client = new OpenAI({ baseURL: "https://api.modular.com/v1", apiKey: process.env.MODULAR_API_KEY, }); async function main() { const response = await client.chat.completions.create({ model: "minimax/minimax-m3", messages: [ { role: "user", content: [ { type: "text", text: "Describe what is happening in this video", }, { type: "video_url", video_url: { url: "https://avtshare01.rz.tu-ilmenau.de/avt-vqdb-uhd-1/test_1/segments/bigbuck_bunny_8bit_15000kbps_1080p_60.0fps_h264.mp4", }, }, ], }, ], max_tokens: 300, // minimax-m3 uses adaptive thinking by default, which can consume the entire // token budget when analyzing videos, leaving message.content empty. reasoning: { enabled: false }, } as OpenAI.Chat.Completions.ChatCompletionCreateParamsNonStreaming); console.log(response.choices[0].message.content); } main(); ``` ```bash cURL theme={null} curl -X POST https://api.modular.com/v1/chat/completions \ -H "Authorization: Bearer $MODULAR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "minimax/minimax-m3", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe what is happening in this video" }, { "type": "video_url", "video_url": { "url": "https://avtshare01.rz.tu-ilmenau.de/avt-vqdb-uhd-1/test_1/segments/bigbuck_bunny_8bit_15000kbps_1080p_60.0fps_h264.mp4" } } ] } ], "max_tokens": 300, "reasoning": {"enabled": false} }' | grep -o '"content":"[^"]*"' | sed 's/"content":"//g' | sed 's/"//g' | tr -d '\n' | sed 's/\\n/\n/g' ``` # Video generation Source: https://docs.modular.com/inference/video Generate videos from text or images with the responses API To use a video generation model, you need to use the [responses API](/api/inference/create-response). The API accepts text and image inputs, and returns generated videos as base64-encoded data. **Notice:** We currently don't offer any shared endpoints for video generation, so the examples below work with dedicated endpoints only. ## Requirements * An [API key](/administration/api-keys). The code below assumes you set it in an environment variable: ```bash macOS/Linux theme={null} export MODULAR_API_KEY="your_api_key" ``` ```bash Windows theme={null} $env:MODULAR_API_KEY="your_api_key" ``` * The `openai` Python package. You can install it with this command: ```bash Python (pip) theme={null} pip install openai ``` ```bash Python (uv) theme={null} uv add openai ``` ```bash Python (pixi) theme={null} pixi add openai ``` ## Generate a video To generate a video, set `input` to a text description of the video and set generation parameters in `provider_options.image`. The following example generates a video from a text prompt and saves it as `output-text-to-video.mp4`. ```python Python theme={null} import base64 import os from pathlib import Path from openai import OpenAI client = OpenAI( base_url="https://api.modular.com/v1", api_key=os.environ.get("MODULAR_API_KEY"), ) response = client.responses.create( model="Wan-AI/Wan2.2-T2V-A14B-Diffusers", input=( "A campfire crackles in a forest clearing at night, with sparks " "spiraling into a star-filled sky." ), extra_body={ "provider_options": { "image": { "height": 512, "width": 512, "steps": 28, } } }, ) video_data = response.output[0].content[0].image_data Path("output-text-to-video.mp4").write_bytes(base64.b64decode(video_data)) ``` ```bash cURL theme={null} curl -X POST https://api.modular.com/v1/responses \ -H "Authorization: Bearer $MODULAR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "Wan-AI/Wan2.2-T2V-A14B-Diffusers", "input": "A campfire crackles in a forest clearing at night, with sparks spiraling into a star-filled sky.", "provider_options": { "image": { "height": 512, "width": 512, "steps": 28 } } }' \ | jq -r '.output[0].content[0].image_data' \ | base64 -d > output-text-to-video.mp4 ``` The generated video should look similar to the following: