Skip to main content
Rate limits determine the maximum number of inference requests your organization can send in a given timeframe. They are measured in requests per minute (RPM) and tokens per minute (TPM).

​
How rate limits work

Rate limits are enforced at two levels:
  • Tier limits apply to all requests made by your organization across every model. Every request counts toward the same organization-wide budget.
  • Model limits apply independently to each model and are enforced per endpoint.
Your request succeeds only if both limits have available capacity. If either limit is exceeded, the request fails with an HTTP 429 response. You’ll have to wait before future requests run successfully. If this happens frequently, contact us to increase your limits.

​
Tier limits

The following table shows the rate limits for starter plan tiers. Higher rate limits are available for Enterprise plans. Contact us to learn more.

​
Model limits

Some models have their own rate limit. The limit is enforced separately for each endpoint serving that model.
  • For dedicated endpoints, the model’s rate limit is shared by all users in your organization.
  • For shared endpoints, the model’s rate limit is shared by every organization using that endpoint.
The following table lists models that have their own rate limit.