AI Gateway Rate Limits

Varity TeamCore Contributors

The AI Gateway (https://ai.varity.app/v1) is a separate surface from the Varity platform API and enforces its own budgets. Nothing on Rate Limits applies here, and nothing here applies there.

Two different mechanisms are at work, and they return different headers. Read the one that covers the route you are calling.

Routes	Bound	Where it comes from
Inference (/v1/chat/completions, /v1/embeddings, /v1/images/*, /v1/audio/*)	120 requests per minute and 31 concurrent requests on the standard tier	Admission, per account
Inference reads (/v1/inference/policy, /requests, /usage, /performance)	120 requests per 60-second window	A request budget, per account
GET /v1/models, GET /v1/openapi.json	No credential and no request budget	Public discovery
Account routes (/v1/keys*, /v1/billing/*, PUT /v1/inference/policy)	See Account routes	A request budget, per account

Read Your Own Limits

Do not hardcode the defaults. GET /v1/inference/policy publishes the numbers your account is actually held to:

curl https://ai.varity.app/v1/inference/policy \
  -H "Authorization: Bearer $VARITY_API_KEY"

{
  "object": "inference.tenant_policy",
  "privacy_floor": "anonymized",
  "limit_tier": "standard",
  "requests_per_minute": 120,
  "concurrency_limit": 31,
  "limits_scope": "account"
}

Field	Meaning
limit_tier	The named tier your account’s bounds belong to. Self-service accounts are standard. Absent when the account’s bounds do not match a named tier.
requests_per_minute	Inference requests admitted in any sliding 60-second window.
concurrency_limit	Inference requests that may be in flight at once.
limits_scope	Always account. Both bounds are counted globally for the account.
credential_requests_per_minute	Present only when you authenticate with a scoped key that carries its own lower rpm bound; null when it does not.
credential_concurrency_limit	The same, for concurrency.

requests_per_minute, concurrency_limit, and limits_scope are absent when the account has no admission policy yet. Absent means “no bound to state”, not “unlimited” — treat it as unknown and pace conservatively.

Higher limits are available on an enterprise agreement.

Caution

limits_scope: "account" means the budget is shared. Every API key your account owns draws from the same per-minute budget and the same concurrent slots, and so does the Developer Portal Playground. The gateway runs on more than one replica, and the bounds are counted across all of them, so do not multiply a limit by the number of replicas you think are running.

A key can carry its own requests_per_minute or concurrency_limit. When it does, the account bound is checked first and the key’s bound second, so the key is held to the smaller of the two. Neither number alone is the whole truth, which is why the policy route reports them side by side rather than merging them.

Inference: 120 Per Minute And 31 In Flight

The standard tier is 120 requests per minute and 31 concurrent in-flight requests. A request that crosses either bound is refused at admission, before any model provider is called, so a refused request costs nothing.

The per-minute bound is counted over a sliding 60-second window. It never resets all at once: one request leaves the window when it turns 60 seconds old, and that frees one request.

The concurrency bound is sized from the per-minute one: 31 is the number of requests a client pacing at exactly 120 per minute has in flight when each request takes the gateway’s measured 95th-percentile chat duration.

The practical consequences:

• Both bounds are account-scoped, so extra keys on the same account share the same budget and buy nothing.
• Read both numbers from GET /v1/inference/policy rather than assuming the defaults; an account may be provisioned differently.

Pacing Headers

An admitted inference request carries RateLimit-Limit, RateLimit-Remaining, RateLimit-Reset, and RateLimit-Policy before the first byte of the response, so you can pace yourself before you are refused. When a scoped key carries its own lower per-minute bound, the headers report whichever of the account and key bounds has fewer requests left.

The Two Refusals

Both are 429, and both pace you — but not with the same header set, because they are not the same kind of bound:

Header	rate_limit_exceeded	concurrency_limit_exceeded	upstream_rate_limited
Retry-After	yes: seconds until the oldest request leaves the window	yes: the expected wait for a slot	only if the provider published its own reset
retry-after-ms	yes: the same wait in milliseconds	yes: the same wait in milliseconds	with Retry-After
RateLimit-Limit	yes	yes	no
RateLimit-Remaining	0	0	no
RateLimit-Reset	yes — same value as Retry-After	yes — same value as Retry-After	no
RateLimit-Policy	yes — 120;w=60	no — the bound has no window	no

Every wait the gateway publishes is at most 60 seconds, inside the retry ceilings of the stock OpenAI SDKs, which read retry-after-ms first and Retry-After second.

Each bound also exists at two scopes. The account’s bound is reported with the message The tenant request rate limit is exhausted. / The tenant concurrency limit is exhausted.; a key carrying its own lower override reports The API key … instead. The code is identical either way, so branch on code and read GET /v1/inference/policy to see which bound applies to you.

rate_limit_exceeded

{
  "error": {
    "message": "The tenant request rate limit is exhausted.",
    "type": "rate_limit_error",
    "code": "rate_limit_exceeded",
    "param": null,
    "request_id": "…",
    "diagnostic_code": "ai_gateway.rate_limit_exceeded",
    "failure_cause": "rate_limit_exceeded",
    "failure_domain": "varity_projection",
    "lifecycle_stage": "admission",
    "action": "Retry after the credential rate-limit window resets.",
    "retryable": true
  }
}

Status 429. It carries the full set of pacing headers, including the policy, because this bound is counted over a window:

retry-after: <seconds until the oldest request leaves the window>
retry-after-ms: <the same wait in milliseconds>
ratelimit-limit: 120
ratelimit-remaining: 0
ratelimit-reset: <same as retry-after>
ratelimit-policy: 120;w=60

The values track your account’s own requests_per_minute, so a key with a lower override reports that lower number here. Because the window slides, the wait is the time until the oldest request in it turns 60 seconds old, not a fixed 60.

concurrency_limit_exceeded

Fire more requests at once than the bound allows and the surplus is refused:

retry-after: <expected seconds until a slot frees>
retry-after-ms: <the same wait in milliseconds>
ratelimit-limit: 31
ratelimit-remaining: 0
ratelimit-reset: <same as retry-after>

{
  "error": {
    "message": "The tenant concurrency limit is exhausted.",
    "type": "rate_limit_error",
    "code": "concurrency_limit_exceeded",
    "param": null,
    "request_id": "…",
    "diagnostic_code": "ai_gateway.concurrency_limit_exceeded",
    "failure_cause": "concurrency_limit_exceeded",
    "failure_domain": "varity_projection",
    "lifecycle_stage": "admission",
    "action": "Retry after an active inference request finishes.",
    "retryable": true
  }
}

Retry-After and RateLimit-Reset name the same instant: the expected moment a slot frees. The gateway estimates it from the oldest request in flight and the typical chat duration, never reports less than 1 second, and never reports later than the nearest in-flight deadline: the instant the serving replica aborts that request and reclaims its slot. Read the header every time; the value changes from one refusal to the next.

Note

RateLimit-Policy is deliberately absent on a concurrency refusal, and it is the only pacing header that is. The header’s w parameter is mandatory and names the window the limit is counted over — but a concurrency bound is not counted over a window at all. Thirty-one in flight is not thirty-one per anything. Emitting a policy with a window would assert a throughput policy that does not exist, and a client would then pace itself by it. The per-minute refusal, which is windowed, carries ratelimit-policy: 120;w=60.

If no wait could be estimated, Retry-After, retry-after-ms, and RateLimit-Reset are omitted rather than guessed. Fall back to your own backoff when they are not present.

upstream_rate_limited

A 429 can also come from the model provider rather than from your Varity budget. Its code is upstream_rate_limited, its failure_domain is supplier_observed, and it is retryable. It carries Retry-After only when the provider published its own reset; when the provider did not, no Retry-After is sent. Branch on error.code, not on the status, to tell a Varity budget refusal from provider backpressure.

Inference Reads: 120 Per 60 Seconds

GET /v1/inference/policy, /v1/inference/requests, /v1/inference/usage, and /v1/inference/performance share one budget of 120 requests per 60-second window, keyed to the authenticated account.

These four routes emit standard headers on every response that passes the limiter, not only on a refusal:

Header	Meaning
RateLimit-Policy	120;w=60 — the budget and its window.
RateLimit-Limit	Requests allowed in the current window.
RateLimit-Remaining	Requests left in the current window.
RateLimit-Reset	Seconds until the window resets.
Retry-After	On the 429 only.

Legacy X-RateLimit-* headers are not emitted here. Do not depend on them.

A refusal is 429 with code: "rate_limit_exceeded" and the message Too many managed inference requests.

Note

The budget is applied after authentication, so a 401 never carries RateLimit-* headers. You will only see them on a request that authenticated successfully.

Public Discovery Is Not Rate Limited

GET /v1/models and GET /v1/openapi.json take no credential and the gateway applies no request budget to them. They return no RateLimit-* headers, so there is nothing to program against there:

curl -sSI https://ai.varity.app/v1/models

HTTP/2 200
content-type: application/json; charset=utf-8
vary: accept-encoding
x-request-id: …

Poll the catalog as often as you need; it is served from the gateway’s own state, not from a provider.

Account Routes

The key, billing, and policy-change routes on this host take a Developer Portal session token, not an inference API key. Their budgets are separate again:

Routes	Budget	Window
GET /v1/keys, GET /v1/keys/audit	300 requests	1 minute
POST /v1/keys (mint a key)	30 requests	1 hour
POST /v1/keys/{id}/revoke	120 requests	1 minute
/v1/billing/balance, /transactions, /settings, /top-up	120 requests	1 minute
PUT /v1/inference/policy	Shares the 120-per-minute inference-reads budget	1 minute

Revocation gets its own generous per-minute budget on purpose: a leaked key you cannot revoke stays live, so a safety control is never throttled into unavailability by the hourly creation budget.

Size Ceilings Are Not Request Budgets

These are enforced by the request contract, not by a budget, and they return 413 or 400 rather than 429.

Route	Body ceiling	Per-part ceiling
/v1/chat/completions, /v1/embeddings	2 MB JSON	—
/v1/images/generations, /v1/images/edits	68 MB JSON	Source image or mask must be under 25 MB decoded
/v1/audio/transcriptions	~34 MB JSON	Audio must be under 25 MB decoded
/v1/audio/speech	1 MB JSON	—
/v1/chat/completions documents per message	—	capabilities.max_documents_per_message on /v1/models (refused with 400 too_many_documents)

The image and audio body ceilings are derived from the per-part ceiling rather than picked independently: base64 inflates by 4/3, and an image-edit request may carry both a source image and a mask.

• A body over the ceiling returns 413 with code: "invalid_request".
• A body that is not valid JSON returns 400 with code: "invalid_request".
• A message carrying more document parts than the model’s capabilities.max_documents_per_message returns 400 with code: "too_many_documents" and param: "$.messages[i].content". The bound is per message, so send the rest in a later message. A model without the field publishes no bound.

Retrying Correctly

import time, random, requests

def call(url, **kw):
    for attempt in range(6):
        r = requests.post(url, **kw)
        if r.status_code != 429:
            return r

        # Every 429 states its own wait when it knows one: the time until a
        # request leaves the sliding window for the per-minute bound, the
        # expected wait for a slot for the concurrency bound, the provider's
        # reset for upstream_rate_limited. Fall back to local backoff only when
        # the header is absent.
        wait = float(r.headers.get("Retry-After", 2 ** attempt))
        time.sleep(wait + random.uniform(0, 1))  # jitter

    raise RuntimeError("rate limit not cleared")

Three rules that matter in practice:

1. Add jitter. A fleet that all reads the same Retry-After and wakes together re-exhausts the budget instantly.
2. Bound your own concurrency. The standard tier is 31 in flight for the whole account. A worker pool wider than that will spend its surplus requests collecting concurrency_limit_exceeded.
3. Branch on error.code, never on message. Messages are written for people and can be reworded; code, retryable, and the HTTP status are the contract.

Next Steps

• AI Gateway Overview: base URL, auth, quickstart, streaming
• Compatibility: the supported surface and every error code
• Rate Limits: the separate budgets for the varity.app/api platform API