AI Gateway OpenAI Compatibility

Varity TeamCore Contributors

“OpenAI-compatible” is a claim about a surface, not a promise about all of it. This page states the boundary. An exclusion you discover at runtime instead of reading it here is a documentation bug. If you hit one, report it.

Everything below was checked against the live gateway at https://ai.varity.app/v1, its canonical host, and against the contract it publishes at GET /v1/openapi.json. The older varity.app/v1 alias serves the same surface; see Endpoints.

Endpoints

Endpoint	Status
POST /v1/chat/completions	Supported. Requires a bearer API key. Streaming and tool-call generation on eligible models.
GET /v1/models	Supported. Public, no key required.
POST /v1/embeddings	Supported. Requires a bearer API key. Only a model whose capabilities.output_modalities includes embedding is routable.
POST /v1/images/generations	Supported. Requires a bearer API key. Image-capable models only; results are returned inline as base64 and never stored.
POST /v1/images/edits	Supported. Requires a bearer API key. The OpenAI SDK’s multipart/form-data upload, or JSON with base64 image data (image/png, image/jpeg, image/webp); a source image or mask at or above 25 MB is rejected.
POST /v1/audio/transcriptions	Supported. Requires a bearer API key. The OpenAI SDK’s multipart/form-data upload, or JSON with the audio as a base64 typed part; the model must advertise capabilities.transcription. Returns JSON, or plain text when response_format is text. No streaming mode.
POST /v1/audio/speech	Supported. Requires a bearer API key. The model must advertise capabilities.speech and the format must be one of its capabilities.speech_formats. Returns audio bytes with a matching audio/* content type; stream: true gives a chunked binary response, not SSE.
GET /v1/models/{id}	Not implemented. Returns 404. Fetch the list and filter client-side.
POST /v1/completions (legacy text completions)	Not implemented. Returns 404.
POST /v1/responses	Not implemented. Returns 404.
/v1/assistants	Not implemented. Returns 404.
/v1/files	Not implemented. Returns 404.
/v1/vector_stores	Not implemented. Returns 404.
POST /v1/moderations	Not implemented. Returns 404.
/v1/batches	Not implemented. Returns 404.
/v1/fine_tuning/jobs	Not implemented. Returns 404.

Beyond the OpenAI surface, the gateway serves its own endpoints. These have no OpenAI equivalent, so a client written against the OpenAI spec will not find them, but they are public and stable.

Endpoint	Status
GET /v1/openapi.json	Supported. Public, no key required. The machine-readable contract for this surface, including the data-handling block below.
GET /v1/inference/*	Supported. Requires a bearer API key. Policy, usage, per-request metadata, and latency; listed on Endpoints.

The host also serves account routes for keys, the prepaid balance, and the privacy floor. They take a Developer Portal session, not an API key, and are listed on Billing and Keys.

Hosted tool execution, batch, files, assistants, fine-tuning, and moderation are not part of the surface. Unimplemented routes return 404 with the JSON error envelope below (code: "invalid_request", message Route not found.). Branch on the HTTP status code, not on the message text.

Unauthenticated GET /healthz and GET /readyz probes are available outside /v1 on the canonical host; see Endpoints.

Caution

Any OpenAI SDK helper that touches an unimplemented route will fail, including the Assistants and Responses APIs, file upload helpers, and batch. The chat.completions, models, embeddings, images.generate, images.edit, audio.speech, and audio.transcriptions namespaces work unchanged, on the models whose capabilities advertise them.

Per-Model Capabilities

Compatibility is not uniform across models. Each catalog entry carries a capabilities object, and you must read it. The same request that works against one model can be unsupported on another.

Capability	Models supporting it
chat_completions	The chat models; image, audio, and embedding models report false
streaming	Chat models reporting streaming: true
tool_calls	Chat models reporting tool_calls: true
streaming_tool_calls	Chat models reporting streaming_tool_calls: true
output_modalities	text, embedding, image, or audio: the route a model belongs on
transcription, speech	The audio models, each on exactly one of the two audio routes

Check tool calling per model

Most chat models accept tools / tool_choice, but not all: a model reporting tool_calls: false rejects them. Resolve this at runtime rather than hardcoding a list:

models = {m["id"]: m for m in client.models.list().model_dump()["data"]}

def supports_tools(model_id, streaming=False):
    caps = models[model_id]["capabilities"]
    return caps["streaming_tool_calls"] if streaming else caps["tool_calls"]

Note

The OpenAI Python SDK exposes only the fields it knows about on typed objects. Varity’s extra fields (capabilities, context_window, customer_pricing) arrive as extra JSON, so read the raw dict (or call /v1/models directly with an HTTP client) rather than expecting SDK attributes for them.

Model Identifiers

Varity model ids are not guaranteed to be OpenAI model ids. Some entries now carry OpenAI-style names such as gpt-4o, but every id is a Varity catalog entry with Varity routing and pricing, and there is no aliasing layer that maps an OpenAI name onto a different model. A request naming an id that is not in the live catalog is a request for a model that does not exist here.

Ids are readable human slugs, for example glm-5-1, glm-5-2, llama-3-3-70b, and qwen-3-5-35b-a3b. Always take ids from the live catalog.

Limits That Differ From OpenAI

• max_completion_tokens is per-model and often small. The ceiling is independent of the context window, so a very large context does not mean a long reply is allowed. Read both fields per model.
• Cached-input pricing is per model. Some models price cached_input below input and others price them the same. Read customer_pricing.cached_input per model.
• There is a concurrency bound as well as a rate limit. Both are per-account tier limits, and a key can carry its own requests_per_minute, concurrency_limit, and spending budget (the smaller of the key’s and the account’s bound applies). See Rate Limits and Billing and Keys.
• GET /v1/models carries no request budget and returns no RateLimit-* headers. The headers live on the inference and inference-read routes, not on catalog discovery.

Request Fields

POST /v1/chat/completions handles these OpenAI request fields as follows. A field it rejects returns 400 naming that field.

Field	Behavior
user	Accepted and dropped; end-user ids never leave Varity
developer role	Treated as system
response_format.json_schema.description	Accepted and forwarded
n	1 only; any other value returns 400
frequency_penalty, presence_penalty	0 only; any other value returns 400
logprobs	false only; true returns 400
top_logprobs	Always returns 400

Errors

The error envelope matches OpenAI’s shape and adds two fields:

{
  "error": {
    "message": "The provided credential is invalid or expired.",
    "type": "authentication_error",
    "code": "authentication_required",
    "param": null,
    "retryable": false,
    "request_id": "504f4a24-1427-4483-89fe-ad332fdad5d6"
  }
}

• retryable is a boolean OpenAI does not send. Use it instead of guessing from the status code whether to retry.
• request_id is also returned as the x-request-id response header on every response, successful or not. A caller-supplied x-request-id that is a canonical UUID is echoed back as the request id; any other value is replaced by a UUID the gateway generates.

A managed-inference failure adds a diagnostic block naming who decided the failure and where in the request’s life it happened:

Field	Meaning
diagnostic_code	ai_gateway.<code>.
failure_cause	The same value as code.
failure_domain	varity_projection (Varity refused), supplier_observed (a model provider answered and Varity observed it), or dependency_observed (a Varity-operated dependency was unreachable, named in dependency).
lifecycle_stage	admission, provider_attempt, settlement, or terminal.
action	What to do next, in one sentence.

Error Codes

Branch on code. error.retryable is on every body and is authoritative for that response; the column below is what the gateway publishes today.

Varity refused the request (failure_domain: "varity_projection"):

Code	HTTP	Retryable	Cause
authentication_required	401	no	No bearer credential, or one that is invalid, revoked, or expired.
invalid_request	400	no	The body failed validation. Also 404 on an unimplemented route, and 413 when the body exceeds the route ceiling.
input_modality_unsupported	400	no	Every route for this model rejected the supplied image or document part. Distinct from invalid_request so you can tell “this model cannot see” from “this body is malformed”.
too_many_tools	400	no	The request carries more tools than the model’s capabilities.max_tools on /v1/models. Send fewer tools.
too_many_documents	400	no	A message carries more document parts than the model’s capabilities.max_documents_per_message; param names the message ($.messages[i].content). Send the rest in a later message.
model_not_found	404	no	The id is not in the live catalog.
model_not_allowed	403	no	The model exists but this credential’s model scope excludes it.
tenant_not_admitted	403	no	Managed inference is not enabled for the account, or the key is expired.
privacy_policy_violation	403	no	The request asked for a privacy floor weaker than the account’s.
duplicate_request	409	no	The idempotency identity was already admitted.
request_cancelled	499, 408	no	499: the caller closed the connection before the response finished. 408: the request’s deadline had already passed when the gateway admitted it.
deadline_exceeded	504	no	With the optional X-Varity-Deadline-Ms header, a request still running at that instant ends 504 deadline_exceeded. Varity has no total response-time limit: without this header a request runs until the model finishes or the client disconnects. Once a stream has started, the status is already 200 and the error arrives as the final data: event before [DONE].
rate_limit_exceeded	429	yes	The account’s requests-per-minute budget, or the key’s own requests_per_minute, is exhausted. See Rate Limits.
concurrency_limit_exceeded	429	yes	The account’s in-flight bound, or the key’s own concurrency_limit, is exhausted. See Rate Limits.
budget_exceeded	429	yes	The account’s inference budget, or the key’s own spending budget, is exhausted. See Billing and Keys.
internal_error	500	no	An unclassified gateway failure. Quote request_id.

Billing refusals (402) and billing outages (503) use the codes listed in API Errors.

A model provider decided it (failure_domain: "supplier_observed"):

Code	HTTP	Retryable	Cause
model_unavailable	503	yes	No callable route for this model right now. Choose another model or retry.
privacy_route_unavailable	403	no	The model has no callable route at the required privacy level.
upstream_rate_limited	429	yes	The provider is rate-limiting. Carries Retry-After only when the provider published its own reset.
upstream_unavailable	503	yes	No qualified route answered.
upstream_balance_exhausted	varies	no	Provider funding is exhausted. Varity’s problem, not yours; pick another model.
upstream_timeout	varies	yes	The provider did not answer in time. Retry only if the request is known not to have taken effect.
stream_terminated	—	no	A stream ended mid-flight. Do not replay an ambiguously chargeable stream; inspect the correlated request instead.

Note

upstream_authentication_failed exists in the gateway’s internal vocabulary but never reaches you. A refused Varity provider credential is something only Varity can fix, so it is published as model_unavailable — the one thing a caller can act on — and the real cause stays in the operator record.

Authentication is evaluated before request-body validation, so an unauthenticated request with an invalid model returns the authentication error, not a model error. Do not use the error type to infer that your payload was accepted.

Caution

These codes are the AI Gateway’s own. The platform API at varity.app/api uses a different envelope and a different code vocabulary — see API Errors. In particular the AI Gateway never emits invalid_api_key; an invalid key here is authentication_required.

Privacy And Data Retention

The gateway publishes its data-handling contract as an x-varity-data-handling block at the root of GET /v1/openapi.json. It is machine-readable, so you can assert against it in CI rather than taking a page’s word for it.

Field	Value
inference_metadata_retention_days	30
stored_content	[]
content_never_stored	prompts, responses, tool_arguments, uploaded_content, generated_image_bytes, playground_conversations
generated_image_retention	inline_only
financial_and_security_records	separate_retention_classes

Prompt, response, uploaded, and generated-image bytes are never written to the inference records. What is retained for 30 days is request metadata: model id, token counts, latency, fallback count, error code, and timestamps. Read it back with GET /v1/inference/requests.

This is zero data retention (ZDR) for Varity’s own records. What the model provider does with a request depends on the model’s privacy class; see Privacy Routing.

Privacy Routing

The live catalog reports two routing privacy classes: privacy: "private" with privacy_routes: ["private"], and privacy: "anonymized" with privacy_routes: ["anonymized"]. This field has no OpenAI equivalent; it is Varity-specific catalog metadata describing how each model is routed. Read it per model.

Your account has a privacy floor. A new account starts at anonymized, with nothing to acknowledge, which lets requests use both private and anonymized models. Set it to private with PUT /v1/inference/policy (or on the portal’s Gateway settings tab) and only private models are callable. A PUT that sets it to anonymized carries acknowledge_anonymized_processing: true. A single chat request can tighten the floor, never loosen it, with "varity_extensions": { "privacy_floor": "private" }; asking for a weaker floor than the account’s returns 403 privacy_policy_violation.

Next Steps

• AI Gateway Overview: base URL, auth, quickstart, streaming
• Models: the full catalog with limits and prices
• Public API Reference: the /api platform contract