Inference reference

The Model API carries inference traffic for your company. The separate Partner interface manages credentials and reads owner-level operational records. Select the interface by the operation you are performing, not by which token happens to be available.

Protocol families

Use Messages for Anthropic-style conversations, Chat Completions for message-list clients, or Responses for typed input/output items and supported hosted tools. Embedding and image calls require models/routes with those capabilities.

Operations

The endpoint index lists paths. Streaming covers progressive output. Deferred inference, Messages batches and Files/Batch are separate execution modes. Use error diagnostics and health probes when investigating availability.

Every example uses illustrative identifiers and model choices. Replace them with the canonical ID or alias returned by your authenticated catalog. Catalog availability is the authority for a configured deployment, not a copied example.

Conversation identity and sticky routing

A Session ID identifies one conversation. Sticky routing is a separate, best-effort preference for an eligible upstream route or account. Model API accepts session identity independently of the selected provider or model on its common JSON inference path, including Chat Completions, Messages and Responses. A provider-specific transport may use that identity when the selected route supports it; unsupported providers do not receive an unknown session_id field.

The first non-empty value wins: JSON session_id, then the X-Session-ID header, then a recognized session in metadata.user_id, then JSON prompt_cache_key. Without these signals, Model API derives an identity from the opening system/developer content and first user content. Appending later messages, including a history growing from 41 to 43 messages, does not change that opening identity. No client changes are required for this fallback.

A preexisting session in metadata.user_id is a compatibility signal, not a conflict. Recognized forms are session_<id>, user_<device>_account_<account>_session_<id> and a JSON string containing a nonempty string session_id. The session component is used only when the higher-priority fields are absent. For example, session_chat-42 supplies the conversation ID chat-42. On a route that uses this metadata transport, only its session component is scoped; device, account and other metadata fields are retained. Different explicit fields follow the stated priority instead of causing a mismatch error.

An ordinary value such as customer_42 is not automatically a conversation ID. It is preserved without a mismatch error. A route whose session transport uses this occupied field cannot also insert a managed session there; internal routing still works, but no external account binding or cache hit is promised. Use a unique session_id or X-Session-ID for precise conversation identity and a compatible upstream transport. The optional metadata envelope parser accepts up to 4096 UTF-8 bytes and validates its session component under the same ID rules; unknown or ambiguous string envelopes remain opaque metadata.

For precise separation, generate a unique ID per conversation and reuse it for subsequent turns. Identical openings can share an automatic identity; changing or removing the opening content can change it. Use an explicit ID across history compaction or protocol changes. Non-empty IDs must be strings of at most 512 UTF-8 bytes without control characters; surrounding whitespace is trimmed. Sending multiple X-Session-ID headers is invalid.

{"session_id":"project-chat-123"}

Add this field to a complete inference request, or send X-Session-ID: project-chat-123 instead. Do not place credentials or personal information in the ID. Identity is isolated by authenticated account, API key and canonical model; changing one of these changes the routing scope. Streaming and supported asynchronous submission preserve the selected identity.

Session identity is not authentication, an idempotency key or server-side conversation storage. Continue sending the context required by the chosen protocol. It does not guarantee a cache hit or a permanently fixed account, and it never bypasses access, balance or request limits. Eligible failover remains available. Stateful Responses requests with previous_response_id retain their stricter routing safeguards; do not rely on an opening-content guess to recover missing state.

Stable opening and separate upstream transport

The private automatic hash ignores cache_control decorations on known content blocks and prompt_cache_breakpoint on text/input_text blocks and treats a single plain text block as the same text string. These normalizations do not modify the request sent to the model. Text whitespace, image data/details, unknown properties and nested tool JSON remain meaningful. Changing the opening itself or the protocol can still change automatic identity; an explicit ID is needed across compaction or protocol changes. A recognized metadata session is stronger than a cache hint, so changing prompt_cache_key does not split that conversation.

Model API does not patch Sub2API and does not use a private receiver protocol. For a Sub2API OpenAI/Codex credential, the managed session is translated to the existing OpenAI-compatible signals used by that branch: a scoped Session_id header and prompt_cache_key. For a Sub2API Anthropic/Claude credential, a recognized session is carried through metadata.user_id; an ordinary occupied user ID is preserved and cannot simultaneously carry the managed session. Client session_id / X-Session-ID remain public ingress signals to Model API, not a claim that X-Session-ID is a universal Sub2API egress contract. Split mixed credentials by the actual protocol group instead of inventing a common transport.

The gateway may provisionally prefer the same eligible route for concurrent first requests, then confirm it after success. This is not permission to exceed limits, bypass Sub2API account eligibility, restore missing Responses state, or guarantee a cache hit. Newer placements cannot be overwritten by an older request completing later.

A provisional route preference remains only while at least one request from that placement is still in flight. There is no separate 60-second routing deadline: the existing request lease is the zombie guard. The last failed in-flight owner releases that provisional placement, while a still-eligible confirmed route takes precedence. These safeguards do not retry a paid request or select an account inside an external gateway.