OpenRouter has launched a Batch API for asynchronous LLM inference, adding a new cost surface for teams that can wait for results instead of paying for immediate synchronous responses.
The September 22 release lets users submit batch workloads to /api/v1/batches for chat completions, responses, messages, and embeddings. OpenRouter says the feature works across more than 70 models. It also says providers generally charge 50% or less of normal per-token pricing when they can complete work within a 24-hour window.
That makes the announcement more than another endpoint. It formalizes a split that many AI teams already manage informally: some model calls need to happen while a user is waiting, while many others can run in the background. Evals, classification jobs, embeddings, document enrichment, labeling, backfills, red-team test suites, and large summarization queues often do not need low-latency responses. Until now, those jobs were frequently pushed through the same synchronous API patterns as interactive product features.
What changed
OpenRouter’s Batch API creates a separate path for work that can be queued and completed asynchronously. Instead of making one request and holding open a response path, users submit a batch, track its status, and retrieve the completed outputs later.
The practical change is economic as much as technical. OpenRouter’s announcement says providers generally price these workloads at half or less of standard per-token rates if they can complete the work within the 24-hour window. The company also reported that its beta handled more than 230,000 completed batches with a 7-minute median completion time.
Those numbers should not be read as a universal service-level guarantee. The available discount depends on provider support and routing, and completion time will vary by model, queue depth, workload shape, and provider capacity. But the direction is clear: API platforms are starting to expose latency as a deliberate cost-control lever.
Why it matters for AI teams
For developers, batch processing changes how LLM work should be designed. A product team may still use synchronous calls for chat, code assistance, agent tool use, or customer-facing workflows. But the same team can move recurring evaluation runs, nightly knowledge-base processing, embeddings generation, and offline analytics into a lower-cost queue.
That matters because token spend is increasingly split across many hidden operational tasks. A company may notice the cost of visible AI features first, but internal jobs can become a large share of inference volume once teams start running evals, quality checks, content moderation, dataset transformations, and retrieval maintenance at scale. Batch APIs make those background workloads easier to separate, budget, and optimize.
The release also reinforces a broader pattern in AI API cost control: enterprises need more than a list of supported models. They need routing, observability, policy, usage analytics, and billing logic that understand the operational difference between interactive and deferred inference.
For teams using a multi-B2B LLMway, batch inference raises immediate governance questions. Which providers are eligible for batch jobs? Are some models approved only for synchronous user-facing use, while cheaper models are preferred for offline evaluation? How are failed batch items retried? Can finance or platform teams attribute asynchronous token usage to the right product, customer, environment, or internal team?
Who is affected
The most obvious beneficiaries are teams with large volumes of latency-tolerant LLM work. AI evaluation teams can run broader test suites without treating every prompt as a real-time transaction. Data teams can process backlogs of support tickets, documents, transcripts, or product catalogs at lower cost. Retrieval teams can generate embeddings and metadata in the background. Security and governance teams can run policy checks or audit workloads without competing with production latency budgets.
Startups and smaller teams may also benefit because batch pricing can make experimentation cheaper. A team comparing several models for summarization or classification could run larger offline tests before committing to a production route. That said, cost savings are not automatic. If a workflow still blocks a user, or if downstream systems are not built to poll for completion and handle partial failures, forcing it into a batch model may create more complexity than value.
For platform and infrastructure teams, the announcement points to a product requirement that is likely to spread. As batch pricing becomes more common, internal AI platforms will need separate controls for asynchronous jobs: quotas, approval flows, retention rules, cancellation behavior, status monitoring, and cost attribution. A single aggregate token counter will not be enough.
What remains uncertain
The main uncertainty is how consistent the savings will be across models and providers. OpenRouter describes the 50% level as a general provider pattern, not a promise that every model or route will always be available at that discount. Buyers will need to check which models support batch mode, how pricing is presented in logs and invoices, and whether routing changes alter the expected economics.
There is also an operational question around completion windows. A 24-hour window is reasonable for many offline workloads, but businesses still need to know whether most jobs finish in minutes, hours, or near the deadline. OpenRouter’s reported 7-minute median beta completion time is encouraging, but median performance does not answer tail-latency questions for large production batches.
The bigger significance is that LLM infrastructure is becoming less monolithic. The old model was simple: send a prompt, get a response, pay the listed token price. The emerging model is more like cloud infrastructure pricing, where timing, routing, workload class, provider choice, and policy all affect cost. OpenRouter’s Batch API gives developers a new way to exploit that difference, provided their systems are designed to treat deferred inference as a first-class workflow.