Како направити посредника за ограничење брзине неутралног добављача за мулти-моделне ЛЛМ АПИ-је

When several teams share multiple LLM providers, rate limiting stops being a provider-integration detail. It becomes a shared-capacity problem. One provider may limit requests and input tokens, another may separate input and output tokens, and a third may apply quotas at the project level or across a rolling spend window. A retry loop inside each SDK cannot coordinate those constraints across tenants, applications, partner keys, and model routes.

The practical solution is a central rate limit broker at the gateway. The broker estimates demand, reserves capacity before dispatch, queues or rejects work according to policy, and reconciles the reservation with actual usage after completion. Provider responses then refine the broker’s view of available capacity instead of triggering independent retry storms.

This architecture is useful for any enterprise LLM API that needs predictable throughput across internal teams, production applications, automation agents, or external partners.

Why provider rate limits cannot be reduced to requests per minute

Fact: Provider quotas use different dimensions and scopes. Common dimensions include requests per minute, input tokens per minute, output tokens per minute, total tokens, requests per day, concurrency, spend windows, and acceleration or ramp-rate limits. The same provider can expose different limits for an organization, project, workspace, model, or API product.

For example, provider documentation describes combinations such as:

• Request and token limits with remaining-capacity and reset information in response headers.
• Separate input-token and output-token limits for message APIs.
• Project-level quotas rather than quotas attached to individual API keys.
• Daily request limits that reset at a provider-defined time.
• Rolling spend-based windows that can reject traffic even when request and token rates appear acceptable.
• Acceleration limits that react to a sharp increase in traffic, even when steady-state usage is below the published quota.

A gateway that stores only one number called provider RPM will eventually over-admit traffic in one dimension or underuse capacity in another. It also cannot explain whether a rejection came from a tenant policy, a provider project quota, a daily limit, or an overload response.

Reference architecture for a rate limit broker

Place the broker in the request path before the provider adapter. Every request should pass through the same reservation flow, regardless of whether it arrived through an SDK, an internal service, an automation agent, or a partner API key.

1. Normalize identity. Resolve the organization, tenant, team, application, environment, partner, API key, model route, and workload class.
2. Estimate demand. Calculate expected input tokens, maximum output tokens, estimated spend, and any concurrency cost.
3. Evaluate policy. Check tenant budgets, queue deadlines, route eligibility, priority, and fallback rules.
4. Reserve capacity. Atomically reserve capacity in the relevant internal and provider-facing buckets.
5. Dispatch. Send the request only after the reservation succeeds.
6. Reconcile. Replace estimates with actual usage, release unused reservations, and record provider headers and response classification.
7. Decide the next action. Return the result, queue the work, retry centrally, fall back, or reject it according to the workload contract.

The reservation service should be horizontally scalable, but its decision must be atomic for each request. A distributed counter that allows two workers to observe the same remaining capacity can recreate the provider 429 problem inside the gateway.

Use a canonical quota model without hiding provider differences

Define a canonical model that can represent provider-specific constraints instead of forcing every provider into one abstraction.

Represent each limit as a bucket with a scope, capacity, refill or reset rule, unit, and source. A provider project bucket might contain requests and input tokens. A workspace bucket might contain output tokens. An internal partner policy might add concurrency and spend limits. The admission decision is successful only if all required reservations succeed.

Do not treat provider headers as a complete source of truth. Use them to correct the broker’s estimates, but retain a configured model for limits that are not exposed on every response. Header names, availability, and reset semantics vary by provider and route.

Reserve tokens before dispatch, then reconcile them

Token reservation is the most important difference between a broker and a retry wrapper. The broker needs a demand estimate before sending the request.

For input tokens, estimate from the serialized messages, tool definitions, structured-output schema, and any retrieved context. For output tokens, use the request maximum when available, or a model- and workload-specific estimate when the client does not provide one. Keep the estimate attached to the reservation so it can be compared with actual usage later.

A simple reservation record can contain:

• A unique request and reservation ID.
• Tenant, application, API key, route, and workload class.
• Estimated input tokens, output tokens, total tokens, and spend.
• Every bucket charged by the reservation.
• Creation time, expiry time, queue deadline, and idempotency key.
• Final actual usage and the amount released or charged during reconciliation.

Conservative estimates reduce provider 429 responses but can strand capacity. Measure estimation error by model, route, prompt family, context length, and structured-output mode. Then adjust estimates using bounded corrections rather than allowing a single unusual request to distort the entire model.

Streaming requires a specific policy. You can reserve the maximum output allowance and release the unused portion when the stream ends, or reserve output capacity in smaller increments as tokens arrive. The first approach is easier to reason about; the second can improve utilization but requires reliable cancellation, timeout, and disconnect handling.

Enforce quotas hierarchically

Capacity should be checked from the narrowest policy to the broadest shared resource. A typical hierarchy is:

1. API key or service identity.
2. Application and environment.
3. Team or customer tenant.
4. Partner or reseller account.
5. Provider workspace or subaccount.
6. Provider project or organization.
7. Model and route.

A request from a partner application may therefore need four successful reservations: the partner’s concurrency policy, the customer tenant’s token budget, the gateway’s route allocation, and the provider project’s available quota. If any reservation fails, release the reservations already acquired and apply the failed bucket’s decision.

For fairness, use weighted fair queuing or deficit round-robin rather than allowing whichever worker arrives first to consume the shared pool. Assign higher weights to production traffic and lower weights to background jobs, while preserving burst credits for each tenant. Hard partitioning is easier to explain but can leave capacity idle when one tenant is quiet. Weighted sharing generally uses capacity more efficiently, provided noisy tenants have bounded burst limits.

Design queues around workload contracts

A queue is not automatically a reliability feature. It can turn a visible 429 into an invisible ten-minute delay. Every queue class needs a deadline, cancellation behavior, maximum depth, and overload action.

• Interactive: Short deadline, small queue, predictable rejection or model degradation when capacity is unavailable.
• Production background: Longer deadline, durable queue, idempotency key, and controlled retry budget.
• Partner: Tenant-specific quotas, published response metadata, and no access to another tenant’s queue state.
• Batchable: Delayed execution, larger queue, and optional routing to a provider batch API when the workload allows it.

Queue admission should return a clear decision. Depending on the API contract, expose a 429 or a gateway-specific overload response with a retry-after value, queue identifier, and reason code. Do not promise a completion time unless the system can enforce that promise.

Batch processing can remove noninteractive work from synchronous capacity, but it changes result timing, retry behavior, cancellation, and reconciliation. Treat it as a separate execution path rather than simply placing synchronous requests in a very long queue. For background workloads, compare the operational trade-offs with AI API cost control policies and existing batch operations.

Centralize retry and overload handling

Fact: Temporary rate-limit responses commonly use HTTP 429, while provider overload conditions may use 503 or a provider-specific error code. Some providers return a Retry-After header, and provider documentation recommends honoring it when present. A rapid traffic ramp can also produce acceleration or slow-down behavior without ordinary bucket exhaustion.

Recommendation: Treat Retry-After as a lower bound, not as an optional suggestion. Pause the affected provider scope until that time, add bounded jitter to prevent synchronized wake-ups, and let the broker decide which queued requests can use the recovered capacity.

When Retry-After is absent or invalid, use capped exponential backoff with jitter. Keep the retry budget in the broker, not separately in every SDK and provider adapter. Otherwise, one logical request can produce a gateway retry, an SDK retry, an HTTP client retry, and a provider-side replay.

Classify errors before retrying:

• Retryable capacity: Temporary 429 or documented transient quota response.
• Retryable overload: Temporary 503 or equivalent provider overload signal.
• Non-retryable policy failure: Daily quota, spend cap, disabled project, or tenant limit.
• Non-retryable request failure: Invalid schema, unsupported parameter, authentication failure, or malformed input.
• Deadline failure: A request whose remaining user or workflow deadline is shorter than the next safe attempt.

Fallback is a separate decision. A different provider may have different output behavior, pricing, latency, residency, or context limits. Use fallback only when the route policy confirms that the workload contract can tolerate those differences.

Expose decisions and measure saturation

Teams cannot troubleshoot throttling if the gateway returns only a generic 429. Return safe metadata such as the applicable limit, remaining capacity, reset time, queue decision, workload class, and rejection reason. Avoid exposing another tenant’s usage or raw provider credentials.

At minimum, monitor:

• Reservation success and rejection rate by tenant, route, and bucket.
• Queue depth, queue wait time, deadline expirations, and cancellations.
• Provider 429 and 503 rates, Retry-After frequency, and ramp-rate rejections.
• Estimated versus actual token usage and capacity wasted by over-reservation.
• Concurrency saturation and the share of capacity consumed by each tenant.
• Retry attempts per logical request and fallback frequency.
• Daily quota and spend-window headroom.

Load tests should include burst traffic, long-context requests, maximum output requests, streaming disconnects, daily quota exhaustion, provider 429 and 503 responses, and one noisy tenant competing with production traffic. Validate not only whether requests eventually complete, but also whether queue deadlines, fairness, and retry budgets remain intact.

Implementation checklist

• Define canonical dimensions without discarding provider-specific limits.
• Assign every request a normalized tenant, route, workload class, and idempotency key.
• Estimate and reserve input, output, total tokens, spend, requests, and concurrency before dispatch.
• Make multi-bucket reservations atomic and expire abandoned reservations safely.
• Reconcile actual provider usage and response headers after every attempt.
• Use weighted fairness, burst credits, and explicit priority classes.
• Give every queue a deadline, maximum depth, and overload action.
• Centralize retries and honor provider Retry-After values.
• Separate temporary capacity errors from permanent policy and request errors.
• Publish throttling metadata and monitor saturation by tenant and route.
• Test provider quota exhaustion, ramp limits, streaming, fallback, and noisy-neighbor behavior.

Conclusion

A multi-B2B LLMway should manage provider quota as shared infrastructure capacity, not as a collection of independent SDK settings. A provider-neutral rate limit broker gives platform teams one admission path, one fairness policy, one retry budget, and one place to reconcile estimates with reality.

The design does require more than a generic token bucket. It must represent requests, input and output tokens, daily quotas, spend windows, concurrency, provider scope, and ramp rate. Once those dimensions are explicit, the gateway can make predictable choices: dispatch now, queue with a deadline, shed work, retry centrally, or use a contract-safe fallback. That is how throughput control becomes an operational capability rather than a recurring source of production 429 and 503 incidents.

Related reading

• batch processing for noninteractive LLM workloads