Com aplicar els límits de despesa de LLM a l'API Gateway

Provider spend caps are useful backstops, but they rarely match the way a company actually owns LLM spend. Finance wants limits by team, application, environment, customer, partner, or workflow. Platform teams need those limits enforced before a request fans out to OpenAI, Anthropic, Gemini, Azure OpenAI, or another model provider. The right control point is usually the API gateway, where every request can be attributed, estimated, admitted, downgraded, blocked, and reconciled.

This guide describes a gateway-layer implementation for AI API cost control. It focuses on real-time budget enforcement, not retrospective chargeback reports or broad cost optimization programs.

What Provider Spend Controls Can and Cannot Do

Fact: Major providers expose useful controls, but they operate at provider-defined boundaries. OpenAI documents monthly API spend limits at the organization and project level, along with distinct errors for organization limits, project limits, assigned usage limits, and exhausted prepaid credits. Anthropic documents workspace spend and rate limits, and also provides APIs to retrieve configured organization and workspace limits for internal alerting. Google Cloud budget alerts alone do not automatically cap spend, while Google also documents spend-capped budgets for eligible services such as Gemini API and Agent Platform. Azure OpenAI quota is assigned per subscription, region, model, and deployment type in tokens-per-minute units, and rate-limit calculations can include a prompt estimate and max tokens, not only final billed tokens.

Recommendation: Keep provider-native spend caps enabled as a last-resort guardrail. Do not treat them as the main mechanism for enforcing internal budgets. They are too coarse for most ownership models, and they usually trigger after the request path has already left your control.

Prediction: Provider controls will keep improving, but enterprises will still need gateway-level policy because internal cost ownership changes faster than provider account structures.

Define the Budget Policy Model First

Start with policy, not code. A gateway cannot enforce a budget that has not been described precisely.

A practical policy record should include:

• Budget owner: team, department, customer, partner, application, or workflow owner.
• Scope: the dimensions that match requests, such as team_id, app_id, environment, customer_id, and workflow_id.
• Reset interval: daily, weekly, monthly, contract period, or custom window.
• Soft threshold: the point where the gateway warns, downgrades, reduces output length, or routes to cheaper models.
• Hard threshold: the point where noncritical traffic is blocked.
• Allowed models: the models this scope may use.
• Fallback models: cheaper or lower-capacity models allowed after a soft threshold.
• Criticality: whether the workload is production-critical, user-facing, internal, batch, or experimental.
• Emergency override: who can extend the limit, for how long, and with what audit trail.
• Notification targets: Slack channel, email group, incident service, webhook, or finance queue.

Example policy:

{
  "budget_id": "budget_support_prod_monthly",
  "owner_type": "team",
  "owner_id": "customer_support",
  "scope": {
    "team_id": "customer_support",
    "app_id": "support_assistant",
    "environment": "production"
  },
  "currency": "USD",
  "limit": 5000.00,
  "reset_interval": "monthly",
  "soft_threshold_percent": 80,
  "hard_threshold_percent": 100,
  "allowed_models": ["premium-chat", "standard-chat", "economy-chat"],
  "fallback_models": ["standard-chat", "economy-chat"],
  "criticality": "user_facing_production",
  "on_soft_limit": ["notify", "downgrade_model", "reduce_max_output_tokens"],
  "on_hard_limit": ["block_noncritical", "allow_emergency_override"],
  "notification_targets": ["slack:#support-platform", "email:[email protected]"]
}

Require Ownership Metadata on Every Request

Budget enforcement depends on attribution. If a request cannot be mapped to a budget scope, the gateway cannot make a reliable admission decision.

Require internal clients to send stable ownership metadata:

{
  "model": "premium-chat",
  "messages": [...],
  "max_tokens": 1200,
  "metadata": {
    "team_id": "customer_support",
    "app_id": "support_assistant",
    "environment": "production",
    "customer_id": "cust_48291",
    "workflow_id": "ticket_reply_draft"
  }
}

Recommendation: Roll this out in phases. First log unattributed traffic. Then warn on it. Then reject unattributed production traffic after owners have had time to update clients. For development traffic, quarantine it into a default budget with a low cap rather than mixing it with production spend.

Also validate metadata against your identity system. A client should not be able to claim another team’s budget by changing team_id. Tie API keys, service accounts, or partner credentials to allowed budget scopes.

Estimate Cost Before Dispatch

Preflight estimation is the core of real-time enforcement. The gateway estimates the maximum likely cost before it sends the request to a provider.

The estimator needs:

• Model price tables for input, output, cached input, batch, tool use, images, embeddings, or other billable dimensions.
• Tokenizer or token estimation logic per model family.
• Prompt token count or estimate.
• Requested max_tokens or maximum output length.
• Cache assumptions, if prompt caching affects billing.
• Provider-specific billing rules for retries, tool calls, reasoning tokens, image generation, or multimodal input.

Example estimator output:

{
  "request_id": "req_7f1b",
  "budget_scope": "team:customer_support/app:support_assistant/env:production",
  "model": "premium-chat",
  "input_tokens_estimated": 1850,
  "max_output_tokens": 1200,
  "estimated_cost": 0.0845,
  "estimate_basis": "prompt_tokens_plus_max_output_tokens",
  "confidence": "conservative"
}

Recommendation: Treat preflight cost as a conservative admission estimate, not accounting truth. If the user asks for max_tokens: 4000, reserve against that upper bound even if the response usually stops at 700 tokens. This reduces overspend risk, but it can reject requests that would have completed under budget. That trade-off is preferable for strict finance controls and less attractive for exploratory internal tools.

Reserve Budget Before Sending the Request

A common failure mode is checking the budget, finding available balance, and then allowing many concurrent requests to pass before spend is recorded. The fix is reservation.

The admission sequence should be:

1. Authenticate the caller.
2. Validate ownership metadata.
3. Resolve the applicable budget policy.
4. Estimate request cost.
5. Atomically reserve that estimated cost from the budget store.
6. Dispatch the request to the provider only if the reservation succeeds.
7. Release unused reservation after completion.
8. Record final estimated or provider-reported usage.

The reservation store must support atomic updates. Redis with Lua scripts, PostgreSQL transactions, or a dedicated ledger service can work. The important property is that two requests cannot spend the same remaining budget.

Example reservation record:

{
  "reservation_id": "res_9d42",
  "request_id": "req_7f1b",
  "budget_id": "budget_support_prod_monthly",
  "estimated_cost": 0.0845,
  "state": "reserved",
  "expires_at": "2026-09-28T12:45:00Z"
}

Use short reservation expirations so abandoned requests do not lock budget forever. A background sweeper should release expired reservations that never reached a terminal state.

Separate Soft Limits From Hard Limits

Soft and hard limits should behave differently. A soft limit is a policy signal. A hard limit is an admission decision.

Soft Limit Behavior

When a scope crosses its soft threshold, the gateway can still serve traffic while reducing spend pressure. Useful responses include:

• Send alerts to the budget owner and finance or platform channel.
• Add response headers that show limit state and remaining budget.
• Route noncritical requests to cheaper models.
• Reduce max_tokens for verbose workflows.
• Disable expensive optional features such as multi-sample generation.
• Require approval for new experiments under that budget scope.

Example soft-limit response metadata:

{
  "budget_scope": "team:customer_support/app:support_assistant/env:production",
  "limit_state": "soft_limit_exceeded",
  "estimated_cost": 0.0310,
  "remaining_budget": 912.44,
  "applied_policy": ["downgrade_model", "reduce_max_output_tokens"],
  "selected_model": "standard-chat"
}

Hard Limit Behavior

At the hard threshold, the gateway should block noncritical traffic with a predictable error contract.

{
  "error": {
    "type": "budget_limit_exceeded",
    "code": "llm_budget_hard_limit",
    "message": "The budget for this scope has been exhausted.",
    "budget_scope": "team:customer_support/app:support_assistant/env:production",
    "retry_after": "2026-10-01T00:00:00Z",
    "override_required": true
  }
}

Do not return provider-shaped errors for gateway budget failures. Internal clients need to distinguish provider rate limits, provider spend caps, gateway spend limits, authentication failures, and policy denials.

Handle Streaming Requests Explicitly

Streaming creates a different enforcement problem. The request is admitted before the gateway knows final output tokens, and generated tokens can continue accumulating cost while the stream is open.

A workable pattern is:

1. Reserve a conservative amount based on prompt tokens and maximum output tokens.
2. Track generated tokens during the stream when possible.
3. Stop generation if the reserved amount is exhausted.
4. Return a finish reason the client can handle.
5. Reconcile actual usage after the provider sends final usage metadata or after gateway estimation completes.

Example stream termination event:

{
  "type": "response.completed",
  "finish_reason": "budget_exhausted",
  "budget_scope": "customer:cust_48291/workflow:ticket_reply_draft",
  "estimated_cost": 0.0845,
  "remaining_budget": 0.00
}

Trade-off: Cutting off a stream protects the budget, but it may produce incomplete output. For workflows that require complete structured responses, prefer preflight blocking or model downgrade before streaming begins. For chat-style experiences, a budget-aware finish reason may be acceptable if the client can ask the user to retry later or continue with a cheaper model.

Expose Budget State to Clients

Budget enforcement is easier to adopt when callers can see what happened. Add response headers or metadata for internal clients and partner APIs.

Useful fields include:

• budget_scope
• limit_state: normal, soft_limit_near, soft_limit_exceeded, hard_limit_exceeded
• estimated_cost
• reserved_cost
• final_cost_when_available
• remaining_budget
• applied_policy
• selected_model

For public partner APIs, avoid leaking internal team names or cost basis if that is commercially sensitive. Expose tenant-safe quota state instead, such as remaining_quota, limit_state, and retry_after.

Reconcile Estimates With Provider Usage

Request-time enforcement and accounting-grade cost reporting are related, but they are not the same thing. The gateway estimate makes an admission decision under uncertainty. Provider billing exports, usage APIs, and invoices provide the record that finance will care about.

Recommendation: Reconcile gateway-estimated spend with provider-reported usage at least daily. Differences can come from tokenizer mismatch, prompt caching, batch discounts, retries, failed calls, provider-specific rounding, image or tool-use billing, and delayed billing exports.

A reconciliation job should:

1. Import provider usage and cost data.
2. Normalize it into the gateway’s usage schema.
3. Match provider records to gateway request IDs where available.
4. Allocate unmatched provider spend to the most specific known scope.
5. Adjust budget ledgers for over- or under-reservation.
6. Emit variance reports by provider, model, team, application, and environment.

Use normalized telemetry fields for input tokens, output tokens, model, provider, request ID, and status. OpenTelemetry GenAI semantic conventions define token usage attributes that can help standardize usage telemetry across providers.

Choose Fail-Open or Fail-Closed Behavior Deliberately

The budget store becomes part of the request path. That is the main architectural cost of real-time enforcement.

If the budget store is unavailable, the gateway needs a defined behavior:

• Fail open: allow requests and mark them for later reconciliation. This protects availability but can overspend.
• Fail closed: block requests until budget state is available. This protects spend but can break production workflows.
• Fail constrained: allow only critical production traffic, use cheaper models, apply low per-request caps, and block experiments or batch workloads.

Recommendation: Use fail constrained for most business systems. Critical production workflows continue with reduced cost exposure, while discretionary workloads pause until the control plane recovers.

Implementation Checklist

• Define budget scopes and ownership metadata.
• Bind API keys or service accounts to allowed scopes.
• Create policy records with soft limits, hard limits, allowed models, fallbacks, overrides, and notification targets.
• Build a provider-aware cost estimator.
• Use atomic budget reservations before provider dispatch.
• Release unused reservations after completion.
• Implement hard-limit errors with a stable contract.
• Add streaming token tracking and budget-aware finish reasons.
• Expose budget state through internal headers or response metadata.
• Keep provider spend caps enabled as backstops.
• Reconcile gateway estimates with provider usage and cost reports daily.
• Alert on variance, unattributed traffic, exhausted budgets, and override usage.

Actionable Conclusion

To enforce LLM spend limits reliably, put the decision where requests enter your system. Provider caps can stop account-level damage, but gateway policy can enforce the business limits leaders actually ask for: team, app, environment, customer, partner, and workflow budgets.

The minimum viable design is straightforward: require ownership metadata, estimate cost before dispatch, reserve budget atomically, apply soft and hard thresholds, handle streaming separately, and reconcile with provider usage data. The hard part is not the first block decision. The hard part is making the system predictable enough that finance trusts it and production teams can keep operating when a budget is under pressure.

Related reading

• per-team LLM cost ledger