Како изградити рутирање ЛЛМ модела са свешћу о трошковима без жртвовања квалитета

Cost-aware LLM model routing should not mean sending every request to the cheapest available model. That creates a quiet failure mode: invoices improve for a few weeks, then product teams discover broken schemas, weaker reasoning, inconsistent tone, or support drafts that require more human cleanup.

The better pattern is an eval-backed routing policy. Classify the request, define the minimum acceptable quality for that task, remove routes that cannot satisfy capability or governance constraints, estimate the route economics, and then choose the lowest-cost eligible model. The routing decision becomes explainable during a cost review because it is tied to a policy version, not to developer preference or a vague provider default.

What Cost-Aware Routing Is Optimizing

A practical router optimizes for the cheapest route that still meets the task's operating requirements. That is different from optimizing for the cheapest model.

Facts: major model providers publish token pricing with separate dimensions such as input tokens, cached input tokens, output tokens, prompt caching, and batch processing. Some providers also expose usage fields that report cached-token behavior. This means the lowest-cost option can change based on prompt shape, expected output length, cache eligibility, and whether the workload can run asynchronously.

Recommendation: treat routing as a policy problem inside a multi-model AI API gateway, not as an application-level switch statement. The gateway sees provider capability records, tenant rules, API key metadata, price tables, usage telemetry, and fallback behavior in one place.

Prediction: teams that keep routing logic scattered across application repositories will struggle to explain cost changes later. They may still reduce spend, but they will have weaker evidence for why a given request used a given model.

Start With Task Classes, Not Provider Names

The router needs a workload label before it can make a useful decision. A single default model cannot represent the risk profile of every request.

Useful task classes include:

• Extraction: convert unstructured text into fields, often with strict schema validation.
• Classification: assign labels, priority, intent, sentiment, or routing categories.
• Summarization: compress long input while preserving key facts and decisions.
• Customer support drafting: generate a reply that a human may review or send.
• RAG answer synthesis: answer from retrieved context with citation or grounding requirements.
• Code review: inspect diffs or source files for correctness, risk, or style issues.
• Long-context analysis: reason over contracts, transcripts, logs, policies, or research packages.
• High-impact decision support: any workflow where a bad answer could affect money, compliance, safety, or customer trust.

The task class can come from explicit request metadata, an API key scope, an application route, or a lightweight classifier. Explicit metadata is usually better than inferring intent from prompt text, because vague prompts force the router toward conservative defaults.

Attach a Quality Floor to Each Task

Cost-aware routing only works if each task has a quality floor. Without one, the router has no principled way to reject a cheaper model.

A quality floor can include several checks:

• Replayable eval set: representative examples from development and production, stripped of sensitive data where required.
• Schema validity: output must parse, validate against the canonical schema, and avoid unsupported fields.
• Content criteria: answer must follow style, refusal, grounding, and completeness rules for the task.
• Safety criteria: output must not leak hidden instructions, ignore policy boundaries, or execute untrusted instructions from retrieved content.
• Human review: required for high-impact workflows or for promotion of a new route.
• Regression threshold: candidate route must meet or exceed the current route on critical failure categories, not only average score.

Fact: provider documentation describes evaluations as a way to test model outputs against specified content and style criteria. One hosted eval product is scheduled for deprecation, so routing systems should store provider-neutral eval datasets, expected outputs, grading rubrics, and results rather than depending on one provider-hosted eval surface.

Recommendation: store eval results by task class, model, provider adapter version, prompt version, schema version, and eval version. A route should not become eligible simply because a model is new or cheaper. It becomes eligible when it passes the task's current quality gate.

Build a Provider Capability Registry

Before estimating cost, remove any route that cannot satisfy the request. This step prevents the router from selecting a cheap model that lacks required capabilities.

A capability record should include:

• Supported input and output modalities.
• Context window limits.
• Structured output and tool-calling support.
• Streaming support.
• Prompt caching support and cache telemetry availability.
• Batch API support.
• Regional processing options.
• Data retention and abuse monitoring terms that matter to your policy.
• Rate limit dimensions and current capacity.
• Known output contract differences in your adapter layer.

This registry should be maintained separately from application prompts. It belongs near the routing and provider adapter layer, where the system can compare models without leaking provider-specific assumptions into every application.

Estimate Route Cost Before Selection

The router cannot know the final cost before execution, but it can estimate well enough to choose among eligible routes. The estimate should include the variables that actually move the invoice.

estimated_cost =
  input_tokens * input_token_price
  + cached_input_tokens * cached_input_token_price
  + output_tokens_expected * output_token_price
  + tool_call_overhead_expected
  + retry_allowance_expected
  - batch_discount_if_eligible

Use recent telemetry to estimate output tokens by task class. For example, extraction tasks may have bounded output length, while long-context analysis may vary widely. Tool-using agents may have additional hidden cost from tool call planning, retries, and intermediate messages.

Cost estimation should also account for cache and batch eligibility:

• Cache eligibility: stable prompt prefix, reusable system instructions, repeated retrieved context, supported provider behavior, and sufficient cache hit probability.
• Batch eligibility: workload can wait, has idempotent request handling, has durable job tracking, and does not need interactive streaming.
• Output-token risk: model tends to produce longer or shorter answers for the same task, and truncation would harm quality.
• Retry risk: provider or model has elevated error rates, schema repair rates, or latency failures for this task.

This is where cost routing connects to broader LLM cost management. The price table alone is not enough. The router needs request metadata, prompt structure, cache hit measurements, and settled usage data to know whether expected savings actually happened.

Use Model Tiers Instead of One Global Default

A tiered policy is easier to explain than a large list of provider-specific exceptions. Each tier should contain one or more eligible models that have passed task-specific gates.

Do not let the tier replace evals. A model can be economy for classification but unsuitable for support drafting. Another model can be strong for summarization but too expensive for short deterministic tagging.

Route With an Explicit Decision Sequence

A cost-aware router should be deterministic enough to debug. The exact implementation varies, but the decision sequence should be stable.

1. Read tenant, API key, environment, and application metadata.
2. Identify task class from explicit metadata or classifier output.
3. Load the routing policy version for that tenant or workspace.
4. Apply governance constraints such as data retention, region, model allowlist, and feature restrictions.
5. Filter by required capabilities: context length, tools, structured output, streaming, modality, and batch support.
6. Filter by eval-backed quality floor for the task class.
7. Estimate cost using input tokens, expected output tokens, cache eligibility, batch eligibility, and retry risk.
8. Apply budget policy and tenant preferences.
9. Select the lowest-cost eligible route.
10. Attach fallback route, timeout, retry, and repair policy.
11. Log the decision with a stable reason code.

In pseudocode, the router should look more like policy evaluation than a hard-coded provider preference:

function selectRoute(request):
  context = buildRoutingContext(request)
  task = classifyTask(context)
  policy = loadPolicy(context.tenant_id, task)

  candidates = modelRegistry.models
    .filter(model => policy.allows(model))
    .filter(model => model.supports(context.required_capabilities))
    .filter(model => evalStore.passes(task, model, policy.quality_floor))

  priced = candidates.map(model => ({
    model,
    estimated_cost: estimateCost(model, context),
    expected_latency: estimateLatency(model, context),
    risk_flags: estimateRisk(model, context)
  }))

  eligible = priced.filter(route => route.risk_flags.noneBlocking())
  return minBy(eligible, route => route.estimated_cost)

Log Every Routing Decision

If finance asks why a premium model was used, or engineering asks why quality dropped, the router should answer from logs. Do not rely on reconstructing intent from prompts after the fact.

Log these fields for every request:

• Request ID, tenant ID, API key ID, application ID, and environment.
• Task classifier result and confidence.
• Routing policy version.
• Eval dataset version and quality gate used.
• Eligible models and rejected models.
• Reason each rejected model was excluded.
• Estimated input, cached input, and output tokens.
• Estimated route cost and final settled cost when available.
• Selected provider, model, adapter version, and tier.
• Fallback route and retry policy.
• Cache hit or miss telemetry when available.
• Batch versus synchronous execution mode.
• Schema validation, repair attempts, refusal status, and safety flags.

These logs also support LLM API cost attribution, because model selection is only one part of spend. Teams still need to allocate cost by tenant, workspace, key, product, workflow, or customer.

Add Drift Controls Before You Promote Cheaper Routes

Model behavior, pricing, prompts, and workload mix change. A route that was safe last month may become weak after a prompt revision or model update.

A basic drift control loop should include:

• Sample recent production requests by task class.
• Remove or transform sensitive data according to policy.
• Replay samples through current and candidate routes.
• Compare quality, schema validity, refusal behavior, cost, and latency.
• Review failures by category, not only aggregate score.
• Promote the cheaper route only if it meets the quality floor and operating constraints.
• Canary the new route for a small percentage of traffic.
• Roll back automatically on schema failure, cost overrun, latency breach, or human-review escalation.

Recommendation: make route promotion a change-management event. Record who approved it, what eval set it passed, what traffic percentage it affects, when it expires or needs review, and how rollback works.

Give Teams Overrides, But Make Them Expire

There are legitimate reasons to override routing: provider incident, budget emergency, compliance review, customer escalation, or a model regression. The problem is not the override. The problem is an override that becomes invisible permanent policy.

Each override should include:

• Scope: tenant, application, API key, task class, model, provider, or region.
• Reason: incident, budget, quality, compliance, latency, or customer request.
• Approver.
• Start time and expiration time.
• Expected impact on cost, latency, and quality.
• Rollback condition.

Expired overrides should fail closed to the normal routing policy or require renewal. This keeps emergency decisions from becoming hidden cost or quality risks.

Trade-Offs to Make Explicit

Cost-aware routing has real operational cost. It is still worth doing when LLM spend is material, but the trade-offs should be visible.

• Metadata dependency: routing quality depends on accurate task labels and workload metadata. Missing labels push traffic toward conservative defaults.
• Eval maintenance: task evals need updates as prompts, products, and user behavior change.
• Provider complexity: cache and batch discounts can improve economics but make routing logic more complex.
• Portability risk: provider-specific features should live in adapters and capability records, not in product code.
• Estimate uncertainty: final output length, tool calls, retries, and cache hits are not fully known before execution.
• Explainability burden: aggressive optimization can make product behavior hard to explain unless every decision is logged with policy version and reason codes.

Implementation Checklist

• Define task classes for the top LLM workflows by spend and business impact.
• Create eval sets and quality floors for each task class.
• Build a provider and model capability registry.
• Store current price tables and pricing modifiers by provider, model, and execution mode.
• Estimate input tokens, expected output tokens, cache eligibility, batch eligibility, and retry risk.
• Filter candidates by governance constraints before cost optimization.
• Select the lowest-cost model only after capability and quality gates pass.
• Log selected and rejected routes with reason codes.
• Compare estimated cost with settled usage and invoice data.
• Run periodic replay evals before promoting cheaper routes.
• Canary route changes and define rollback triggers.
• Require expiration dates for manual overrides.

Conclusion

The practical goal is not to route every request to a cheaper model. The goal is to stop overpaying for routine work while protecting quality, contracts, governance rules, and explainability.

Build the router around task classes, quality floors, provider capabilities, cost estimates, and policy logs. Then choose the lowest-cost eligible route. That one word, eligible, is what separates cost-aware routing from cheapest-model routing.

Related reading

• gateway-level spend limits