Како изградити провајдер-неутралну ЛЛМ опсервабилност помоћу ОпенТелеметри

LLM observability breaks down when every application team logs provider responses in its own format. During an incident, platform teams need to answer concrete questions: which team made the request, which API key scope was used, which model was requested, which provider actually served it, whether a retry or fallback occurred, how many tokens were consumed, which latency bucket changed, and what error class caused the failure. They also need those answers without copying raw prompts, completions, provider keys, or customer identifiers into generic log stores.

The practical answer is to instrument the multi-model API gateway as the observability control point. Application-level tracing still matters, but the gateway is where provider selection, retries, fallbacks, usage accounting, request ownership, and privacy enforcement can be normalized before telemetry reaches downstream tools. For teams operating an enterprise LLM API, this becomes the difference between scattered debugging data and a provider-neutral operating model.

Define the Observability Contract First

Start with a written attribute contract, not dashboard screenshots. The contract should define the minimum metadata every LLM request must emit, which fields are allowed to leave the gateway, which fields are restricted, and which fields are never exported.

A useful baseline contract includes ownership, routing, model, usage, latency, and outcome metadata. Ownership attributes identify the organization, team, environment, workload, application, and API key scope. Routing attributes identify the gateway route, policy version, requested provider, requested model, selected provider, selected model, retry count, fallback count, and whether streaming was used. Usage attributes capture input tokens, output tokens, cached input tokens, reasoning tokens when available, request count, and estimated cost class. Outcome attributes capture status, provider error code when safe, normalized error class, validation result, timeout state, and final response model.

Keep the contract provider-neutral. A provider-specific error such as a rate-limit response, overloaded model response, schema validation failure, safety refusal, timeout, or context-window error should map to a normalized error class that dashboards can group across providers. The original provider error code may be useful, but it should be treated as restricted diagnostic data, not a primary dashboard dimension.

Use a Gateway-Centered Span Model

A clear trace structure prevents LLM telemetry from becoming a flat pile of spans. Use one inbound application request span, one logical LLM operation span, and child spans for the work the gateway performs.

A typical hierarchy looks like this:

inbound.http_request
  llm.operation
    llm.provider_attempt
    llm.provider_attempt.retry
    llm.provider_attempt.fallback
    retrieval.query
    tool.call
    output.validation

The inbound request span represents the caller's request to the gateway. The logical LLM operation span represents the user-visible LLM task, even if the gateway makes multiple provider attempts. Provider attempt spans capture each actual call to an upstream model API. Retrieval, tool, and validation spans should be children of the logical operation when they are part of producing the final response.

This separation matters during incidents. If a request falls back from one model to another, the logical operation may succeed while the first provider attempt fails. If the final output fails schema validation, the provider attempt may be technically successful while the logical operation fails. Without separate spans, dashboards either hide provider failures behind successful end-user responses or overstate failures that were successfully recovered.

Map OpenTelemetry GenAI Attributes Behind an Internal Mapper

OpenTelemetry semantic conventions exist so telemetry producers use common names and attributes across codebases, libraries, and platforms. The OpenTelemetry project now points Generative AI conventions to a separate GenAI repository, and those conventions define model request and response attributes such as provider name, requested model, response model, input tokens, output tokens, reasoning tokens, and cached input tokens.

Use those GenAI attributes where they fit, but do not scatter raw attribute names throughout gateway code. Wrap them behind a small telemetry mapper. That mapper should accept an internal request record and produce standardized span attributes.

record LlmTelemetryContext {
  organization_id
  team_id
  environment
  api_key_scope
  route_name
  route_policy_version
  workload_class
  provider_requested
  model_requested
  provider_selected
  model_response
  input_tokens
  output_tokens
  cached_input_tokens
  reasoning_tokens
  retry_count
  fallback_count
  normalized_error_class
}

The mapper can emit OpenTelemetry GenAI attributes for the standard fields and private gateway attributes for business context. Keep private attributes under a controlled namespace, for example llm_gateway.organization_id, llm_gateway.team_id, llm_gateway.key_scope, llm_gateway.route_policy, and llm_gateway.workload_class.

This design gives you vendor-neutral telemetry while protecting the codebase from churn. GenAI conventions are still evolving, so the mapper is where names can change without editing every handler, retry module, streaming adapter, and validation component.

Separate Metadata From Sensitive Content

LLM traces are unusually sensitive because the most useful debugging data may be the same data that should not be widely retained: prompts, retrieved documents, tool parameters, completions, user identifiers, customer identifiers, and raw provider credentials. The default posture should be metadata-first observability.

Recommended default fields include prompt hash, prompt template ID, prompt version, schema ID, tool name, retrieval collection, route name, token counts, latency, normalized error class, and validation status. These fields answer many operational questions without exporting raw content.

Raw prompt and completion capture should be opt-in, time-limited, access-controlled, and redacted before export. Do not enable it globally as a convenience feature. When content capture is needed for a production incident, attach a debug flag to a narrow set of requests, route those traces to a restricted backend, and set a short retention window.

Hashing identifiers is not automatically anonymization. If the input space is small or predictable, hashed values can be reversed by guessing. For sensitive customer or user identifiers, consider deletion, surrogate IDs, bucketing, truncation, or keyed hashing with strict key controls. Use hashes primarily for stable correlation when the risk is understood, not as a blanket privacy answer.

Enforce Privacy in the Collector

OpenTelemetry guidance says sensitive data should be handled before export. The Collector can remove, hash, filter, redact, or transform telemetry attributes using processors such as attributes, filter, redaction, and transform. That makes the Collector a second line of defense after the gateway instrumentation itself.

A practical Collector pipeline should use an allowlist for low-risk attributes, delete raw prompt and response fields by default, mask values matching blocked patterns, and route higher-sensitivity traces separately when content capture is explicitly enabled. The OpenTelemetry Collector attributes processor supports actions such as delete and hash. The redaction processor is designed to delete attributes that are not on an allowlist and mask values matching blocked patterns.

For example, the gateway can avoid emitting raw content by default, while the Collector still deletes attributes such as llm.prompt.raw, llm.completion.raw, provider.api_key, authorization, and cookie if they appear because of a bug or library integration. Treat this as a defense-in-depth control, not as a reason to emit unsafe data casually.

Instrument Retries, Fallbacks, and Validation Separately

Retries and fallbacks are often invisible in aggregate API metrics. That invisibility causes two operational problems. First, latency can degrade because a successful response may hide multiple failed attempts. Second, cost can rise because a single user request may create several upstream model calls.

Each provider attempt span should include attempt number, retry reason, selected provider, selected model, timeout budget, latency, token usage when available, and outcome. If a fallback occurs, the fallback span should identify the original route policy, the fallback reason, and the model actually used for the final response.

Structured output validation should be its own span or event. Capture schema ID, schema version, validation result, repair attempt count, and final status. Do not store the full invalid output by default. A validation failure is operationally different from a provider transport error, and dashboards should not blur them together.

Build Dashboards Around Questions, Not Providers

Provider-neutral observability should answer operating questions directly. Useful dashboards include p95 and p99 latency by model, provider, team, environment, and workload class; token usage by team and route; failure rate by normalized error class; retry amplification by route policy; fallback frequency by requested model; context-window errors by prompt template; structured-output validation failures by schema; and sampled trace exemplars for failed, expensive, or slow requests.

Keep billing and total usage reporting out of sampled traces. Tail sampling is useful for debugging, but it can distort totals. Token totals, request counts, and cost ledgers should come from unsampled counters, request ledgers, or billing reconciliation systems. Traces should explain why an expensive or slow request happened; they should not be the only source of financial truth.

Use Tail Sampling for High-Value Traces

The OpenTelemetry Collector tail sampling processor can make sampling decisions after spans are completed, including policies based on attributes and numeric values. That is useful for LLM traffic because you often know whether a trace is valuable only after the response finishes.

Retain traces for provider errors, validation failures, very high latency, unusually expensive token usage, context-window failures, fallback attempts, and manually debug-tagged requests. Sample routine successful traffic at a lower rate. This keeps the traces most likely to explain incidents while controlling storage volume.

A reasonable policy model is simple: always keep errors, always keep requests above a latency or token threshold, always keep debug-tagged traces, and probabilistically sample ordinary successful requests. Revisit thresholds as traffic changes. A threshold that is useful during early adoption may be too low once production traffic grows.

Add Contract Tests for Telemetry

Telemetry should be tested like an API contract because downstream dashboards, alerts, cost controls, and incident workflows depend on it. Add tests that run representative gateway requests and assert that required spans and attributes are emitted.

The tests should verify that a successful request emits the logical LLM operation span, a provider attempt span, requested model, response model, provider name, token counts when available, team or key scope metadata, route policy version, and final status. Retry tests should assert attempt counts and retry reasons. Fallback tests should assert both the failed attempt and the final successful provider attempt. Validation tests should assert schema ID, validation status, and repair attempt count.

Security tests are just as important. Assert that raw provider keys, authorization headers, cookies, raw prompts, raw completions, and unapproved customer identifiers are absent from exported telemetry. These tests should fail closed when a developer adds a new span attribute without classifying it.

Implementation Checklist

• Attribute contract: Define required, optional, restricted, and prohibited attributes.
• Span hierarchy: Separate inbound requests, logical LLM operations, provider attempts, retries, fallbacks, retrieval calls, tool calls, and validation.
• Semantic mapping: Use OpenTelemetry GenAI attributes where they fit and private gateway attributes for ownership and routing context.
• Privacy policy: Default to hashes, template IDs, schema IDs, tool names, token counts, latency, and error classes instead of raw content.
• Collector processors: Delete prohibited fields, redact blocked patterns, allowlist low-risk attributes, and route sensitive traces separately.
• Sampling policy: Tail-sample high-value traces and keep aggregate metrics unsampled.
• Dashboards: Build views around latency, usage, failures, retries, fallbacks, validation, and cost drivers.
• Alerts: Alert on normalized error classes, p99 latency shifts, retry amplification, fallback spikes, validation failures, and token anomalies.
• Contract tests: Assert required telemetry is present and sensitive telemetry is absent.

Trade-offs to Decide Explicitly

Gateway-level instrumentation gives consistent metadata quickly, but it can miss application-specific context unless teams pass structured request metadata such as tenant, workflow, prompt template, and feature name. Make that metadata part of the gateway request contract instead of relying on free-form logs.

Full content capture improves debugging and evaluation, but it increases privacy, security, retention, and access-control burden. Many teams should start with hashes, template IDs, schema IDs, and redacted excerpts only.

Provider-neutral error classes make dashboards usable across multiple model APIs, but they compress provider-specific details. Keep raw provider error codes only when safe, and restrict access to them.

OpenTelemetry GenAI conventions reduce lock-in, but the conventions will continue to evolve. An internal mapper gives you the benefits of standardization without letting unstable attribute names leak through the codebase.

Conclusion

Provider-neutral LLM observability is not just tracing added to model calls. It is an operating contract at the gateway layer: consistent spans, normalized model and provider metadata, ownership context, token usage, retry and fallback visibility, privacy controls, sampling rules, and tests that prevent sensitive data from leaking.

Start with the gateway because it already sees the decisions that matter most: who made the request, which policy routed it, which provider served it, what it cost in tokens, how long it took, whether recovery logic ran, and what failed. Once that foundation is stable, application teams can add richer workflow context without each team inventing its own LLM observability system.

Related reading

• fallback routing telemetry