Kuidas luua mitme teenusepakkuja LLM API-de jaoks andmete säilitamise poliitika ruuterit

A multi-provider LLM router should not choose a model first and ask retention questions later. If a support workflow is tagged as no-retention, the gateway has to know whether the selected provider, endpoint, region, account setting, and feature combination can honor that promise before the request leaves your boundary.

The failure mode is easy to miss. A team starts with stateless chat completion. The route is acceptable. Later, someone adds file upload, stored conversation history, web grounding, batch processing, or a vector store. The model name may stay the same, but the data behavior changes. A simple latency or cost router can silently send regulated or customer-contract-bound prompts to a path that stores content longer than the workload allows.

The practical fix is to put LLM data retention controls in front of model routing. The router should classify the request, match it against a canonical retention policy, filter provider capabilities, block disallowed features, and emit auditable evidence for the decision.

Facts to Treat as Design Constraints

Provider data handling is not uniform, and it is rarely captured by provider name alone. The following facts matter when designing a retention-aware router:

• OpenAI states that API data is not used to train or improve models unless the customer opts in. Eligible customers can configure Modified Abuse Monitoring or Zero Data Retention at the organization or project level after approval.
• OpenAI's Zero Data Retention changes endpoint behavior, including treating the store parameter for Responses and Chat Completions as false, but some endpoints and capabilities can still store application state and are not Zero Data Retention eligible.
• Anthropic documents Zero Data Retention for the Claude API as not storing customer prompts or responses at rest after the API response is returned, but coverage depends on endpoint and feature eligibility.
• Anthropic states that some Claude features or covered models may require retention and may be unavailable under Zero Data Retention unless specifically authorized.
• Google Cloud documents that Gemini Enterprise Agent Platform data is not used to train or fine-tune AI/ML models without prior permission or instruction, but zero data retention requires specific actions and may not be possible for some advanced features.
• Google Cloud documents that Grounding with Google Search stores derived query and context logs for up to three days and cannot disable that storage for that feature. It recommends Web Grounding for Enterprise when zero data retention is required.
• Microsoft documents that Azure-hosted models sold by Azure do not make prompts and completions available to OpenAI or other model providers and do not use them to train foundation models without permission or instruction.
• Microsoft documents that Azure deployment type affects processing location. Global deployments may process prompts and responses in any geography where the model is deployed, while DataZone deployments constrain processing to the selected zone. Stored data remains in the customer-designated geography.
• Microsoft documents that some stateful Azure features store data, including files, vector stores, Responses message history, Assistants threads, batch processing inputs, and stored completions.
• Microsoft documents that approved customers can modify abuse monitoring, and that the Azure portal or CLI can show ContentLogging=false when abuse-monitoring data storage is off.

These are not edge details. They mean the router must evaluate endpoint, feature, account approval, and region together. A provider-wide label such as "ZDR enabled" is too coarse for production policy.

Reference Architecture

A retention-aware multi-model AI API gateway needs five components in the request path:

1. Request classifier: extracts data sensitivity, tenant contract, region requirements, feature usage, file inputs, tool use, and logging permissions.
2. Policy resolver: maps the request to a canonical retention policy, usually from tenant, project, environment, API key, and workload metadata.
3. Provider capability registry: records provider, endpoint, model, feature, region, and account-level retention capabilities with dated evidence.
4. Policy evaluator: filters candidate routes and returns either an approved route or a policy denial with a reason code.
5. Audit emitter: records the chosen route, matched policy, capability assumptions, region, storage flags, and content logging mode.

This architecture keeps retention and residency decisions separate from cost, latency, and fallback. Cost routing can still happen, but only inside the set of routes that satisfy the retention policy.

Define a Canonical Retention Policy

Start with a provider-neutral policy object. Do not copy one provider's terminology into your internal contract. Your internal policy should describe what the workload is allowed to do, not how a specific provider markets a control.

{
  "policy_id": "support_no_retention_eu_v1",
  "max_content_retention": "none_after_response",
  "training_allowed": false,
  "human_review_allowed": false,
  "allowed_regions": ["EU"],
  "stateful_storage_allowed": false,
  "file_storage_allowed": false,
  "grounding_allowed": false,
  "batch_processing_allowed": false,
  "full_content_logging_allowed": false,
  "redacted_content_logging_allowed": true,
  "audit_level": "route_decision_with_provider_evidence"
}

For less restricted workloads, use explicit relaxation rather than missing fields:

{
  "policy_id": "internal_analytics_standard_v1",
  "max_content_retention": "30_days",
  "training_allowed": false,
  "human_review_allowed": false,
  "allowed_regions": ["US", "EU"],
  "stateful_storage_allowed": true,
  "file_storage_allowed": true,
  "grounding_allowed": true,
  "batch_processing_allowed": true,
  "full_content_logging_allowed": false,
  "redacted_content_logging_allowed": true,
  "audit_level": "metadata_and_decision"
}

The important fields are not only retention duration. Stateful features, files, grounding, batch processing, and logging mode all affect whether a route is acceptable.

Build the Provider Capability Registry

The capability registry is the router's source of truth. It should be maintained as controlled configuration, not scattered through provider adapters.

A useful record includes:

• provider: provider or cloud platform.
• account_scope: organization, project, deployment, subscription, or tenant where a setting is approved.
• endpoint: chat, responses, messages, batch, files, assistants, embeddings, grounding, vector store, or other API surface.
• model_family and optional model_id: because model eligibility can differ.
• region_mode: global, regional, data zone, customer-designated geography, or provider-specific equivalent.
• content_retention: no storage after response, abuse-monitoring retention, application-state retention, fixed-duration derived logs, or unknown.
• training_use: disabled by default, opt-in, or unknown.
• human_review: enabled, modified, disabled, or unknown.
• stateful_storage: whether the endpoint or feature stores conversation state, files, vector data, completions, or batch inputs.
• feature_flags: grounding, files, tools, cache, stored outputs, batch, assistants, or equivalent capabilities.
• evidence: source document name, URL, retrieval date, and owner review date.

Example:

{
  "capability_id": "azure-openai-gpt4o-datazone-chat-zdrlike-v3",
  "provider": "azure_openai",
  "account_scope": "prod-subscription-123",
  "endpoint": "chat_completions",
  "model_family": "gpt-4o",
  "region_mode": "datazone_eu",
  "content_retention": "abuse_monitoring_disabled_when_contentlogging_false",
  "training_use": "disabled_without_permission",
  "human_review": "modified_abuse_monitoring_approved",
  "stateful_storage": false,
  "features": {
    "files": false,
    "vector_store": false,
    "stored_completions": false,
    "grounding": false,
    "batch": false
  },
  "evidence": {
    "source": "provider_data_privacy_and_abuse_monitoring_docs",
    "retrieved_at": "2026-10-08",
    "review_owner": "security-architecture"
  }
}

The recommendation is to treat unknown as deny for sensitive workloads. If a capability record lacks a retention field, region mode, or feature eligibility, it should not satisfy strict policies.

Classify the Request Before Routing

Classification should happen before model selection. A good classifier combines static metadata and request inspection:

• API key scope: tenant, team, environment, application, and workload type.
• Contract tags: no-retention, EU-only, HIPAA-like controls, financial data, customer confidential, or internal-only.
• Payload type: free text, document, image, file reference, embedding input, tool result, or retrieved context.
• Feature use: web grounding, file upload, vector store, batch job, stored conversation history, tool calls, or response storage.
• Logging mode: full content allowed, redacted content allowed, metadata only, or no application log content.
• Regional hints: user location, tenant data residency, project region, or deployment region.

Do not rely only on automatic sensitive-data detection. Use declared workload policy from the API key or project as the primary source, then use classifiers to tighten policy when the payload contains risk signals.

function resolvePolicy(request) {
  const base = policyForApiKey(request.apiKeyId);
  const tenant = tenantContractPolicy(request.tenantId);
  const features = detectRequestedFeatures(request);
  const sensitivity = classifyPayload(request);

  return strictestApplicablePolicy([base, tenant, features, sensitivity]);
}

In practice, this means a customer-support key can default to no-retention, while an internal experimentation key can allow broader routing. If the same support key attempts a file upload, the effective policy should include file_storage_allowed=false unless an approved file-capable route exists.

Evaluate Policy Before Cost or Latency Routing

The evaluator should work as a filter. It receives a policy, request features, and candidate provider routes. It returns only routes that satisfy the policy.

function eligibleRoutes(request, policy, capabilities) {
  return capabilities.filter((route) => {
    if (!regionAllowed(route, policy.allowed_regions)) return false;
    if (!retentionAllowed(route, policy.max_content_retention)) return false;
    if (!trainingAllowed(route, policy.training_allowed)) return false;
    if (!humanReviewAllowed(route, policy.human_review_allowed)) return false;
    if (request.usesFiles && !policy.file_storage_allowed) return false;
    if (request.usesFiles && route.features.files !== true) return false;
    if (request.usesGrounding && !policy.grounding_allowed) return false;
    if (request.requiresState && !policy.stateful_storage_allowed) return false;
    if (route.content_retention === "unknown") return false;
    return true;
  });
}

function routeRequest(request) {
  const policy = resolvePolicy(request);
  const candidates = capabilityRegistry.findByModelIntent(request.modelIntent);
  const eligible = eligibleRoutes(request, policy, candidates);

  if (eligible.length === 0) {
    return deny("RETENTION_POLICY_NO_ELIGIBLE_ROUTE", policy.policy_id);
  }

  return chooseByCostLatencyAndHealth(eligible);
}

This ordering matters. Fallback routing should not jump to an ineligible provider during an outage. Batch routing should not move a no-retention request into an async job store. Grounding should not turn on a feature that introduces derived logs when the policy prohibits it.

If your platform exposes a OpenAI-compatible multi-model API, keep the compatibility layer honest: the same request body may compile to different provider-native features. The policy evaluator must inspect the compiled request, not just the original SDK shape.

Block Risky Feature Combinations Explicitly

The safest implementation is to create explicit denial rules for known high-risk combinations. Examples:

• NO_RETENTION + FILE_UPLOAD: deny unless the specific file endpoint and account setting are approved for the policy.
• NO_RETENTION + VECTOR_STORE: deny unless the vector store is inside your own controlled storage and the provider never receives persistent file state.
• NO_RETENTION + STORED_RESPONSE: force response storage flags off or deny if the endpoint cannot disable storage.
• EU_ONLY + GLOBAL_DEPLOYMENT: deny if processing may occur outside the allowed region or data zone.
• NO_RETENTION + WEB_GROUNDING: deny if the grounding feature stores derived query or context logs beyond policy.
• METADATA_ONLY_LOGGING + DEBUG_TRACE: suppress prompt and completion content from application logs and traces.
• ZDR_REQUIRED + STATEFUL_ASSISTANT_THREAD: deny unless the feature is specifically eligible under the account's approved controls.

Return specific errors. A useful denial tells the developer what failed and which approved alternatives exist.

{
  "error": {
    "code": "RETENTION_POLICY_DENIED",
    "reason": "NO_RETENTION_POLICY_DISALLOWS_FILE_STORAGE",
    "policy_id": "support_no_retention_eu_v1",
    "requested_feature": "file_upload",
    "approved_alternatives": [
      "send extracted text without provider file storage",
      "use internal document retrieval with metadata-only provider call",
      "request policy approval for controlled file processing"
    ]
  }
}

Separate Inference Logging from Operational Telemetry

Retention policy should not force the gateway to become blind. You can usually keep operational telemetry without storing prompt or completion content.

For strict workloads, preserve:

• request ID and trace ID;
• tenant, project, API key, and workload identifiers;
• policy ID and policy version;
• provider, endpoint, model, deployment, and region used;
• feature flags requested and allowed;
• decision reason codes;
• token counts, latency, status code, and cost metadata;
• hashes or stable fingerprints for deduplication, if approved by policy.

Suppress or redact:

• raw prompts;
• raw completions;
• uploaded file content;
• tool outputs that contain customer data;
• retrieved context from sensitive systems;
• provider debug payloads that duplicate content.

This is where retention routing connects to provider-neutral LLM observability. Observability should prove what happened without expanding the set of systems that retain sensitive content.

Emit Audit Evidence for Every Decision

An audit record should explain why a route was allowed or denied. It does not need full content to be useful.

{
  "request_id": "req_01J...",
  "tenant_id": "tenant_42",
  "policy_id": "support_no_retention_eu_v1",
  "policy_version": 7,
  "decision": "allow",
  "provider": "anthropic",
  "endpoint": "messages",
  "model_family": "claude",
  "region_mode": "approved_region",
  "features": {
    "files": false,
    "grounding": false,
    "stateful_storage": false,
    "batch": false
  },
  "logging_mode": "metadata_only",
  "capability_record_id": "anthropic-claude-messages-zdr-v5",
  "capability_evidence_date": "2026-10-08",
  "reason_codes": [
    "POLICY_MATCHED_NO_RETENTION",
    "REGION_ALLOWED",
    "STATELESS_ENDPOINT",
    "CONTENT_LOGGING_SUPPRESSED"
  ]
}

For denied requests, keep the same structure with decision=deny and reason codes such as REGION_NOT_ALLOWED, GROUNDING_RETENTION_EXCEEDS_POLICY, or STATEFUL_STORAGE_NOT_ALLOWED.

Use Contract Tests for Retention Rules

Retention routing should have automated tests. The important tests are not just unit tests for individual providers; they are contract tests for policy behavior across provider combinations.

High-value test cases include:

• No-retention policy plus file upload should deny unless a specific approved file-capable route exists.
• EU-only policy plus global deployment should deny even when the model is otherwise available.
• Zero-retention policy plus stored conversation history should deny or force storage off when supported.
• Grounding feature with derived-log retention should deny under no-retention policy.
• Metadata-only logging policy should suppress prompt and completion content from traces, logs, and error reports.
• Provider capability record with unknown retention should deny for regulated workloads.
• Fallback after provider outage should remain inside the eligible route set.
• Batch processing should deny when the batch endpoint stores input files longer than policy allows.

Also test drift. When a capability record changes, the system should show which policies and routes are affected before the change reaches production.

Operational Ownership

The hard part is not writing the first evaluator. It is keeping the registry current and preventing undocumented exceptions.

Assign owners for:

• provider documentation review cadence;
• approval status for account-level controls such as modified abuse monitoring or zero data retention;
• regional deployment inventory;
• feature eligibility for files, vector stores, grounding, batch, and stored responses;
• policy exception workflow;
• audit record retention and access control;
• developer-facing error messages and approved alternatives.

Every capability record should have a review date. If it expires, strict policies should stop trusting it or require manual approval. Provider docs and feature behavior change too often for this to be static configuration.

Trade-Offs

A strict retention router reduces available model choices. Some preview models, stateful APIs, grounding tools, and file workflows will be unavailable for sensitive workloads. That is the point of the control, but product teams need clear alternatives.

Regional routing can increase latency or cost when the fastest deployment is global but the workload requires a specific geography or data zone. Make this visible in telemetry so teams understand why a request used a more expensive or slower route.

Blocking stateful features can push teams to build their own memory layer. That may be the right architecture, but it still needs storage policy, deletion workflows, encryption, access control, and audit logs. Moving state out of the provider does not remove governance work.

Prompt and completion redaction protects sensitive data, but it weakens debugging and quality review. Use reproducible request metadata, deterministic test fixtures, controlled sampling for allowed workloads, and documented break-glass access where policy permits it.

Recommendations

• Enforce retention and residency before model routing, not as a post-processing check.
• Model capabilities at provider, endpoint, feature, region, and account-approval level.
• Use deny-by-default behavior for regulated or customer-contract-bound workloads.
• Treat file upload, vector stores, web grounding, batch processing, and stored conversation history as separate policy dimensions.
• Keep operational telemetry even when prompt and completion content must be suppressed.
• Emit audit records that include the policy, route, capability evidence, region, storage flags, and reason codes.
• Write contract tests for high-risk feature combinations and fallback behavior.

Prediction: Retention Routing Will Become a Standard Gateway Layer

The likely direction is that retention routing becomes a standard layer in enterprise LLM platforms, similar to rate limits, budget controls, and observability. As providers add more stateful tools, grounding options, caches, files, agents, and managed memory features, the safe route will depend less on the model name and more on the full execution path.

That is a prediction, not a current universal requirement. The current requirement is simpler: if your team promises no retention, regional processing, no training use, or no human review, your gateway needs enough policy logic and evidence to prove that each request followed those constraints.

Actionable Conclusion

Build the retention router as a policy gate in front of provider selection. Define a canonical retention policy, classify each request before routing, maintain a dated provider capability registry, deny unsafe combinations by default, and log audit evidence without retaining sensitive content.

The design goal is not to find a provider that is always safe. The goal is to prove that a specific request, using a specific endpoint and feature set, under a specific account and region, satisfied the policy before it was sent.

Related reading

• provider-neutral LLM observability design