Како смањити трошкове ЛЛМ API уз групну обраду без прекида операција

Many teams overpay for LLM workloads because every request goes through the same synchronous path. That is the right pattern for chat, agent loops, user-facing actions, and decisions that need an immediate answer. It is usually the wrong pattern for evaluations, document enrichment, classification backfills, embedding refreshes, nightly quality checks, compliance scans, and reporting jobs that can wait.

The hard part is not deciding whether batch discounts are attractive. The hard part is moving work into asynchronous processing without losing operational control. Batch processing changes the shape of the system: requests become jobs, failures arrive later, results may be partial, and cost attribution depends on reconciliation rather than immediate response metadata.

This guide separates facts, recommendations, and predictions so teams can make a practical implementation decision without treating batch processing as a universal replacement for synchronous APIs.

What is factually true about LLM batch APIs

Several major model providers now offer batch APIs for high-volume asynchronous work. The specific formats and limits differ, but the commercial pattern is similar: submit many requests, wait for asynchronous processing, then retrieve results later.

• Fact: OpenAI documents a Batch API for large asynchronous batches with completions returned within a 24-hour window and a 50% discount. Its documented supported endpoints include Responses, Chat Completions, Embeddings, Completions, and Moderations, with JSONL input files uploaded for processing.
• Fact: Anthropic documents Message Batches for high-volume asynchronous Messages requests, with a 50% cost reduction and increased throughput. Its documentation says most batches finish in less than one hour and that the pattern is appropriate when immediate responses are not required.
• Fact: Anthropic also documents separate rate-limit behavior for Message Batches, including limits for batch endpoints and the number of batch requests that can be in the processing queue at the same time.
• Fact: Google documents the Gemini Batch API for large volumes of asynchronous requests at 50% of the standard cost, and its pricing documentation lists Batch API as a 50% cost reduction feature.
• Fact: Amazon Bedrock model invocation logging can collect request data, response data, and metadata for supported runtime calls, with delivery to CloudWatch Logs and/or S3. Logged fields can include requestId, operation, modelId, requestMetadata, input token count, and output token count, which are useful for reconciliation.

Those facts support a clear recommendation: use batch processing where the workload can tolerate delayed completion and where the business value of lower unit cost outweighs the operational complexity of asynchronous execution.

Do not start with the provider API

The common implementation mistake is binding application code directly to a provider's batch file format or status model. That works for a prototype, but it creates avoidable migration cost when a team adds a second provider, changes models, or needs consistent reporting across synchronous and asynchronous traffic.

A better starting point is an internal batch-job abstraction. It does not need to be elaborate, but it should normalize the operational fields your platform needs regardless of provider.

Minimum batch job fields

• job_id: Your internal identifier for the batch job.
• provider: The provider selected for execution.
• provider_batch_id: The external batch identifier once submitted.
• workflow: The business workflow, such as eval_run, embedding_refresh, doc_enrichment, or qa_scan.
• team_id or customer_id: The accountable owner for quotas, reporting, and cost allocation.
• model: The requested model or model policy.
• endpoint: The provider endpoint type, such as chat, responses, embeddings, moderation, or messages.
• submitted_at and deadline_at: The timestamps needed for SLA handling.
• status: A normalized state, such as queued, submitted, processing, completed, completed_with_errors, failed, cancelled, or expired.
• input_uri and result_uri: Pointers to durable input and output artifacts.
• item_count, success_count, failure_count: Job-level reconciliation totals.
• input_tokens and output_tokens: Realized usage after processing.
• estimated_cost and realized_cost: Submission-time estimate and post-reconciliation cost.
• cost_basis: The pricing rule used, including whether batch discounting was applied.

This abstraction gives the rest of the organization one contract. Provider-specific fields can still be stored for debugging, but they should not become the interface every application team has to learn.

Classify workloads by SLA before routing

Batch processing should be driven by service-level expectations, not by a blanket desire to reduce spend. A simple routing policy is easier to operate than a clever one.

Interactive class

Interactive requests are user-facing or time-sensitive. Examples include chat UX, agent loops that require tool calls, support workflows where an operator is waiting, fraud or payment decisions, and any workflow where delayed partial output is worse than no output. These should stay synchronous unless the product explicitly changes its user experience.

Delayed class

Delayed workflows can wait without hurting the user or business process. Examples include offline evaluations, document tagging, classification backfills, embedding refreshes, historical moderation, nightly QA checks, and report generation. These are the best candidates for provider batch APIs.

Borderline class

Some workloads are not truly interactive, but they still have a deadline. For example, a document imported by a customer may need enrichment within 20 minutes, not immediately. For these, use an internal queue with a deadline. If the job is still pending near the deadline, it can be promoted to synchronous processing, cancelled, or rerouted according to policy.

This is where AI API cost control becomes an infrastructure concern rather than a pricing exercise. The system needs a routing policy, deadline awareness, accounting, and observability in the same place.

Use stable request IDs for every item

Batch jobs fail in more varied ways than synchronous calls. A provider may reject the file. A job may complete with item-level errors. A retry may resubmit work that partially succeeded. An ingestion process may crash after writing some results but before marking the job complete.

The most important protection is a stable request key for each item. This key should be deterministic from your own data, not generated randomly at submission time.

For example, a document enrichment request key might be:

doc_enrichment:v3:customer_812:document_9912:section_04:model_policy_standard

An embedding refresh key might be:

embedding_refresh:v2:workspace_44:asset_2107:chunk_18:embedding_model_small

The key should include the workflow, version, tenant boundary, source record, item boundary, and model or model policy where relevant. This allows downstream writes to be idempotent. If the same result is received twice, the database can upsert by request_key rather than creating duplicate rows or double-counting usage.

Design the state machine explicitly

Provider states differ, so the internal state machine should be small and durable. A practical version looks like this:

• created: The job exists internally but has not been packaged.
• validated: Input records passed schema and eligibility checks.
• submitted: The provider accepted the batch submission.
• processing: The provider has started or queued execution.
• completed: All items reached a successful terminal state.
• completed_with_errors: The job finished, but some items failed and need review or replay.
• failed: The job failed at the provider or orchestration layer.
• cancelled: The system or operator stopped the job before completion.
• expired: The job missed its deadline and should not write normal results without review.

Each transition should be append-only in an event table. That gives operations teams an audit trail when a job misses its deadline or when a downstream system disputes whether a record was processed.

Build reconciliation as a first-class workflow

Submission is only half the system. The expensive operational bugs usually appear during result ingestion.

A reconciliation worker should do five things every time it processes a completed batch:

1. Load the original manifest of submitted request keys.
2. Load provider results from output files, result endpoints, or provider logs.
3. Map every provider result back to exactly one internal request key.
4. Write successful results idempotently to the destination system.
5. Emit terminal records for failures, missing items, token counts, and realized cost.

The reconciliation process should verify that every submitted item has a terminal state. Missing results should not disappear silently. Mark them as missing, failed, or retryable according to the provider response and your policy.

Cost attribution should also happen here. Submission-time estimates are useful for budget checks, but realized cost should be derived from actual token counts, provider output, billing exports, or invocation logs where available. This avoids a recurring problem: teams report estimated savings while the finance ledger later shows a different number.

Handle partial failures without replaying everything

Batch systems must assume partial failure. A prompt schema may fail for 2% of records. A subset of requests may hit length limits. Some inputs may violate a moderation or content policy. A result file may contain valid output for most items and structured errors for the rest.

The correct response is usually not to replay the entire batch. Replay only the failed or missing request keys into a new job, with a parent_job_id and retry_attempt. This limits cost, reduces duplicate writes, and keeps the original job's audit trail intact.

A useful retry record includes:

• request_key
• parent_job_id
• retry_attempt
• original_error_code
• retry_reason
• new_job_id
• final_status

Do not retry all errors automatically. Transient provider failures, queue timeouts, and recoverable transport issues may be retryable. Prompt validation failures, schema mismatches, and unsupported input size errors usually require a fix before replay.

Control blast radius with batch size and versioning

Large batches can improve throughput and reduce orchestration overhead, but they also increase the cost of a bad prompt, wrong schema, or incorrect model selection. Smaller batches give faster feedback and lower blast radius, but they create more jobs to track.

A practical approach is progressive rollout:

1. Run a small validation batch with representative inputs.
2. Confirm schema validity, output quality, token range, and reconciliation behavior.
3. Increase batch size by workflow after the first successful run.
4. Keep prompt version, schema version, and model policy in the request key or manifest.
5. Stop automatically if failure rate, token usage, or output validation crosses a threshold.

This is especially important for backfills. A backfill can create a large bill quickly even with a discount if it processes millions of records with an unexpectedly verbose prompt or output format.

Expose savings as estimated and realized

Batch discounts are attractive, but reporting should not blur estimates with actuals.

At submission time, the platform can show an estimated cost using expected token counts, selected model, item count, and known batch discount rules. After reconciliation, the platform should show realized cost based on actual token usage and provider billing or usage data.

A useful dashboard includes:

• queue age by workflow and provider
• oldest pending job
• completion percentage
• failed item rate
• retry count
• estimated cost submitted
• realized cost reconciled
• effective discount captured
• synchronous fallback volume
• jobs approaching deadline

This distinction also helps finance and engineering speak the same language. Engineering can optimize routing and retries; finance can reconcile realized spend; product teams can see which workflows are actually benefiting from delayed processing.

Recommendations for implementation

Use the following checklist before moving a workload to batch processing.

Eligibility checklist

• The workflow does not require an immediate user-facing response.
• The business can define an acceptable completion window.
• Each item has a stable request key.
• The destination write can be idempotent.
• Partial results are acceptable and reconcilable.
• Failures can be retried selectively.
• Prompt and schema versions are recorded.
• Usage can be attributed to a team, customer, product, or environment.
• Sensitive payload logging has been reviewed for privacy and retention obligations.

Operational checklist

• Normalize provider status into an internal state machine.
• Store original input manifests durably.
• Poll or receive provider completion events on a schedule.
• Validate every result against expected schema.
• Compare submitted request keys with received terminal states.
• Emit usage records only after reconciliation.
• Separate estimated cost from realized cost.
• Alert on queue age, failed item rate, missing results, and deadline risk.
• Document which workloads must remain synchronous.

Trade-offs to make explicit

Recommendation: Treat batch processing as a separate execution class, not a hidden optimization behind the synchronous API path. Teams should know when a workflow is delayed, what deadline applies, and what happens when the deadline is missed.

Trade-off: Batch APIs can reduce token costs materially, but they replace simple request-response behavior with queueing, polling, result ingestion, delayed errors, and more complex support workflows.

Trade-off: Provider batch APIs are not identical. A normalized internal model reduces application complexity, but the platform still needs to preserve provider-specific identifiers and errors for debugging.

Trade-off: Logging full prompts and outputs can improve auditability and reconciliation, but it can also create privacy, security, and data retention obligations. Sensitive content should be classified, redacted, or excluded where required.

Prediction: More LLM infrastructure teams will treat batch execution as a standard routing class alongside synchronous calls, streaming calls, and fallback routes. The teams that benefit most will be the ones that build reconciliation and cost accounting early, rather than adding them after finance or operations asks why the numbers do not match.

Actionable conclusion

Batch processing is one of the clearest ways to reduce LLM unit costs for delayed workloads, but it should be implemented as an operational system, not just a cheaper endpoint. Start with workload eligibility, stable request keys, a normalized job model, and a reconciliation workflow that proves every submitted item reached a terminal state.

Keep interactive requests synchronous. Move delayed workflows to provider batch APIs where the cost and completion window make sense. Use an internal queue for borderline workloads with explicit deadlines. Report estimated savings at submission time, but reconcile realized cost from actual usage. That is the pattern that captures batch discounts without breaking downstream records, team ledgers, or SLA expectations.

Related reading

• per-team cost attribution and reconciliation