Retries in AI APIs: How to Recover from Failures Without Duplicating Requests or Costs

A retry is not a harmless repeat

Retrying a call means sending a request again after an earlier attempt did not produce a response the client could use. The difficulty is that “I did not receive a response” does not necessarily mean “the server did not process the request.” The connection could fail before the request is sent, during transmission, or after the service has received and carried out the request. In the last case, repeating it could trigger a second operation or a second charge, even though the client saw only a network error.

That is why a useful policy begins before you choose a pause between attempts. You need to know what operation is being repeated, what result is known, what result is uncertain, and how much additional effort is allowed. A new query with no external effects does not carry the same risk as an operation that starts a task, consumes resources, or changes state. Nor should you confuse restoring availability with guaranteeing that the same generation runs only once: that guarantee depends on the API’s documented capabilities and the design of the integration.

This guide describes decisions that need to be adapted to the specific provider. The Kimi and ShareAI documentation referenced here provides context about their respective APIs, but the available excerpts do not confirm specific error codes, waiting headers, or idempotency mechanisms. Do not transfer those details from one service to another, or assume an API offers them because they are common elsewhere.

Classify the failure before deciding what to do

Do not group every failed response under one automatic rule. At a minimum, distinguish errors that indicate a configuration or validation problem, responses associated with usage limits, timeouts, and connection failures. This classification is a starting point for investigation, not a universal table of error codes: each provider defines its own responses and their meanings. Consult the API’s current documentation and record which signals justify each action.

A validation error, for example, will usually require correcting the request rather than sending it again unchanged. A usage limit may mean you can continue later, but you need to follow the guidance documented by that service. A timeout or disconnection raises another question: did the service receive the request? If you cannot determine that, classify the outcome as uncertain rather than assuming it was not processed. This distinction matters especially when the operation can incur costs or produce repeated effects.

The decision to retry should also take into account whether the failure appears temporary, whether repeating the same request makes sense, and whether there is enough time and attempt budget left. The secondary source discussing whether to retry or switch models recommends limiting retries to transient failures, operations that can be repeated safely, and the available budget. This is a general design principle, not an official definition of errors for any provider.

Define an explicit, finite policy

A retry policy should state in writing which failures can trigger another attempt, how many attempts are allowed, how long the full cycle can last, and what the client does when it reaches the limit. Distinguish the initial request from retries: if you set a maximum number of total attempts, say whether that number includes the initial request. Being precise prevents different interpretations between code, configuration, and metrics.

The total time limit—sometimes called the time budget—should include the waiting periods and subsequent calls, not just the connection time for a single attempt. If the budget expires, stop scheduling retries even if the maximum number of attempts has not yet been reached. Conversely, reaching the maximum must stop the sequence even if time remains. This prevents infinite cycles and requests from holding resources long after the caller can still use the result.

There is no attempt count or duration that is appropriate for every API. Justify those parameters based on the latency your application can tolerate, the provider’s guidance, the operation’s potential cost, and measured behavior under failure. Kimi’s troubleshooting help mentions exponential backoff and controlling the number of requests; the available excerpt does not prescribe universal values or confirm that this guidance applies to every error.

When an API’s documentation specifies a waiting signal, check its meaning and scope and follow the instruction set by the provider itself. Do not invent a header name or interpret a signal from another service as equivalent. If you have not verified a documented signal, the client should not pretend it can determine the correct time to resume from that signal.

Progressive waits and random jitter

A progressive wait increases the time between consecutive attempts. It can reduce the pressure caused by repeating calls immediately when a service is overloaded or experiencing an outage. Kimi’s troubleshooting documentation mentions exponential backoff alongside controlling the number of requests. This reference supports the fact that these concepts are covered in Kimi’s documentation; it does not establish a configuration that is valid for other APIs or prove that a particular wait is suitable in every case.

Random jitter adds variation to pauses so that many clients that failed at the same time do not call again in sync. It is one possible design tool, not a universal requirement. Assess whether it fits with the signals published by the provider and the operation’s maximum duration. If there is a documented waiting instruction, do not replace it with a random pause that could contradict it.

Document the parameters you choose and the reason for each one: how waits increase, what their limit is, what variation is applied, and how the sequence stops when the budget expires. Measure the effect using operational data: attempts per request, success after a retry, total duration, final errors, and associated usage where it can be measured. Do not turn an indicative recommendation into a universal figure without testing it with the actual provider and workload.

Avoid duplicates: idempotency and uncertain outcomes

Idempotency is a property that allows an operation to be repeated without producing additional effects, within the conditions defined by the system. In an API, it is not enough to find the term on a page: find out whether the provider offers an idempotency key or an equivalent mechanism, which endpoints or operations it covers, how to send it, how long it is recognized, and what happens if it is reused with different parameters. Only the provider’s current documentation can confirm those guarantees.

The available sources do not verify idempotency support for Kimi or ShareAI. ShareAI’s page explicitly mentions that writes require care, but the excerpt does not confirm a specific key or guarantee. Therefore, do not claim that either provider deduplicates a request by default. If the API does not document a mechanism that applies to the operation, treat a disconnection after sending as an outcome that may be uncertain, and check whether the provider offers a reconciliation mechanism, if any.

Distinguish between checking an already identified operation and starting another one. If the API documents a queryable identifier, use it only as described for its scope and lifecycle. If there is no documented way to verify the outcome, the integration has to decide whether to stop, report uncertainty, or allow a retry with an explicit risk. Do not promise exactly-once execution unless you have verified a guarantee that covers the actual case.

Worked example: the response is lost after sending

Imagine an application sends a generation request. The client waits for a response, but the connection drops. It does not know whether the server received the request, started processing it, or finished. Rather than retrying automatically, it marks the attempt as uncertain and checks the documentation for that API: is there an applicable idempotency key? Is there a documented way to check the status? What policy does the provider specify for this kind of interruption? The answers determine the next action.

If there is a documented mechanism that covers this operation, the integration can retry or check the status exactly within the mechanism’s stated scope. If there is not, the client cannot infer from a transport error that the service did not process the request. The team must follow its previously approved risk policy: for example, stop and show a status pending review, or allow a retry while making clear that it could incur another charge. The right choice depends on the impact of duplicating the operation and the mechanisms available; there is no universal answer.

If the provider returns a definitive response, the cycle ends according to the documented meaning of that response. If it returns success, no more attempts are scheduled. If a temporary failure is confirmed and it is safe to retry, check the limit, budget, and wait again. If the budget expires or the maximum is reached, end the sequence and record the final or uncertain outcome. Do not reset the counter simply because a different process is now handling the request.

Test realistic failures before deployment

Tests should check both the decision to retry and the decision to stop. Simulate a failure that the configuration classifies as temporary and verify that the client respects the limit. Simulate persistent failures and confirm that retries do not continue indefinitely. Let the time budget expire while attempts are still available; then check that reaching the maximum also stops the cycle even when time remains.

Include a late response and a disconnection immediately after sending. These are different cases: the late response may arrive after the client considers its wait to have expired, while a disconnection after sending leaves the processing status uncertain. Verify that the application does not duplicate an operation by assuming both situations mean “the server did nothing.” If the provider offers an idempotency feature, tests should confirm its scope in an appropriate environment and against current documentation, rather than simulate a guarantee the service has not promised.

Add validation-error and usage-limit cases to check that a generic rule is not applied to every response. If the provider publishes waiting signals, test their documented interpretation. Finally, check that logs make it possible to reconstruct the order of attempts without unnecessarily storing prompts, responses, or sensitive data. Correlation identifiers, outcome type, timing, and attempt number are often more useful for troubleshooting than copying the entire request content.

Criteria for putting the policy into production

Before deployment, record which documentation you consulted, when you verified it, and which version or endpoint it applies to. Keep provider instructions separate from application decisions: a documented recommendation does not replace an assessment of duplicate risk, and a local policy should not be presented as a guarantee from the service.

Review outcomes after deployment. An increase in attempts per call, exhausted cycles, or uncertain outcomes may indicate degradation, an overly aggressive configuration, or a change in service behavior. Treat fallback—for example, an approved alternative route—as a decision distinct from retrying: the secondary source on LLM APIs recommends switching only to a pre-approved alternative. Verify compatibility, quality, privacy, and cost before activating a fallback; do not use it to conceal the fact that the original call may have been processed.

Finally, measure technical recovery and response quality separately. A call that succeeds after several attempts does not prove that the generated content is correct or suitable. A robust policy limits harm and clarifies outcomes; it does not replace model-output evaluation or the application’s safety decisions.

Keep exploring

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.