How to instrument an AI application to investigate failures, latency, and usage

1. Start with operational questions, not fields

Observability for an AI application means being able to reconstruct its behavior from related signals: which request arrived, which stages ran, what failed, and how long each part took. For an application that already calls a model through an API, instrumentation does not mean copying every input and response into a log. It means collecting enough context to answer operational questions consistently, within a clearly defined scope and data policy.

Before choosing a library or building a dashboard, write down the questions the team needs to answer during an incident. For example: Which application and prompt versions handled this request? Was the model called once, or were there retries? Which tool ran, and how long did it take? Was the delay in the application, network, model, or a tool? What usage did the API report? These questions shape the design; fields that do not help answer them should not be added out of habit.

It is useful to distinguish observability from content evaluation. This article focuses on a request’s technical and operational path: failures, latency, retries, and reported usage. It does not propose a method for determining whether an answer is true, choosing a provider, or designing a conversational flow. Keeping that boundary clear helps keep logs small and prevents a technical signal from being presented as proof of quality.

2. Map the request lifecycle and define correlation

A request that looks like a single call from the interface may involve several operations: input validation, prompt construction, a model call, response processing, and possibly a tool call. If retries occur, each attempt should remain visible as a distinct operation. Logs should let you link these events without relying on identifiers that the external API does not guarantee.

Generate your own identifier for the request when it enters your application, then propagate it through the components you control. Add child identifiers or stage numbers to distinguish model and tool calls. If the provider returns a request identifier, store it as an additional field for later queries or support, but do not use it as the only link: it may not be available for every error or execution path.

Also define the timing boundary for each measurement. Total duration starts when the application accepts the request and ends when it delivers a response or records the final failure. A model call’s duration starts when that call begins and ends when a response or error is received. A tool call has its own start and end times. Recording milliseconds without defining these boundaries produces figures that cannot be compared reliably.

3. Define a minimum schema and record what the API actually returns

A minimum schema should separate data generated by your application from fields received from the provider. Application-owned fields commonly include the request identifier, stage, attempt number, status, timestamps, and code and prompt versions. The requested model, configuration, and any identifier returned by the provider can help reconstruct an execution, but their origins should be labelled.

Usage reported by an API is not universal. It can vary by endpoint, model, modality, version, or response type; it may also be absent from errors or partial responses. The documentation available for this guide does not specify a particular provider, so it cannot establish a universal list of fields. Instead, implement an adapter for each API and consult its current primary documentation to check field names, units, and conditions. Store the result as “API-reported usage,” including which fields were present, and do not silently fill missing values with zeros.

Also keep application-measured time separate from any duration reported by the provider. The former measures the path through your system; the latter, if available, follows the provider’s definition. Make units explicit—for example, milliseconds for durations—and define whether timestamps are stored in UTC. Consistent conventions make it possible to compare requests and detect changes without mixing clocks or different intervals.

4. Instrument structured events end to end

Structured logs are easier to filter and group than prose messages. Define stable names for events, statuses, and fields; avoid having every component use a different label for the same concept. You do not need a specific observability vendor to get started: you can emit JSON events from the application and send them to the platform the team already uses.

One useful pattern is to emit a start and end event for each stage, or a single end event containing start time, end time, and outcome when the stage is short and does not need real-time updates. For longer operations, the end event should retain its link to the original request. If you use distributed traces, propagate context across services and queues you control; verify that the identifier survives the transition from synchronous to asynchronous work.

The example below is schematic pseudocode, not a working integration with an API: response and error field names depend on the SDK and provider. In a real implementation, adapt usage and error extraction to current documentation, and make sure the exception handler does not automatically log bodies that may contain sensitive data.

5. Minimize sensitive data before storing or exporting it

Observability does not require keeping full prompts, complete responses, or tool arguments for every request. This content may contain personal data, secrets, commercial information, or material that is unnecessary for operational questions. Start with metadata and derived signals: opaque identifiers, versions, lengths, error categories, statuses, and timings.

Decide for each field whether to exclude, redact, transform, or store it with restricted access and limited retention. To correlate with a business entity, a pseudonymous key or internal reference that only an authorized system can resolve may be sufficient. A hash does not guarantee anonymity if the original value is predictable or can be checked against a list; do not automatically treat it as a privacy solution.

Test redaction before enabling logging in production. Use synthetic examples covering typical cases and variations, such as an email address in an input, a token in a header, or confidential data in a tool response. Inspect both the event leaving the application and the event reaching the store: a rule applied only in the dashboard or after export leaves the data exposed in transit.

6. Build dashboards and alerts for investigation, not for accumulating charts

A useful dashboard answers a question and makes clear what action might follow. Start with three views: request health, latency distribution, and reported usage. For health, show completed, failed, and unresolved requests by error category. For latency, show total duration and duration by stage, with percentiles as well as averages, because an average can conceal a long tail of slow requests. For usage, show only available fields and break them down by model, version, or request type when that helps reveal changes.

Alerts should represent a condition someone can investigate or fix. An error rate that exceeds the usual pattern over a defined interval may warrant an alert; an isolated failure can remain in the log without waking a team. Thresholds depend on volume and service expectations, so there is no need to set a universal value without data. Include the observation window, owner, and a link to the relevant query or response procedure.

Do not present a cost estimate based on reported usage as a billed amount. For an internal estimate, multiply the available units by the prices and rules that apply to the specific account and configuration, and label the result as an estimate. Actual amounts may depend on conditions that cannot be inferred from the response event alone; verify any reconciliation against the provider’s current official billing information and documentation.

7. Investigate a failed request reproducibly

When a report arrives, first pin down the request ID and confirm that it matches the right environment and time period. Gather the events in order by time and stage. Before interpreting the result, verify that the trace is complete: is there a start and end event? Are tool spans missing? Does the log record a terminal error or just a failed attempt? An incomplete execution can look like a model failure when the evidence simply was not captured.

Next, reconstruct the versions and sequence: application code, prompt version, requested model and any observable resolved model, number of attempts, tools, and statuses. Compare total duration with the sum of the stages, accounting for local work, queues, and time that was not instrumented. If the sum does not match, record that as a telemetry gap, not as proof of a cause.

Finally, separate facts from hypotheses. A provider timeout may be recorded as the outcome of an attempt; on its own, it does not explain why the timeout happened. A slow tool may account for part of the measured time, but not necessarily the final behavior. Record what is known, what is missing, and what check could distinguish between hypotheses. This discipline helps prevent claims about causes that the log does not establish.

8. Verify the instrumentation and expand in stages

Instrumentation also needs testing: an application may work while dropping IDs, omitting end events, or logging data that should have been redacted. Run controlled tests for success, error, timeout, retry, and tool calls. For each case, verify the complete sequence from the incoming event to the final outcome, along with permissions and redaction before export.

Start with a limited production path and check that queries answer real questions. Review how much volume the logs generate and whether the team can find a request without manually searching several systems. Adjust event selection and retention before expanding. The goal is not to maximize telemetry, but to maintain enough searchable evidence to operate the system.

Telemetry conventions and API usage fields can change. Version your internal schema and validate the adapters when the SDK, API, or configuration changes. If the provider changes a response, the normalization layer should be adaptable without making every dashboard and query depend on external field names. General observability practices can guide the organization of signals and traces; they do not replace primary technical documentation for a specific provider’s fields or billing.

Keep exploring

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.