One-sentence definition
Tiempo transcurrido entre una solicitud y un hito de respuesta, como el primer token o la finalización completa.
Definition: An Interval with Two Milestones
Latency is the time elapsed between a starting event and an ending event that have been defined in advance. To interpret a measurement, you need to know what those two milestones are: for example, when a request is considered to start and when a response is considered to arrive. Without that precision, two figures both labeled “latency” may describe different intervals and may not be comparable.
In an AI application, the start could be the point at which a particular component receives a request. The end could be when a decision becomes available, when the first part of a response is delivered, or when the complete response is available. You should not assume that one of these is always the correct definition. It depends on the question being asked and on how the system has been instrumented.
As a general definition, the European Union’s Interoperable Europe Portal describes latency as the delay between initiating a request and receiving a response. This helps establish the idea of an interval, but it does not by itself define a specific metric for AI inference, streaming generation, or the response experienced by a user.
In this article, “latency” is used as a general name for the measured interval. When the boundaries matter, they are stated separately—for example, “from receipt to the first token” or “from the end of the voice input to the start of the audible response.” These formulations are proposed ways to define measurements; they do not claim that every system follows a universal convention.
Which Interval Can Be Measured?
A system may offer several useful milestones within the same workflow. For a request that produces text, you can measure the interval until the first token appears and, separately, the interval until the response is complete. The first describes when some of the output begins to be available; the second describes when the complete result is available according to the chosen criterion. These figures are not interchangeable.
The start can also be defined at different points. Measuring from when an interface receives a user’s action is not the same as measuring from when a service receives the request. The difference between those points may include stages that an internal measurement of the model does not. A rigorous description therefore names the entry point rather than simply saying “response time.”
Time to first token, often abbreviated as TTFT, can be useful when an application displays text progressively. Time to complete response may matter more when a task requires the entire output before anyone can act on it. These are operational distinctions for designing a measurement. The sources available for this article do not establish a universal definition of TTFT or a shared procedure used by all providers.
Possible milestones and the question each interval answers
| Proposed start | Proposed end | What it helps you observe |
|---|---|---|
| The application receives a request | The system delivers the first part of the output | When some of the response first becomes available |
| The application receives a request | The response is complete and available | How long it takes to obtain the complete result |
| A component receives an input | That component emits its result | The interval observed within the instrumented boundaries |
Inference Latency and End-to-End Latency
Inference latency usually refers to the time attributed to running a model to produce an output from an input. However, the term does not by itself determine where the clock starts or which operations are included. One study may define its boundaries one way and a platform another way. If the documentation does not explain the scope, the figure is incomplete for comparison.
End-to-end latency, by contrast, is measured from a starting point in the application workflow to an observed result at another point. Depending on the selected boundaries, it may include stages before or after model execution, such as request transit, input processing, or output delivery. This list describes components worth checking; it does not guarantee that every system includes all of them.
The latency perceived by someone using a product should not automatically be equated with inference latency either. To approximate a person’s experience, you would need to define which visible or audible event counts as a response and which action starts the measurement. If a system displays content in parts, the first indication of a response and its completion may affect that experience differently.
Research on latency optimization in edge computing can provide context for recognizing that the execution of multiple models in that kind of environment is a subject of study. But the available excerpt is not enough to derive a general definition of inference latency or to establish that the study’s boundaries match those of a complete application.
A process for documenting a measurement
- 01Write down the exact event that starts the interval and the component where it occurs.
- 02Write down the exact event that ends it: the first part of the output, an available decision, the complete output, or another observable milestone.
- 03Note which stages fall inside and outside those boundaries. If this is unknown, state it as an uncertainty.
- 04Record the execution conditions accompanying the figure and the kind of summary reported, such as a single observation or a distribution.
- 05Compare only measurements whose definitions and conditions are sufficiently similar for the question you are trying to answer.
Three Practical Examples: Define Before You Measure
The following cases show how to formulate a measurement; they are not empirical results or expected values. In each case, the proposed milestones make it possible to ask a specific question. A team could choose different boundaries if its objective were different, provided it states them.
In a voice conversation, if the goal is to find out when the system starts responding, you could start the interval when the voice input ends and stop it when the audible response begins. If you need to know how long it takes for the system to finish speaking, the ending event should instead be the end of the voice output. These measurements describe different milestones. To interpret them, you would also need to clarify which component detects the end of the input and which signal counts as the start or end of the audible response.
For transaction classification, the start could be when the event is received by the system that must evaluate it, and the end could be when the decision becomes available to the component that needs it. The definition does not require every preceding or subsequent stage to be included. That depends on whether the goal is to measure the classifier, a particular service, or the application’s complete workflow. A figure without those boundaries does not, by itself, reveal which of these questions it answers.
For streaming text generation, you can define two intervals with the same start—for example, when the service receives the request. One interval ends when the first token is delivered; the other ends when the response is complete. If you report only one, say which one it is. The available excerpt from the source on AI latency discusses processing a unit of data or completing a task, but does not verify a specific first-token metric. This example should therefore be understood as a proposed instrumentation approach, not as a claim attributed to that source.
Latency, Throughput, and Other Common Confusions
Latency and throughput do not answer the same question. Latency is expressed as the duration of an interval associated with a defined request, task, or event. Throughput is used to describe an amount processed over a period, but the evidence supplied for this article does not include a source specifying its units or a technical definition for AI systems. The distinction here is therefore conceptual guidance, not a quotation from a universal standard.
A system may process many requests over time, while a particular request still takes longer than a user considers acceptable. The reverse may also happen: an individual response may arrive quickly in a test, without showing that the system will maintain the same behavior under a heavy workload. These are possible scenarios that illustrate why a duration measurement and a measure of accumulated processing should not be treated as synonyms.
The first token is not the same as the complete response either. In incremental output, the first milestone tells you when content begins to become available; it does not tell you how long completion takes or whether the final content meets the task’s needs. Likewise, a measurement of inference does not automatically show how long someone using the application has to wait.
Finally, latency is not a measure of response quality. A fast output can be incorrect or incomplete, and a slower output is not validated merely because it took longer. Assessing quality requires separate evaluation criteria; the sources available here do not provide a methodology for evaluating it.
Questions that should not be confused
| Term or milestone | Question it asks | What it does not prove on its own |
|---|---|---|
| Latency | How much time passes between these two events? | Output quality or sustained capacity |
| First token | When does the first part of the output appear? | When the response is complete |
| Complete response | When is the entire result available, according to the stated criterion? | When content first began to appear |
| Throughput | How much work is processed over a period, according to the stated unit? | The duration of an individual request |
How to Interpret a Figure—and Its Limits
A single figure does not characterize behavior across every input and output size, network condition, workload, or configuration. Nor does it establish that the same latency will be observed in other environments. To assess its scope, first review how the measurement is defined and the conditions under which it was obtained.
Check whether the documentation identifies the starting point, the ending event, and the stages included. Then look for information about input and output sizes, concurrent workload, execution environment, and any other conditions the source declares. If those details are absent from the material you consulted, do not fill them in by assumption; state that they are unspecified.
It also matters how the observation is summarized. A single record and a distribution of multiple observations do not provide the same kind of information. This article recommends checking which is being reported before comparing figures, but the supplied sources do not prescribe a particular statistical summary or establish acceptance thresholds.
The cited literature on latency optimization addresses a technical problem, but the excerpt provided does not detail the evaluation conditions. Similarly, the research article about running multiple models in edge computing is relevant as research context, although the excerpt is not enough to reconstruct its operational definition. Accordingly, no values, rankings, or general rules are derived from those excerpts.
Related Concepts and Common Mistakes
Inference is the process of obtaining an output from a model based on an input. Inference latency is the time attributed to that process according to the boundaries defined by whoever is measuring it. They are not synonyms: naming the process does not automatically specify the chronological interval or its components.
A token is a unit that may appear in the representation or generation of text. In this article, “time to first token” names a proposed milestone for measuring progressive output. It does not mean that every response is delivered token by token or that this milestone is sufficient to assess the complete experience.
A context cache is a related concept involving the reuse of contextual information in certain processing workflows. It should not be confused with a latency unit. To attribute an effect to it in a particular measurement, you would need documentation and comparable conditions. The supplied sources are not sufficient to quantify that effect or to claim it applies to every service.
AI evaluation examines properties of an output or system using defined criteria. Latency measurement may be part of an evaluation, but it does not replace assessment of accuracy, usefulness, or safety. No sources have been provided here that are sufficient to establish how to evaluate those properties.
Practical Criteria for Describing and Comparing Latency
To describe a measurement usefully, record the starting and ending events in observable terms, identify which component records each one, and state which stages are included and excluded. If the milestone is the first token, say so. If it is the complete response, define how you know the response is complete. Avoid the generic label “response” when it could mean several different things.
Add the execution conditions provided by the source, such as concurrent workload or input and output size, and clarify when those details are unavailable. State whether the figure represents one observation or a summary of several. Do not extrapolate beyond what the documented scope allows.
Before comparing two figures, check that they answer the same question, with sufficiently similar milestones and conditions. If one measures time to first token and another measures time to complete a response, present them as different metrics. If one concerns a component and the other the complete workflow, do not attribute the difference to a model without additional evidence.
In short, latency is not a property that can be interpreted correctly without context: it is the time between two defined milestones. The safest practice is to name those milestones, explain what falls within the interval, and treat any detail that cannot be verified in the consulted documentation as unresolved.
Checklist before publishing a figure
- 01Are the starting point and ending milestone identified?
- 02Is the first result distinguished from the complete response?
- 03Is it clear whether the measurement covers a component, inference, or the end-to-end workflow?
- 04Are the execution conditions and the type of reported data described?
- 05Does the conclusion stay within what the measurement can support?
