DuplexJev: Batched Voice Decisions Without Token-by-Token Decoding

The Latency Problem in Voice Agents

Full-duplex voice agents must make numerous small, closed decisions while the user is speaking: determining if a turn is complete, identifying a barge-in, or selecting an appropriate filler. Currently, these systems typically respond by generating text autoregressively, a slow process that blocks the agent's response until the decoding is complete.

In an in-car voice assistant analyzed in the study, these decisions consume 30% of all LLM input tokens. This load not only increases the latency perceived by the user but also requires additional computing capacity to maintain fluency during peak hours, raising operational costs.

The decisions described here are narrow, bounded questions rather than requests for an open-ended spoken answer. Even so, handling them through autoregressive text generation means the system must produce a sequence before it can act on the result. The delay therefore accumulates not only in longer conversational responses, but also in brief judgments that need to happen while speech is still in progress. The study’s motivation is to make these intermediate decisions cheaper to obtain, without claiming that every part of a voice interaction is made faster.

The 30% figure is specific to the in-car assistant examined in the study; it should not be read as a general measurement for all voice agents. It illustrates how frequent, small decisions can account for a substantial share of LLM input in that setting. If an agent must make those decisions repeatedly, their token and compute demands can compound across turns, linking response timing to the resources needed to preserve conversational fluency. The reported finding concerns this measured workload, rather than a demonstrated reduction in operational costs.

Architecture: Connection Between Encoder and Frozen LLM

DuplexJev proposes a modular architecture where an ASR (Automatic Speech Recognition) encoder and a general-purpose LLM remain frozen. Only a small connector is trained to transform the encoder's hidden states into embeddings compatible with the LLM's embedding space.

Instead of decoding a complete textual response, the system formulates each question as a probability distribution over a closed set of options (represented by letters). The LLM performs a single inference pass (forward pass), and the answer is extracted from the logits at the answer token position, without sampling or decoding steps.

Keeping the encoder and LLM frozen separates the learned audio-to-LLM mapping from the parameters of those two existing components. In the proposed arrangement, the trainable connector is the part that adapts the encoder’s audio representations for use by the LLM. This describes which elements are updated during training; it does not, by itself, establish effects such as faster convergence, reduced overfitting, or improved performance across different backends.

The closed-option format changes what the model has to produce for each decision. Rather than generating a sentence and then interpreting it, the system assigns probabilities to the available answer choices and reads the answer-token logits after one forward pass. In the example, the two options are “Yes” and “No,” so the distribution directly represents the decision. This is a method for extracting a bounded answer without sampling or text decoding, not evidence that the system handles unrestricted responses in the same way.

Because the answer is read from the logits at a designated token position, the procedure avoids waiting for a complete textual response to be generated. The distinction matters for the small, closed decisions described in the study: the output is an option distribution, not a conversational sentence. The architecture and measurements presented here therefore concern this form of decision reading, and should not be taken as a test of every stage involved in a full voice-agent exchange.

Single-Token Supervision and Paralinguistic Understanding

A key finding of the preprint is that traditional transcript distillation prevents the LLM from learning speaker characteristics (such as gender or emotion), because the distillation 'teacher' only sees the text and does not hear the voice. By supervising only the answer token with cross-entropy, the model learns to extract these paralinguistic signals directly from the audio.

The authors report that, with this technique, accuracy in identifying gender and emotion reaches 90%, starting from a chance level (55% and 28% respectively) when only distillation is used. This demonstrates that the bottleneck is not in the connector, but in the training objective.

The contrast between transcript distillation and answer-token supervision concerns what information the training signal can convey. A transcript-based teacher receives text, so it cannot directly supply labels about vocal properties that are absent from the transcript. In the reported approach, the answer token is supervised against the decision target while the model has access to audio representations. The authors interpret the resulting improvement on gender and emotion identification as evidence that this objective makes paralinguistic information available to the model during learning.

The reported 90% accuracy is tied to the authors’ evaluation and datasets; it is not a guarantee for other speakers, recordings, or conditions. The comparison also concerns the stated baselines: with transcript distillation alone, the reported starting figures are 55% for gender and 28% for emotion. These results support the authors’ account of the training objective as an important factor in those tasks, but they do not establish that the connector is never a bottleneck or that the same accuracy will hold beyond the evaluated data.

The example’s probabilities make clear that the system’s answer is represented as a choice with an associated score. A value of 0.92 for “Yes” and 0.08 for “No” expresses the model’s distribution over those two options; the method does not need to formulate a sentence explaining the decision. The example illustrates how a bounded question can be answered through a token-level readout, rather than demonstrating accuracy across all possible turn-taking decisions.

Latency and Compute Capacity Results

In tests conducted on an 8-GPU node (H200), the system answers 80 decisions on 8 utterances in approximately 0.1 seconds. In comparison, a cascade approach that transcribes and then generates a JSON response takes 1.57 seconds per event, representing a 17-fold reduction in latency.

Service capacity also improves significantly: while the JSON approach cannot serve any load within a 0.5-second budget, the single-token reading method serves between 17 and 20 events per second on a single GPU. The main advantage lies in latency under conversational budgets, not necessarily in raw throughput at maximum saturation.

The 0.1-second result is reported for a particular setup: an eight-GPU node using H200 hardware, answering 80 decisions associated with eight utterances. It is therefore a measurement under the study’s stated hardware configuration, not a general latency estimate for a single device or for every deployment environment. The comparison with 1.57 seconds is likewise between the tested single-token reading method and the tested cascade that transcribes audio and then generates a JSON response. The reported 17-fold reduction describes that comparison, rather than a universal advantage over every alternative.

The capacity figures address a specified conversational time budget of 0.5 seconds. In that comparison, the JSON approach serves no events within the budget, while the single-token method is reported at 20.4 events per second in the table and described in the text as between 17 and 20. These figures should be read in relation to the tested methods and conditions. They do not show that the same rate will be achieved on other hardware, nor that maximum-saturation throughput is the principal advantage; the original account identifies latency within conversational budgets as the main point.

Limitations and State of Evidence

It is fundamental to highlight that these results come from a preprint (arXiv:2610.02638) and have not been peer-reviewed or independently replicated. Performance figures, such as the 90% accuracy in paralinguistics or the 0.1 s latency, are author claims based on their own datasets and hardware configurations.

Although the authors announce the release of weights, training recipes, and a bilingual dataset (qa100), the effective availability of these resources has not been independently verified. Furthermore, the evaluation focuses on closed-set tasks and does not evaluate continuous streaming input, which limits direct generalization to all production voice agent scenarios.

The evidence remains preliminary because the work is a preprint without peer review or independent replication. Its numerical results, including the reported latency and paralinguistic accuracy, are claims based on the authors’ datasets and hardware setup. The hardware-specific timing result does not establish performance on less powerful equipment, and the evaluation does not measure continuous streaming input. These limits leave open how the reported results would transfer beyond the tested configuration and closed-set tasks.

The authors announce weights, training recipes, and the bilingual qa100 dataset, but their effective availability has not been independently verified. That uncertainty should not be converted into a claim that the resources are unavailable: the supplied evidence establishes neither verified access nor inaccessibility. Anyone considering the resources for research or development would need to confirm their availability directly. The evaluation’s focus on closed-set decisions and its lack of continuous-streaming tests also limit what can be concluded about broader production use, without showing that the method cannot be useful there.

Keep exploring

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.