Semantic entropy as a signal for routing queries between AI models

The challenge of routing requests between models of different capabilities

Systems that combine smaller and larger AI models need to decide, at run time, which queries should go to the more capable model. The aim is to balance cost and latency without sacrificing answer quality. Sending every request to the larger model can waste resources; sending every request to the smaller one can reduce accuracy on harder tasks.

The preprint investigates a low-cost signal that can be calculated before calling the larger model and used to predict whether escalation would help. It evaluates semantic entropy, a measure of how much the meanings of a smaller model’s answers differ when it generates multiple responses to the same query.

How disagreement can guide escalation

The approach generates several answers to the same query using the smaller model, then calculates their semantic entropy. If that value exceeds a chosen threshold, the system treats the smaller model as unreliable and escalates the query to the larger one.

The underlying premise is that smaller models are more consistent on tasks they can handle and less consistent when a task exceeds their capabilities. Semantic disagreement therefore serves as a proxy for task difficulty or for the smaller model’s inability to solve the task. The supplied abstract, however, does not specify the exact calculation, threshold or escalation policy used in the experiments.

Reported benchmark results and metrics

The paper’s abstract says the study evaluates the signal on three benchmarks and two model families. On GSM8K, using a small–large model pair whose sizes differ by about a factor of twelve, semantic entropy distinguishes mistakes made by the smaller model with an AUROC of 0.871.

The abstract also reports that routed accuracy improves by up to nine percentage points over random escalation at matched cost. These findings suggest that the signal can help identify errors in this particular evaluation setting. They do not, by themselves, establish that routing will improve performance in other tasks or in production.

Methodological limits and potential pitfalls

The preprint describes an apparently strong result on a synthetic benchmark that turned out to be misleading: a simple rule based only on question difficulty, without using a model, matched semantic entropy almost exactly. This raises the possibility that a routing signal is tracking how difficult a question looks rather than adding information about the model’s uncertainty.

The abstract also notes that two reasonable definitions of “successful escalation” can produce very different results on the same data. Another concern is that a benchmark may leave little room for any signal to outperform the simple strategy of always using the larger model. Finally, the cost of sampling answers live may make a routing system more expensive than calling the larger model directly.

What to check before production use

Before deploying this approach, teams would need to establish whether the findings can be reproduced and whether the signal remains useful on their own models, tasks and traffic. The preprint presents a general checklist for evaluating escalation signals, and its abstract describes a cheaper alternative that reuses cached outcomes from past queries. It says the authors predicted that this alternative would fail to transfer to a new dataset, where its AUROC fell from 0.908 to 0.518, close to chance.

The results on GSM8K do not establish that semantic entropy will generalize to other domains, languages or model families. A production evaluation should measure end-to-end inference costs, including the extra calls needed to sample multiple answers, as well as latency and the frequency of escalation. The available abstract does not provide enough detail to determine the exact sampling setup, cost accounting or escalation thresholds used in the experiments.

Keep exploring

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.