Claude Opus 4.5 Evaluations: What the Tests Compare and What They Can Actually Tell Us

A score only makes sense alongside its protocol

Language-model evaluations do not measure a single capability. One test may examine how an agent and its tools solve programming tasks; another may measure performance on an internal task suite; a third may assess responses to knowledge questions. Even if each publishes a score or a ranking, that does not mean they measure the same thing or allow models to be ordered interchangeably. At a minimum, interpreting a score requires knowing the task, the evaluation set, the metric, and the execution conditions.

For Claude Opus 4.5, the supplied sources identify several kinds of material: an Anthropic announcement that mentions testing with Junie, the model’s System Card, an independent paper on models as IDE agents, and an aggregated benchmark listing. An Artificial Analysis analysis was also proposed for review, but no page from that organization is included among the verified sources provided here. It is therefore not possible to document the method or the position attributed to the model in that analysis.

The available evidence makes it possible to describe the general scope of some documents and identify what would need to be checked in their full versions. It is not enough, however, to reconstruct complete protocols, verify every result, or establish an overall ranking. This distinction is not a judgment about the model’s quality; it reflects the limits of what can be supported by the available materials.

What the located sources contribute

Anthropic’s official page about Claude Opus 4.5 is a provider source. The available excerpt says that, in tests with Junie, Anthropic’s coding agent, Opus 4.5 outperforms Sonnet 4.5 on the benchmarks mentioned in that announcement and needs fewer steps to solve tasks. However, the supplied material does not name those benchmarks or detail the tasks, number of runs, active tools, or exact criterion for “fewer steps.” The statement should therefore be presented as an assertion by Anthropic, not as an independent comparison with a fully reconstructed protocol.

The verification note for the same page also refers to GitHub Copilot, whereas the quoted excerpt mentions Junie. The supplied information does not establish whether these references concern different parts of the page, an imprecise description, or a discrepancy in the record. The full page should be reviewed before describing the environment used. The ambiguity should not be resolved by assuming that Junie and GitHub Copilot were part of the same experiment.

Anthropic’s System Card is an original institutional document. The information provided indicates that it compares Opus 4.5 with other models across a task suite, but it does not include the test names, metrics, or specific conditions. The existence of a comparison in the document is not enough to claim that all the tasks represent programming or agent use; that characterization requires checking the full content.

The paper located on arXiv concerns the evaluation of models as IDE agents in real-world environments and, according to the available note, mentions variability for Opus 4.5. The supplied summary does not identify the specific tasks, model set, harness, number of runs, or calculation behind that variability. In particular, it does not let us verify the standard deviation value of 0.027 raised in the review questions, or establish which metric or set of results it applies to.

Finally, BenchLM provides an aggregated listing that assigns Opus 4.5 a position and an overall score based on result rows displayed by the listing itself. It is a secondary aggregation source, not a primary description of the original protocols. Without knowing how those rows are selected, normalized, and combined, its score should not be treated as a universal measure of capability.

Junie testing: a provider claim, not a complete protocol

Anthropic’s excerpt is useful for identifying a specific claim: in tests conducted with Junie, Opus 4.5 reportedly outperformed Sonnet 4.5 on the benchmarks mentioned and needed fewer steps to solve tasks. The statement comes from Anthropic and should be attributed accordingly. The supplied material does not confirm the evaluation sets, sample size, or whether each model received identical context, tools, and budget.

“Fewer steps” does not, by itself, mean better overall performance. Interpreting that measure would require knowing what counts as a step, whether failed attempts or tool use are penalized, whether tasks must be completed correctly, and how partial solutions are handled. Fewer interactions could matter in a particular environment, but they do not automatically tell us about cost, latency, reliability, or code quality.

The notes also associate Anthropic’s Opus 4.5 page with GitHub Copilot. Because the full text explaining that reference has not been provided, it is not possible to tell whether it describes another evaluation context, an integration, or a different task. A rigorous audit would keep the two mentions separate until their conditions have been reviewed, rather than merging them into a single experiment.

System Card: a relevant document, with details still unverified

The Claude Opus 4.5 System Card is a primary provider source and therefore the appropriate place to check which evaluations Anthropic chose to publish and how it presents them. The available verification note only confirms that the document compares Opus 4.5 with other models across a task suite. It does not name the tasks or allow us to reconstruct results by category.

To interpret that suite, each test would need to be reviewed to establish which capability it aims to measure, which models are compared, and which metric is used. It would also be necessary to determine whether the models received the same instruction format, tools, and budget. If the System Card reports results on different tasks, they should not be summarized as if they formed a single measure of “intelligence” or programming performance.

Anthropic’s platform documentation identifies a page describing Opus 4.5 that may help verify model identifiers and limits. However, the supplied note does not include the exact identifier or specific limits, nor does it connect that documentation to the runs for each benchmark. As a result, we cannot claim that the cited tests used a particular configuration until the complete documents and the correspondence between model version and test have been checked.

Independent IDE-agent evaluation: what is missing to interpret variability

The paper located on arXiv offers a different perspective from provider announcements: it studies models as IDE agents in real-world environments. The verification note says that it addresses variability in Opus 4.5’s results. The difference in provenance matters, but it is not enough to assess methodological independence or compare its figures with Anthropic’s. Doing so requires examining the complete method, data, and evaluation procedure.

Variability can only be interpreted if we know which result is repeated and under what conditions. A standard deviation might describe differences across runs, tasks, instances, or some other unit; each choice changes what it means. The available material does not confirm what the value 0.027 mentioned in the verification questions represents, how many runs support it, or whether it is calculated from a success metric, a score, or another variable. The value should therefore not be presented as a verified result.

It is also necessary to distinguish variation between model runs from variation between tasks. An agent might obtain similar results when the same task is repeated while performing very unevenly across different tasks. Separating these phenomena requires at least the number of tasks and repetitions, the statistical unit, the metric definition, and the method used to aggregate results.

The label “real-world environments” does not, by itself, specify which conditions were reproduced. To understand what the test covers, it would be necessary to verify the repositories or projects, instructions, access to terminals and tools, time or interaction limits, and how a change is judged to solve the issue. The supplied notes do not detail these elements, so the evaluation cannot be used here to claim that it represents everyday use across all development teams.

Artificial Analysis and aggregators: a position does not replace methodology

The editorial proposal calls for verifying Artificial Analysis’s Omniscience Index and the position attributed to Opus 4.5. However, the list of verified sources does not include an Artificial Analysis page. For that reason, it is not possible to state what the index measures, what method it uses, what result it assigns to the model, or under which configuration that result was obtained. The question must remain open until the relevant source is available and reviewed.

The BenchLM listing is among the materials located. Its note reports a position, an overall score, and a set of benchmark rows presented on the page as results. These details describe what the aggregator publishes; they do not, by themselves, validate the test selection or the process of combining different results. Interpreting a position would require the rules for normalization, weighting, source inclusion, and updating each row.

An aggregated ranking can be useful for exploring results or locating tests, but it should not be presented as a neutral measure of general capability without explaining how it is constructed. If a score combines heterogeneous tasks, changes in selection or weighting could change the position. And without traceability to the original publications, it is not possible to check whether the conditions for all rows are comparable or whether they refer to the same model version.

Comparability, contamination, and reproducibility

Two results can be directly compared only when their conditions are sufficiently equivalent for the question being asked. For IDE-agent evaluations, this means reviewing, among other things, the exact model and version, reasoning level, instructions, available tools, the harness connecting the model to its environment, and execution limits. The summarized sources do not provide these details for every cited test, so equivalence between them cannot be established.

Contamination of evaluation sets requires a specific review. Assessing the risk would require knowing how tasks were constructed, when they were added, whether their solutions or prompts could have appeared in training data, and what controls the authors applied. The notes for the supplied sources do not describe contamination controls or analyses for the System Card, Junie tests, IDE-agent study, or BenchLM aggregation. This means the risk has not been assessed here; it does not mean contamination has been demonstrated.

Reproducibility also depends on more than publishing a score. Prompts, evaluation data, harness code, model configuration, scoring criteria, and run logs are all useful. If any of those elements are not published, the account should explain which part of the result cannot be independently repeated or inspected. The available summaries do not confirm what materials accompany each document or whether they are sufficient for reproduction.

These gaps should be treated as questions awaiting verification, not as proven flaws in the evaluations. A later review of the complete documents could provide details absent from the notes. Until then, the rigorous approach is to keep the results separate, attribute each claim to its source, and avoid inferences about an overall position.

Qualified conclusions

Based on the verified sources and notes, we can say that Anthropic reports favorable results for Opus 4.5 compared with Sonnet 4.5 in tests associated with Junie; that the System Card compares Opus 4.5 with other models across a task suite; that an independent study of models as IDE agents mentions variability for Opus 4.5; and that BenchLM publishes an aggregated score and position. Each statement corresponds to a different source and a different kind of evidence.

This material does not allow us to reconstruct the complete protocols or determine whether the tests use the same model identifier, tools, budget, harness, or metric. Nor can we verify the value and interpretation of the proposed standard deviation, contamination controls, availability of reproducible materials, or Artificial Analysis’s methodology. Consequently, the scores do not form a common ranking and do not justify an overall conclusion about which model is better.

For a team evaluating Opus 4.5 for programming or IDE-agent use, the practical next step is to request or consult documentation for the evaluation that most closely resembles its own workflow. Compare equivalent tasks, measure quality and completion rate, account for cost and latency, and observe variation across repeated runs. If the methodology is not published, that limitation should appear alongside the result, and the decision should not rely on an isolated aggregated position.

Keep exploring

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.