AesCode Infographics Chart: what the evidence lets us verify—and what is missing for comparison

State of the evidence: documentation and code are available, but not a complete reproduction

AesCode’s public documentation makes it possible to say more than simply that the suite has a channel called Chart. The official repository describes the evaluation system, lists its verifiers, and provides instructions for running them. The model card adds results, generation settings, and a general description of how the score is obtained. It is therefore possible to examine the stated design and run parts of the harness on data available to the evaluator.

That does not mean the published figures can be reproduced from scratch. The repository says that it distributes code, not datasets. It also explains that a sample consists of a prompt, verifiable requirements, and a reference image. However, an available format or example does not show that the 300 evaluated cases have been published. The main gap is between having evaluation tools and having all the inputs needed to repeat the original experiment.

The evidence considered here comes from two primary project resources: Microsoft’s repository and the AesCode-8B model card. Both describe the work as AesCode and link to an article identified as an arXiv preprint. The information consulted does not allow the article’s peer-review status to be independently verified, so this piece does not present it as peer-reviewed research or an independent replication. The results should be understood as figures reported by the project, not as external validation.

How Chart fits into the suite

The results table presents Text, Bound, and Chart as programmatic checks aggregated into Rule. Content, Layout, and Style make up Visual; Overall is the mean of Rule and Visual. This structure matters because the Chart score is neither the overall visual-evaluation result nor a standalone measure of aesthetic quality. It is one dimension of verifiable correctness associated with charts.

The repository describes chart requirements using expected data, such as series with names and values or lists of numbers, along with the chart type. In the system description, the tablechart channel checks table and chart data. The AesCode-8B model card presents Chart as a programmatic component of Rule. This documentation supports the broad relationship, but it is not enough to establish every detail of the scoring function or how individual cases are aggregated.

The terminology requires care: the initial proposal names Bound, Content, Layout, and Overall as related components, but the documentation also includes Text, Chart, Rule, and Style. These are distinct categories, not synonyms. For example, Bound has its own score, as does Chart; Rule groups them together with Text. Similarly, Content, Layout, and Style are combined into Visual, which then contributes to Overall. Comparing one model’s Chart result with another model’s Overall result would mix different aggregation levels.

What the figures show—and what they do not

In the published table, AesCode-8B scores 87.79 on Chart and AesCode-32B scores 90.37. The model card says the results were calculated on 300 infographic samples, with three generations per prompt and no selection among generations. These details let us describe the reported result and the accompanying aggregate procedure. By themselves, they do not tell us how much the figures might change with a different case selection, implementation, or independent rerun.

The same source reports generation settings for the scores: temperature 0.8, top-p 0.95, up to 12,000 output tokens, and a 24,576-token context. This information helps define the reported configuration. It should not be mistaken for an exhaustive specification of the experimental conditions for every compared system: models, available tools, versions, and environments may differ. Attributing a difference to model capability would require comparable, case-by-case documented configurations.

The figures must also be interpreted according to the channel’s scope. Chart evaluates one aspect of the suite related to chart data; it does not automatically establish that visualizations are aesthetically good, that outputs are factually correct beyond the supplied requirements, or that one model is superior across all visual-generation tasks. The documentation describes how Rule and Visual are composed, but the evidence supplied here does not include an external audit of the case set or an estimate of statistical uncertainty for each score.

Gaps that limit a methodological audit

The first limitation is data availability. The repository says it publishes code, not datasets, and describes a sample format containing a prompt, verifiable requirements, and a reference image. Without the complete Chart case set, it is not possible to check which chart types appear, how difficulty is distributed, whether there are duplicates, or whether some chart families carry more weight than others. An example in the repository helps explain the format; it does not necessarily represent the full evaluation.

The second limitation is the complete operational specification. Evaluation code is available, and the documentation explains the channels and dependencies, but auditing the results requires fixed versions, parameters, failure handling, aggregation rules, and behavior when a requirement cannot be verified. The repository itself warns that if PaddleOCR is missing, the Layout channel returns zero rather than abstaining. That warning concerns Layout and does not demonstrate a defect in Chart; it does show why dependencies and error paths must be recorded when reproducing any score.

The third limitation concerns comparisons across models. The documentation provides generation settings for AesCode-8B scores, but a fair comparison requires knowing and aligning the relevant conditions for every system: model versions, templates, inputs, tools, output limits, and retry or selection policies. The available evidence is not sufficient to claim that every model in the table received exactly the same treatment in all these respects. The table can therefore be read as a project-reported result, but comparability should still be checked before using it for high-stakes decisions.

What to require for a future audit

At a minimum, a reproducible evaluation would require the exact version of the code used, an unambiguous description of the Chart metric, specifications for its inputs, and an explanation of how results are aggregated from individual charts to the reported score. It should also document dependencies and the execution environment, along with the criteria for rendering errors, undetected charts, missing data, and non-applicable cases. The repository provides code and setup steps, but the list of required inputs remains incomplete while the evaluated dataset is unavailable or cannot be requested through a transparent process.

Publishing the cases would make it possible to review their coverage and establish reproducible splits. If there is sensitive material to protect, the project could publish identifiers, specifications, generation scripts, or controlled access under clear rules. The essential requirement is that authorized third parties can check what was scored and that the dataset does not change silently between versions. A version record and checksums for the files would help detect changes.

An audit should also distinguish replication from extension. Replication means running the same evaluation with the same dataset and configuration, then comparing results. Extension means testing other domains, chart types, or models; that may be useful, but it does not by itself verify the original figures. Finally, results should be reported per case, together with measures of variation across generations—not just aggregate averages. This would help distinguish a consistent improvement from a difference concentrated in a few examples.

Conclusion: comparison is possible, but its scope is limited

We can say today that AesCode documents a Chart channel within a broader evaluation, publishes scores for specific models, and provides code for running the evaluation. We can also say that the project describes 300 samples, three generations per prompt, and an aggregation in which Chart is part of Rule. These facts allow an informed reading of the reported structure and inspection of the software.

The available evidence does not support the claim that any team can reproduce the published table exactly: the repository says the data are not distributed. Nor does it establish an independent replication or a fully controlled comparison across all models. The rigorous position is to treat the scores as results reported by the project—useful for guiding questions and experiments, but requiring external verification before they are used as a definitive selection criterion.

For researchers and evaluation teams, the practical decision depends on the intended use. As an exploratory reference, the results may help identify reported differences between systems. To establish a conclusive ranking, support a purchase, or make a general performance claim, it is prudent to require the data, versions, operational specification, and per-case results. Without those elements, the evidence can describe the benchmark but cannot complete an audit of its comparability.

Keep exploring

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.