How to Verify AI Claims: A Protocol for Deciding What to Keep, Correct, or Remove

A convincing answer is not yet a substantiated answer

An AI response can sound precise and still get a date wrong, overstate what a source says, or attribute a fact to a document that does not support it. To review these responses rigorously, separate three questions: Is the claim true? Does the cited source exist and is it relevant? Does the passage cited support exactly what the claim says? These checks are related, but they are not interchangeable.

A citation can be genuine and lead to a reputable source without establishing the specific claim attached to it. The document may concern a different period, population, location, or measure. It may also qualify its conclusion in ways the AI response leaves out. So confirming that a link opens is only the beginning: find the passage and compare it with the full wording of the claim.

AI can help find candidate sources, separate ideas, and summarize long documents. The editorial decision, however, should rest on the underlying source and a reading of the relevant passage—not on the model’s confidence or its summary. European guidelines for trustworthy AI include transparency, traceability, human oversight, and auditability as general principles; they do not prescribe a specific method for validating citations. This guide proposes an operational protocol of its own, not a certified standard.

Define what you will check and how much effort it deserves

Start by breaking the response into claims that can be assessed separately. “The proposal is appealing” is an opinion; “the plan includes five accounts” is verifiable. Dates, quantities, names, requirements, attributions, citations, causal relationships, and descriptions of features can often be checked. A compound sentence may contain several elements: “The 2024 study surveyed 2,000 people and proves the program increased productivity” requires separate checks of the date, sample size, and causal conclusion.

You do not need to spend the same amount of time on everything. Prioritize what supports the conclusion, what could cause harm if false, and what is specific or likely to change. A figure used to justify a decision, a legal interpretation, or a sensitive claim about people deserves more scrutiny than a secondary detail that adds context. This prioritization is a practical working rule, not a scale validated by the available sources.

Before searching, define what evidence would resolve the question: which document has direct authority, what passage should be present, and which conditions—date, jurisdiction, population, version, or definition—could change the result. Setting that criterion in advance helps prevent confirmation-seeking and makes it easier to recognize when a source is not enough.

A claim-by-claim protocol: from source to verdict

Copy the claim exactly as it appears and assign it an identifier. Then look for the source with direct access to the information: the original report, an official dataset, the applicable legal text, product documentation, or the relevant primary record. A page citing a study can point you in the right direction, but it does not automatically replace the study. If the original source is unavailable, record that limitation instead of treating its contents as verified.

Open the document and locate the exact section, table, or passage. Check whether it supports the entire claim or only part of it. Review conditions, definitions, exceptions, and scope: an association does not necessarily establish causation; a result for one subgroup does not automatically describe the whole population; and a figure from one year should not be presented as current without checking. Preserve enough context for another person to repeat the review.

Also record the date you checked the source and any details that may change. If two pages lead to the same original research, do not count them as independent corroboration: they are two paths to the same evidence. If sources disagree, identify whether the difference stems from date, method, definition, or population. If the available evidence cannot resolve the disagreement, preserve the uncertainty or refrain from using the claim as the basis for a decision.

Classify the finding without turning uncertainty into falsity

A useful editorial scale distinguishes direct support, partial support, contradiction, insufficient sourcing, and unverifiable claims. “Direct support” means the evidence reviewed supports the claim with the right scope and conditions. “Partial” means it supports only part of the claim, or the wording omits an important qualification. “Contradicted” requires evidence that conflicts with the claim under relevant conditions; it does not mean the reviewer simply failed to find confirmation.

“Insufficient source” describes a problem with the available evidence—for example, a page mentions the topic but provides no data to support the conclusion. “Unverifiable” may be appropriate when the proposition is too vague, omits a time period, or has no reasonable way to be checked. None of these labels automatically proves the content false. Keep the state of the evidence separate from a judgment about truth.

The action should follow the finding. Keep a directly supported claim; correct or qualify a partially supported one; remove or carefully frame a contradicted claim; seek another source or explain the limitation when the source is insufficient; and do not present an unverifiable statement as fact. For high-impact uses, send the review to someone with relevant expertise.

An audit template another person can use to repeat the review

A review is reproducible if someone else can find the claim, retrieve the source, and understand why the decision was made. Save the original wording and the corrected version so the history of changes is not lost. Record the source’s title or identifier, its location within the document—for example, a section or table—any publication or update date shown, and the date you checked it.

Include the relevant passage or a precise description of where to find it, along with your assessment of support and any limitations you noticed. A list of links alone is not enough: if a link leads to a long page, another person needs to know which part you used and how you interpreted it. For sensitive content, record who reviewed and approved its use, in line with your team’s procedure.

Traceability helps teams review changes and correct errors, but it does not guarantee that the initial interpretation is right. If a source changes, an earlier version is no longer available, or the assessment depends on a translation, note that and avoid presenting the record as absolute certainty.

Test the protocol on known cases and understand its limits

Before using the process for real decisions, apply it to a small set of responses whose sources and problems the team already knows. Include nonexistent citations, genuine citations that do not support the conclusion, partly correct claims, outdated information, and cases with insufficient evidence. The goal is not to prove that the model is generally reliable, but to check whether the procedure catches specific failures and leaves an understandable record.

Track support failures—the source exists but does not support what is attributed to it—citation defects, corrections, and abstentions separately. Also check whether different people reach comparable classifications when they use the same instructions and documents. An accuracy rate on a small set can help improve the process; it cannot promise general model quality or establish that results will transfer to other topics, languages, or risk levels.

Some limits cannot be removed with a template. Sources may be inaccessible or incomplete; compound claims need several checks; sources may disagree because of definitions or dates; and a reviewer may trust a citation without reading it. CIAEM’s educational guidance recommends cross-checking information and verifying that references exist and correspond to the claims, but it does not, by itself, technically validate the protocol proposed here. If the evidence does not resolve the question, the responsible response is to say what is missing, request review, or abstain.

Keep exploring

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.