AI Decision Logs: How to Document Risks, Controls, and Evidence

What an AI decision log is and what it is for

An AI decision log is a structured history of significant decisions about a system’s design, configuration, validation, deployment, and changes. Its purpose is not to accumulate documents, but to make it possible to reconstruct what was decided, why, who took responsibility for the decision, what control was put in place, and what evidence supports the claim that the control was verified. It can cover technical and organisational decisions, from choosing a threshold to defining who reviews the system’s outputs.

The log helps explain the process to internal teams, auditors, or authorities, but it is not, by itself, proof of compliance. Nor does it replace technical documentation, operational records, risk assessments, or legal obligations that may apply to a particular organisation or system. The Spanish Agency for the Supervision of Artificial Intelligence’s guidance on records addresses retention obligations for certain operators of high-risk systems; those obligations should not automatically be extended to every AI tool.

Set the scope from the outset. State which system and version the log covers, which intended uses are included, who maintains it, and which kinds of decisions must be recorded. A change that does not affect behaviour or use might be documented briefly; a change to data, thresholds, the user population, or human oversight may require a fuller assessment. Decisions about what to record and how much detail to include should be proportionate to impact and context, rather than presented as a universal legal rule.

Practical fields for documenting each decision

There is no set of fields that can be declared the minimum sufficient set for every system, use, and obligation. As an operational starting point, an entry should identify the system and version, date, owner, decision, risk or need that prompted it, associated control, available evidence, and verification outcome. Adding status, limitations, and a review date or trigger makes subsequent maintenance easier.

A clear entry distinguishes between risk, control, and evidence. Risk describes possible harm in a specific context; a control is a concrete measure intended to prevent, reduce, or detect it; evidence shows what was checked and with what result. For example, “classification errors” is too vague unless the population and use are identified. “Human review is retained for low-confidence cases” describes a measure, but it is still necessary to specify how a low-confidence case is identified and how it is demonstrated that review took place.

Record stable links to artefacts, not just generic names. A useful reference can include an identifier, version, date, authorised internal location, and custodian. Protect personal or confidential information with appropriate access controls, and avoid copying sensitive content into the log when it is not needed. If evidence is no longer available, record the issue and the action required to recover it or repeat the check.

Linking risks, controls, and tests

Build a traceable chain: specific risk, mitigation objective, implemented control, verification method, and result. Each link should be traceable through consistent identifiers. If one control mitigates several risks, record each relationship; if one risk requires several measures, document each control and the dependencies between them. This prevents a list of risks from becoming disconnected from engineering activities or operational decisions.

The evidence must match the configuration and use being assessed. A test report for an earlier version can provide context, but it does not automatically demonstrate the effectiveness of a later change. State the environment, data or scenarios used, acceptance criteria, outcome, and limitations. If a test does not cover all real-world conditions, record the gap and the additional oversight or measure applied.

Quantitative testing is not always enough. For an organisational measure—such as human review—it may be relevant to retain current procedures, samples of reviewed cases, escalation records, and checks that staff received instructions. The required level of evidence depends on the risk and the claim being made. Algorithmic auditing can help structure a review of mitigations and controls, but a general recommendation does not, on its own, demonstrate that a specific measure was applied or is effective.

Creating and maintaining the log

Start with an inventory of systems and versions, and define who has authority to approve decisions and who maintains the evidence. Agree on shared conventions for identifying risks, controls, and artefacts. Every entry should have an owner; a generic team or an unassigned mailbox makes follow-up difficult. Also define who can access the log and how changes and corrections are managed.

Record decisions when they are made or when the evidence is validated, not months later during an audit. Link each relevant change to the affected version and maintain a history that shows when the log was updated. When reviewing an entry, check whether the risk is still relevant, the control is still in place, the evidence matches the current version, and any limitations or outstanding actions remain open.

There is no single review frequency that can be recommended as a general obligation for every system. Set a periodic review schedule proportionate to the context, and add event-triggered reviews for a new version, a change of data or provider, a change in intended use, incidents, monitoring results that differ from expectations, or applicable regulatory changes. If nothing has changed, record the review and its conclusion rather than leaving the absence of an entry open to interpretation as an omission.

Auditability criteria and common mistakes

An auditable log can be understood without relying on the memory of the person who created it. Each entry should be readable, dated, attributable, and linked to artefacts that authorised reviewers can access. Completeness does not mean including everything; it means that the information needed to reconstruct the decision is present, gaps are identified, and claims do not go beyond what the evidence supports.

Avoid controls described with imprecise verbs such as “monitor” or “improve” unless the entry explains who performs the activity, when it happens, and how completion is verified. Also avoid presenting a policy, an impact assessment, or an approval tick box as evidence of effectiveness. These documents may be relevant, but they demonstrate different things. The Spanish Data Protection Agency’s guidance on auditing processing activities that include AI provides institutional context for reviewing processing, but it does not make a log template an automatic guarantee of compliance.

Before an audit, select several decisions and trace them in both directions: from each risk to its control and evidence, and from a test back to the decision that uses it. Check that links still work, versions match, owners are identified, and exceptions are explained. If you find a gap, record it as a finding, with an owner and follow-up; do not conceal it by retrospectively replacing incomplete evidence.

Practical criteria for getting started

Start with a short template and test it against a real decision before turning it into a long form. If a field does not help identify, explain, verify, or maintain a decision, consider whether it belongs in the log or in another linked artefact. The structure should make it possible to follow the chain from risk to measure to evidence without unnecessarily duplicating technical documentation or operational records.

Adjust retention, access permissions, and level of detail to the applicable obligations, data type, and audit needs. Consult current regulatory documentation and institutional guidance for the specific case: obligations can depend on the organisation’s role, the type of system, and its use. The log helps organise and explain evidence; it does not replace legal analysis or, on its own, demonstrate that a system complies.

As a final test, consider an entry ready when another authorised person can reconstruct what was decided, which risk was addressed, what control was applied, what evidence was examined, what its limitations are, and when it should be reviewed again. If any of those elements is missing, state the uncertainty and assign an action. Being transparent about a documented gap is more useful than claiming effectiveness without verifiable support.

Keep exploring

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.