Many AI programmes now have dashboards, scores and red-team reports. That is useful, but it is not the same as auditability. An auditable AI evaluation lets someone who did not run the test reconstruct the conclusion: what system was assessed, what changed, what evidence was used, what failed, who accepted the remaining risk and whether the same conclusion still holds in production.
Auditability starts where the score ends
A scorecard can tell a team that a system passed 42 controls out of 50. It cannot, by itself, prove that the right system was tested, that the evidence came from the right environment, that the failed scenarios were preserved, or that the scoring method was reliable enough for the decision being made.
That distinction matters because AI systems change quickly. A prompt update, retrieval reindex, model provider change, new tool permission, observability connector or policy exception can all change the risk surface without changing the name of the system. If the evidence does not capture those changes, the audit trail describes a past system, not the one running today.
The audit question is not only whether the AI passed. It is whether the organization can prove what passed, when, why and under which conditions.
The system is the object of audit, not the model
A production AI system is more than a model endpoint. It includes instructions, datasets, retrieval sources, access controls, tools, connectors, human review rules, monitoring, fallback paths, owners and suppliers. An evaluation that ignores that context can still be technically interesting, but it is weak evidence for governance.
For a customer support assistant, the audit object is not only the LLM. It is the policy corpus, the permission model, the answer templates, the logging configuration, the escalation rule, the CRM connector and the process for correcting bad answers. For an agent, it is also the tool catalogue and the approval rule for each action. For a scoring model, it is the dataset lineage, feature pipeline, threshold policy and human override process.
A defensible evaluation record has five layers
The minimum useful unit is not a PDF. It is a chain that links the system, obligation, control, run, result and evidence. Without that chain, teams end up with screenshots and narratives assembled after the fact.
- System context: owner, purpose, users, data, model, provider, tools, environment and version at the time of the run.
- Applicable controls: obligations, internal policies, risk dimensions, criteria, thresholds and the reason each control applies.
- Evaluation method: test source, sampling rule, scorer, judge model, prompt or rubric version, deterministic tools and human review rules.
- Evidence: scenarios, production traces, retrieved sources, tool calls, findings, logs, screenshots only when necessary, and the raw observations behind each verdict.
- Decision trail: failed controls, insufficient evidence, remediation, accepted risk, accountable person, expiry date and next review trigger.
This is why auditability belongs inside the evaluation workflow, not after it. If the run does not preserve context, method and evidence while it executes, the team has to reconstruct the file later from partial logs and memory.
Synthetic tests and production traces answer different questions
Offline evaluations are controlled. They can probe prompt injection, harmful content, data leakage, grounding, fairness variants and known edge cases before a release. They are good at asking: can this system fail if pushed in a specific way?
Production traces are different. They show what real users, real documents, real tools and real latency did to the system. They are good at asking: did the system behave properly under actual use, and did the controls still work after deployment?
An auditable programme needs both. The offline run should preserve the generated scenarios and their metadata. The online run should preserve the sampled sessions, timestamps, source trace identifiers, tool calls, retrieved documents and sampling rule. Otherwise the organization cannot explain why a clean pre-production score did or did not predict production behaviour.
Logs are not evidence until they are connected to a control
The AI Act makes logging and lifecycle monitoring concrete for high-risk systems. Article 12 requires automatic record-keeping capabilities. Article 19 requires providers to keep generated logs under their control for an appropriate period. Article 72 requires post-market monitoring, and Article 73 creates serious incident reporting duties.
But logs alone are not enough. A million trace rows do not tell an auditor which obligation was tested, which control failed, whether the evidence came from staging or production, or whether the issue was remediated. Auditability starts when logs are attached to the control they support.
- Weak evidence: a dashboard showing token count, latency and error rate.
- Better evidence: a replayable session showing prompt, retrieved source, tool call, response and timestamp.
- Audit-ready evidence: the same session linked to a control, a criterion, a verdict, a system version, a remediation decision and an owner.
The judge is part of the method
Many AI evaluations use LLMs as judges. That can be practical, especially for grounded answers, policy reasoning and conversational behaviour. It is also a methodological dependency. The judge can be sensitive to prompt wording, answer order, verbosity, provider policy and its own preferences.
A serious report should therefore say which parts of the verdict are deterministic, statistical, machine-learning based, LLM-mediated or human-reviewed. It should preserve the judge configuration and explain whether deterministic findings can override an LLM judgement. Otherwise a failed control and a judge preference can look identical in the final score.
Point scores need uncertainty
An evaluation score is an estimate, not a property of the system. Ten synthetic scenarios can reveal a serious failure, but they cannot prove that the system is safe with audit-grade confidence. A 100 percent pass rate on three tests should not read like a certification.
For auditability, the report should expose the sample size, sampling strategy, pass rate, confidence interval where applicable, underpowered criteria and assurance level. Some controls can be exhaustive, such as scanning every connected table for a class of data. Others are sampled, such as production conversations or adversarial prompts. The report has to say which is which.
Readiness is not approval
A useful AI governance platform can summarize production readiness, but it should not pretend to approve production on behalf of the organization. The decision depends on the system context, applicable obligations, client risk policy, failed controls, missing evidence, monitoring coverage and accepted residual risk.
The clean decision object is a dossier, not a single grade. It should show blockers, failed critical controls, insufficient evidence, accepted risks, last evaluation date, monitoring coverage, owner and exportable evidence. A risk committee can then decide whether the remaining risk is acceptable, whether remediation is required, or whether the system should stay out of production.
What to build first
Teams do not need a perfect audit archive on day one. They need to stop losing the evidence that makes their conclusions defensible. Start with the systems where the business impact is highest: customer-facing assistants, agents with tools, systems touching regulated data, and models supporting eligibility, fraud, compliance or financial decisions.
- Inventory the system: purpose, owner, data, model, tools, providers, environments and fallback path.
- Map controls to evidence: obligation to control, control to criterion, criterion to run, run to trace or artifact.
- Version the method: criteria, prompts, rubrics, thresholds, judge model, deterministic tools and sampling settings.
- Separate statuses: passed, failed, insufficient evidence, not evaluated and accepted risk should never collapse into one score.
- Keep replayable records: scenarios, production traces, tool calls, retrieved documents and findings should remain inspectable after the run.
- Export for humans and systems: PDF for committees, CSV for control matrices, JSON or API for GRC, SIEM and workflow tools.
The organizations that get this right will not be the ones with the longest AI policy. They will be the ones that can answer a narrow question quickly: for this system, in this version, what evidence shows that the controls worked, and what did we do when they did not?
References
- Regulation (EU) 2024/1689 (AI Act), Articles 11, 12, 19, 72 and 73, Annex IV, eur-lex.europa.eu and ai-act-service-desk.ec.europa.eu.
- NIST, Artificial Intelligence Risk Management Framework 1.0, Govern, Map, Measure and Manage, nist.gov/itl/ai-risk-management-framework.
- NIST AI 800-3, Expanding the AI Evaluation Toolbox with Statistical Models, February 2026.
- NIST AI 800-4, Challenges to the Monitoring of Deployed AI Systems, March 2026.
- ISO/IEC 42001:2023, Artificial intelligence management system, iso.org/standard/42001.
- OpenTelemetry, Generative AI semantic conventions and AI agent observability materials, opentelemetry.io.
- Anthropic, Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations, 2024.
- RAND, Judge Reliability Harness: Stress Testing the Reliability of LLM Judges, 2026.
- W3C, PROV-DM: The PROV Data Model, w3.org/TR/prov-dm.
- IETF Internet-Draft, Verifiable AI Provenance Framework, draft-kamimura-vap-framework-00, 2026.