AI Safety Audit
Contextual evaluation engine
The engine turns your requirements (regulations, standards, internal policies, customer clauses) into tests, runs them on the real system and produces verifiable evidence. AI writes the tests within strict rules, their logic is formally verified, and our experts settle what stays ambiguous.
What is an AI Safety Audit?
An AI Safety Audit tests an AI agent against the requirements that apply to it, on the real system, and produces evidence a third party can review. This page describes the engine behind it: what it does today, what is in development, and the research it builds on.
From legal text to testable requirement
Law is ambiguous: one article allows several readings, and two texts can contradict each other. A language model picks one reading without flagging it. Our formal verification makes every ambiguity explicit. Our experts resolve it, or it is kept as a documented assumption. Every tested requirement carries its source text and the interpretation applied.
Under the hood
- Regulatory knowledge base
- Constrained generative AI
- Formal verification
- Expert review
A structured, cumulative memory
Thousands of test transcripts do not add up to knowledge. Our engine structures what it observes into linked entities: applicable obligations, system components, observed behaviors. It reasons on established facts, and each campaign starts from what is already known.
Under the hood
- Knowledge encoding
- Entity-based memory
- Persistent test plan
Personas drawn from real usage
A generic test set evaluates a system against users who do not exist. Our personas are built from the system's context, or rebuilt from your production logs. The tests reproduce how your system is actually used.
Under the hood
- Generic tests, without personas
- Personas based on context
- Personas based on production logs
Adaptive exploration of the behavior space
A fixed test list only finds what you already knew to look for. Our engine picks each scenario based on the answers already received. Judges assess every answer criterion by criterion, and testing effort concentrates where the system fails.
Under the hood
- Adaptive strategy for each conversation
- Adversarial scenario generation and quality control
- LLM judges, personal data detection, fairness
Full traceability, from text to verdict
A global score tells you neither what failed nor why. Every verdict is linked to the test run, the answer received and the requirement checked. Results aggregate by requirement, export for audit and plug into CI. This traceability exists today; measuring the reliability of the whole chain is one of our research challenges.
Under the hood
- Compliance table linked to requirements
- Remediation plan
- Exportable evidence pack
- Checks in CI
See the engine at work on your agent
A call with an expert: your agent, the requirements that apply to it, and how the engine tests them.