Most AI harms don’t come from bugs. A hiring model trained on years of biased resumes isn’t malfunctioning; it’s reproducing the patterns in its training data exactly as built. Cybersecurity has spent decades building tools and processes around finding and fixing defects like this, but a defect is exactly what’s missing here. That gap is a big part of why incident databases matter for AI in a way they never did for software.
In January, Trustible announced a partnership with the AI Incident Database (AIID), a registry of over 1600 real-world harm events caused by deployed AI systems. We’re now deepening that integration: linking individual incidents directly to entries in the AI Inventory, and building a recommendation system that surfaces relevant incidents for a client’s specific use case. Trustible regularly syncs with AIID to pull in newly documented incidents and automatically proposes links to relevant use cases, so users can catch foreseeable risks during pre-deployment review and stay current as new incidents emerge.
Analogy to Cybersecurity
Traditional cybersecurity operates on a reassuring premise: systems are safe when they function according to specification. An attacker typically needs a bug, a misconfiguration, or an unpatched vulnerability to gain access, and the entire discipline is built to hunt down exactly that. The CVE registry, a standardized catalog of known software flaws, lets vendors and researchers track and name vulnerabilities; bug bounties pay people to find them, and disclosure processes give vendors time to patch before details go public. Vulnerability scanners and patch cycles all rest on the same assumption that safety problems are discrete, identifiable defects that can be found, named, and fixed.
There have always been exceptions. Phishing succeeds against a login page that works perfectly, since the flaw sits in the person clicking, not the code. But cases like this stayed secondary, handled through training and process rather than the patch cycle itself.
For AI systems, the exception is the rule. Most incidents happen when the system works exactly as intended. A hiring model trained on biased historical data isn’t defective; it’s faithfully reproducing patterns in its training data. A generative model that hallucinates hasn’t hit a bug; it’s done exactly what it’s built to do, produce plausible text, in a context where plausible and true have diverged. There’s no CVE registry for that, because there’s no defect to name.
Role of Incident Databases
The database covers a broad range of incidents, including those that occurred when the system functioned as expected and when it failed. While they are often tagged with specific AI models, vendors and/or tools, they often apply more broadly. By reviewing the database, AI governance professionals can learn from previous incidents and build appropriate mitigations into their systems. For example, a company building a customer service support chatbot may want to learn about previous instances of chatbots criticizing the company and generating irrelevant banter.
A general risk identification process will cover broad categories like ‘Inappropriate Content Generation’, but incident review can allow organizations to test for specific failure modes (e.g. check how a chatbot includes off-topic anecdotes during interactions). Beyond this, the Incident Database can help with identifying novel risks that were not considered during a standard risk identification exercise.
Trustible’s Recommendation System
The Challenge
Alerting on new vulnerabilities in cybersecurity is relatively straightforward because a new entry in a CVE registry will often be tied to a specific library. System developers can easily be alerted to a problem with a library they use; many tools exist to do this. While AIID incidents can sometimes be linked to a system (e.g. ChatGPT), this approach is limited for two reasons: first, ChatGPT is a broad tool and not every incident involving it will be relevant to a user; second, some important and relevant incidents may have used a different system (e.g. Claude or Gemini) and thus would be missed. The signal that actually matters is intended use, not tech stack. A tool that reviews resumes carries similar risks regardless of which model powers it.
The raw text of incidents and use cases can’t be compared directly, since they serve different purposes. Incident reports focus almost entirely on what went wrong, with little to no description of the system itself. Use case documentation includes intended use explicitly, but it’s mixed in with many additional details relevant for AI governance and implementation, details that dilute the signal that actually indicates intended use and add enough length to work against embedding the document as-is.
The Solution
To solve these problems, we created a two step process. First we transform all AIID incidents and use cases into intended use summaries using Generative AI. For the incident reports, we generate a hypothetical intended use text that could have been written when the system was being proposed. For the use cases, we generate a summary that only contains intended use information and strips the other details. This solution is inspired by RAG systems, which face a similar problem where a question looks different from the documents that answer it. One fix there is “Hypothetical Document Embeddings,” where a question is transformed into a hypothetical document containing the answer, to help retrieve the actual documents that contain one. For the second step, we create embeddings (compact numeric representations of a text) and use a similarity metric to identify related incident-use case pairs. Using embeddings instead of text-overlap metrics ensures that similar meanings are captured accurately, even if the summaries reference different specific systems (e.g. Claude vs Gemini).
As an example, an incident report stating that GitHub Copilot users can produce source code subject to license requirements without attributing and licensing the code to the rights holder could be transformed into an intended use summary like “The AI-powered code completion assistant is integrated into a software development environment and suggests context-aware code and boilerplate as developers write code.” This would match a use case with detailed documentation for Claude Code, whose intended use summary may be “an AI-powered coding tool that provides real-time code suggestions and completions within developers’ IDEs to help write, review, and debug code.” These transformed summaries are similar, so a user documenting Claude Code in Trustible would be alerted to the Copilot issue and know to check for licensing concerns in their own risk evaluation.
Conclusion
Our new feature allows users to ground the risk assessment process in real-world examples of potential harms. Because we run the syncing and recommendation process on a regular basis, Trustible users will be able to review all relevant existing incidents when first documenting a system, and be alerted on new ones as we ingest them. The incident database will not cover every potential problem, since it focuses on incidents that got media attention or came from research papers. Public incident monitoring can assist with regulatory monitoring obligations, such the post market monitoring requirements of the EU AI Act. Many legal liability frameworks require organizations to mitigate ‘foreseeable risks’; harms that have happened in the past and are known publicly are the very definition of foreseeable risk.