“AI governance” might be the most overloaded term in enterprise software right now. It can mean low-level agent action interception and prompt filtering, it can mean mid-level model registries, evaluation harnesses, and MLOps oversight, and it can mean high-level IT intake workflows, vendor risk management, and regulatory compliance documentation. Different vendors use the same words to mean fundamentally different things, and the buyer is left trying to compare products that are barely in the same category.
We had to work through this ourselves before we could decide what kind of platform we were building. We wrote up our taxonomy of AI governance platform types separately, and it’s worth reading if you want the full picture of how the market actually segments.
Out of all that conceptual sorting, one question keeps coming up as the most contested. Should the platform that authors governance policy also enforce it at runtime? It sounds like a technical question about feature scope, but it’s actually a question about where governance lives, who owns the decisions, what compliance actually requires, and how the AI stack will mature over the next several years. The answer shapes the category.
We’ve come to a clear view. The governance platform’s job is to determine what enforcement needs to do, where, and why. The job of actually enforcing, whether that’s intercepting prompts, blocking outputs, restricting agent tool calls, or gating deployments, belongs somewhere else. This post explains why.
Governance by design: The use case is the unit of governance
Start with what we mean by governance by design. It’s the idea that the substantive decisions about how an AI system should behave, what risks need mitigating, and what evidence has to exist need to happen before the system ships, and they need to be specific to what the system is actually for. Say it, do it, prove it. Define what’s required, configure it where it gets enforced, collect evidence it’s working.
The unit of all this is the use case, not the model.
That distinction does more work than it looks like. Consider the configurations real AI systems actually take. One model can power a hundred different use cases, where the same Claude or GPT-4 deployment might answer customer FAQs, screen résumés, draft clinical notes, and summarize financial filings. Each of those has a different risk profile, different regulatory exposure, and different definitions of what “working correctly” means, so governing the model uniformly is either too loose for hiring or too tight for FAQs.
One system can also be powered by a hundred models, with routing logic that picks among them, fallback chains, ensembles, A/B tested swaps, and models that change every quarter as new ones come out. Governing each model individually misses the system-level behavior that actually matters. And most modern AI systems are neither one model nor a hundred discrete ones. They’re combinations with retrieval layers, routers, sub-agents, and broad tool access, where the behavior that matters is emergent across components rather than attributable to any single piece.
Bias, drift, and “working correctly” don’t have stable meanings at the model level. They have meaning relative to a use case definition: what is this AI for, who does it affect, what does success look like, what’s the failure mode that matters. A model that’s biased in a hiring screen may be fine in a customer service bot, and drift only matters relative to a use case’s definition of acceptable behavior. Trying to govern this purely technically, at the model artifact, at the prompt level, at the tool call level, fragments into a thousand disconnected controls that don’t add up to anything a regulator, a board, or a risk officer would call governance.
Agents intensify this. The value of an agent comes from flexible judgment across many situations, and trying to govern an agent the way we govern a fixed algorithm, by constraining every action, kills the thing being governed. Governing agents looks more like how organizations govern humans: define the role, set scope, configure the tools they can use, escalate on specific conditions, monitor outcomes, review after the fact. Hard runtime limits exist for specific high-stakes actions, but they’re exceptions inside a much larger structure of role-based oversight rather than the primary mechanism.
If the use case is the unit of governance, then the governance platform’s job is to determine what enforcement needs to do for each use case. Not to be the enforcement.
Why the governance platform shouldn’t own runtime enforcement
Three reasons, each from a different angle:
AI runs in too many places for one enforcement plane
The current AI deployment picture doesn’t have one pattern. Models run locally on developer laptops and edge devices, they run in hyperscaler clouds behind API gateways, and they run embedded inside SaaS products you don’t operate, with no way to interpose anything between the model and its users. They run inside agent frameworks that orchestrate tool calls across systems, and they run as fine-tuned variants inside vertical applications. The same enterprise often has all of these patterns at once.
A single runtime enforcement plane can’t cover this, and for some deployment patterns it’s outright impossible. Embedded AI inside a SaaS product has no place to insert a third-party guardrail, and even when routing is technically possible, doing it cleanly often requires proxy gymnastics, packet rewriting, or middleware that introduces latency and operational fragility. The cost of forcing everything through one enforcement point grows faster than the coverage it provides.
The right architecture enforces close to the model. That means many enforcement points across the stack, run by different teams, with different latency budgets and different operational characteristics. The governance layer’s job is to orchestrate them, to know which one needs which configuration for which use case, rather than be one of them and lose coverage on the rest.
The runtime layer is consolidating into the model and the platform
Jailbreak resistance, prompt injection defense, content classification, output filtering. Most of what was a third-party guardrail product two years ago is moving into foundation models and inference APIs. Anthropic ships these capabilities directly in Claude, Bedrock has Guardrails, Azure has Content Safety, and every major model release closes more of the gap. The third-party runtime guardrail layer is getting squeezed between native model safety and platform-level controls.
Jailbreak protection in particular has a clear sunset on a meaningful chunk of its product surface, and building a durable category around runtime interception means building on ground that keeps shifting underneath. The durable layer above it is governance: deciding which native controls to turn on, where, with what thresholds, for which use case.
The same pattern holds for agent tool restrictions. Every major agent framework already has mechanisms for tool allow-lists, scope restrictions, and permission gating, so the hard part isn’t building the technical capability to restrict an agent’s tool calls. It’s figuring out what should be allowed for a given use case, what counts as an appropriate escalation, and what evidence of correct behavior needs to be collected. That’s governance work, and it has to happen before any tool restriction gets configured.
Regulation rewards documented due care, not interception
This is the point that gets the most resistance and is the most important. The conventional argument against governance-only platforms is that policy alone is theater. That’s not quite right, and it matters why.
Major AI regulation doesn’t measure compliance by interception rate. EU AI Act Article 9 requires a documented risk management system, Article 14 requires demonstrable human oversight design, and Article 27 requires a fundamental rights impact assessment. ISO 42001 requires a management system with documented decisions, monitoring, and review, and NIST AI RMF organizes around govern, map, measure, manage as a process discipline. The legal standard is intent, design, and best efforts, not whether any particular output got blocked.
Concretely, an output that violates policy after a successful jailbreak isn’t automatically a regulatory finding if the organization documented the risk, implemented reasonable mitigations, monitored for the failure, and responded when it occurred. The opposite case is also true. Perfect runtime blocking with no risk register, no documented decisions, and no impact assessment still fails an Article 9 audit, because the system was never properly governed in the first place.
This is showing up in how organizations are structuring AI oversight. Stanford’s 2026 AI Index documented AI-specific governance roles growing 17 percent in 2025, with the share of organizations operating without any responsible AI policy falling from 24 percent to 11 percent year over year. More telling than the growth rate is where the ownership is landing: legal, risk, and compliance functions, not the ML engineering teams building the models. Many regulated organizations are explicitly treating AI governance as a second line of defense function, the same model used for credit risk, operational risk, and financial compliance. Second line functions in every other domain don’t run the controls. They define what controls are required, verify they’re in place, and own the evidence that they’re working. The market is already moving toward the architecture this post is arguing for, because that’s the architecture regulators expect.
The “policy alone is theater” critique isn’t entirely wrong, though. Policy without configured mitigations and monitoring evidence doesn’t pass regulatory review either, and a serious governance platform should only sign off on a regulatory requirement when there’s actual evidence the relevant mitigations are in place. The disagreement isn’t whether mitigations exist or get configured. It’s whether the governance platform has to be the system enforcing them. Under any major AI regulation, it doesn’t. The mitigation can run in Bedrock, in the agent framework, in the application layer, or in human review, as long as it’s documented, configured for the right reasons, and producing evidence it’s working.
There’s a flip side that runtime-first vendors don’t always address. Runtime enforcement only meaningfully maps to one part of Article 9, which is risk mitigations, and it does nothing for risk identification, impact assessment, human oversight design, post-market monitoring obligations, transparency requirements, or conformity assessment. An organization that bought a runtime-only product and treated it as their AI compliance solution would still be exposed across most of the regulation.
The cybersecurity, IT, and privacy analogy
This is not a new architecture. Every other governance domain settled it decades ago.
In cybersecurity, ServiceNow GRC and Archer author control libraries, track evidence, and run audit campaigns, but they don’t run firewalls or EDR. Splunk, CrowdStrike, and Zscaler handle enforcement, and the GRC platform documents that the controls exist, gets them tested, and produces audit evidence. In IT, ServiceNow ITSM doesn’t push configurations to switches and routers; it tracks change requests, while the network appliances handle the actual routing. In privacy, OneTrust and TrustArc maintain data processing inventories, consent records, and DPIA workflows, and they don’t sit inline on data flows. DLP tools, encryption gateways, and data masking services do the enforcement, and the governance platform’s job is to know what should be in place and prove it is.
All three converged on the same split for the same reasons that apply to AI: heterogeneous enforcement points across many systems, specialized operational teams, different latency budgets, and the need for one unified audit story across many enforcement layers. There’s no reason AI governance should reinvent the architecture that every adjacent domain already settled on.
What about smaller organizations?
So far, AI governance has mostly been the focus of larger organizations who have a lot of existing liabilities and governance concerns. This architecture assumes a sophisticated buyer with the operational maturity to run separate governance and enforcement layers. Large enterprises operate comfortably with many tools, but smaller organizations often prefer bundling, and there’s a real argument that a small team should be able to buy one product that does governance and enforcement together.
The reframe is that smaller organizations using major AI platforms already have substantial enforcement capability built in. Claude exposes extensive content controls, agent scope restrictions, tool permissions, and audit logging directly through its APIs, and Bedrock and Foundry expose more. A small organization doesn’t actually lack enforcement options. It lacks the time and expertise to figure out which ones to turn on, with what configurations, for which use case, and how to demonstrate to a regulator or a customer that the controls are working.
That’s a governance problem, not a runtime problem. The right answer for a small organization is a governance platform that helps configure the controls already available inside the platforms they’re using, rather than a separate enforcement product that competes with the native controls or routes around them. Bundling doesn’t have to mean owning the enforcement layer. It can mean making the enforcement layer that already exists actually usable.
The vision: determine, configure, collect, prove
Our view of how the governance platform should operate across the AI lifecycle:
Determine. Use case context, regulatory classification, risk profile, and prior incident patterns feed into a rules engine that determines what runtime posture the use case requires. Which controls have to be on, what thresholds, what monitoring, what escalation criteria, what review cadence. This is the governance decision, made before deployment and revisited as the use case evolves.
Configure. The governance platform pushes those configurations into the systems that actually enforce them. Bedrock Guardrails for prompt and output controls, the agent framework for tool restrictions and human-in-the-loop, the inference layer for citation or output formatting requirements, and the MLOps platform for monitoring thresholds. None of this enforcement runs in the governance platform itself.
Collect. Monitoring integrations pull signals back: which controls fired, which outputs were flagged, which human reviews happened, what overrides occurred. Some signals come from runtime systems, and some have to be human-generated, because deciding whether a system is actually working as intended often requires judgment that no log line can produce. Looking at model inputs and outputs alone isn’t enough.
Prove. The governance platform holds the audit trail. What was required, why, where it was configured, what evidence came back, what’s been reviewed, and what changed when something went wrong.
That’s where we believe the category is going. Not because it’s the only possible architecture, but because it matches how every other governance domain has matured, what regulators actually require, and where the AI stack is heading as native controls get stronger.
Closing
The space is nascent. No runtime enforcement approach today is guaranteed to work, deployment patterns will keep multiplying, foundation models will keep absorbing more of the safety stack, and the right architecture in 2026 may not be the right architecture in 2029.
But the principle holds. Governance by design means deciding what needs to be enforced, where, and why, before any of it gets enforced. The governance platform’s job is to author that decision, push it into the systems that can act on it, and prove it worked. Bundling governance and runtime enforcement into one product collapses two layers that need to stay separate, and gets the order wrong.
Policy doesn’t have to do the enforcing. It has to come first.