When executives say "there's a human in the loop," they believe it is a control. In most deployments, it is not. A reviewer approving hundreds of agent actions per day stops being a safeguard within weeks. Approval becomes reflex, and the human becomes a liability shield rather than a check.
This blog post lays out four principles for building exactly that and for answering the question boards will ask after the first incident: who is accountable?
The fundamental shift is that risk moves from a wrong answer to a wrong action.
A chatbot that states an incorrect fact creates a correction problem.
An agent that takes a wrong action creates a remediation problem, whether it is money moved, records altered, customers contacted, or commitments made. The blast radius is categorically different, and traditional controls were not designed for it.
Four structural features set autonomous agents apart from earlier automation. Each one amplifies the risk of any single mistake:
Agents choose their own steps. You cannot review every execution path in advance the way you can with a scripted workflow.
An error can repeat thousands of times before a human notices, as machine speed outpaces human review by orders of magnitude.
One agent's output becomes another's input. That is how small errors compound across systems, creating cascading failures that are difficult to unwind.
Agents act with your business's credentials, blurring who actually "did" the action in any legal or regulatory sense.
Traditional controls, such as code review, change management, and segregation of duties, were built for deterministic software and for human employees. However, autonomous agents sit between these two categories. They need controls designed specifically for that in-between space.
Set each agent's autonomy level by the cost of being wrong. A four-tier model provides the necessary structure for most enterprises — from pure advisory roles to fully autonomous execution. The key discipline is assigning tiers rigorously at deployment and revisiting them continuously as evidence accumulates. Promotion must be earned; demotion must be automatic.
What the agent may do: Recommend only; takes no action.
Example: Draft a contract redline.
Human role: Human decides and acts on every output.
What the agent may do: Prepare an action and wait for approval.
Example: Propose a customer refund.
Human role: Approves each action before execution.
What the agent may do: Execute within defined limits.
Example: Refunds under a fixed dollar amount.
Human role: Samples work and handles exceptions.
What the agent may do: Execute broadly within policy.
Example: Reorder low-value inventory.
Human role: Monitors metrics and responds to alerts.
Can the action be reversed?
How much money is exposed?
Is it subject to regulation?
How visible is it to customers?
An agent should start at the lowest tier that makes it useful and earns promotion through demonstrated evidence including error rates, human override rates, and incident history. One serious incident should trigger automatic demotion, pending formal review.
An autonomous agent is a non-human employee with access to your systems. It should be governed exactly like one. The instinct to reuse service accounts or developer credentials is understandable. They are faster to deploy. But that is also the governance failure that regulators and auditors will identify first. Identity hygiene for agents is an accountability prerequisite.
Never reuse a shared service account or a person's credentials. Each agent receives its own identity, making audit trails unambiguous and revocation clean and immediate.
Grant access only to the systems, data, and actions the job requires. Limit the access by time windows and transaction size to avoid scope creep in agent permissions.
Instructions in prompts can be misread, ignored, or manipulated. However, hard limits enforced in payment systems, approval workflows, and systems of record cannot be. So, build constraints into the infrastructure, not using the language model.
Agent-to-agent and agent-to-tool connections expand an enterprise's attack surface. So treat them with the same scrutiny as any third-party integration.
If you cannot reconstruct what an agent did and why, you cannot defend it. That is why every consequential action an agent takes should generate a decision record that captures the full chain of reasoning.
Store decision logs where the agent cannot alter them. Retain them on the same schedule as comparable business records. It is also important to watch for drift in real time and to detect any rising override rates, spikes in action volume, new tool usage, and unusual spend patterns.
You'll need to build a kill switch that can pause one agent, or an entire class of agents, within minutes.
Every agent needs a named human owner accountable for its outcomes, similar to the way a manager is accountable for a team's performance. Technology is responsible for how the agent runs. The business is responsible for what it does. These are not the same responsibility, and conflating them is how accountability disappears entirely after an incident.
Organizations should record the owner's name in an agent registry alongside the agent's tier, permissions, and stated purpose. The owner should be a business leader, not an engineer. Escalation triggers should be explicitly defined before deployment, not after the first exception:
When the agent's certainty falls below a defined level, it stops and hands off to a human with full context rather than improvising.
Any action that exceeds a financial or operational threshold routes to human approval regardless of tier, without exception.
Novel scenarios, conflicting policies, or anything touching legal or regulated matters trigger an automatic stop and escalation with full decision context attached.
Without a clear accountability map at the executive level, responsibility defaults to whoever is physically closest to the incident , i.e., typically the engineer who deployed the agent, or the team lead who approved the workflow. That is not accountability; it is rather scapegoating. Organizations should define roles explicitly and record them before their first autonomous agent goes to production.
Sets the organization's risk appetite and defines the criteria for each autonomy tier. Owns the tier model itself and the process for promotion and demotion.
Governs agent identity, access controls, and security architecture. Responsible for the agent-to-system and agent-to-agent connection review process.
Interprets applicable regulatory frameworks for each agent use case and records the legal classification in the agent registry. First call on anything touching compliance.
Sets and enforces financial limits by tier. Any agent that can move money operates within thresholds the CFO has explicitly approved and systems enforce.
Accountable for business outcomes across the agent portfolio. Receives the quarterly board view and has authority to pause deployment classes when warranted.
Business leadersers should assume they will be asked to explain a specific agent decision to someone outside the company, whether it is a regulator, an auditor, or a plaintiff's attorney. They must design their governance posture with that assumption as a baseline, not an edge case. Regulatory frameworks such as the EU AI Act impose oversight, documentation, and transparency obligations for high-risk AI applications. Sector regulators in finance, healthcare, and insurance have long required explainability in automated decisions affecting customers or markets.
Ask your General Counsel which frameworks apply to each agent and record the answer in the registry. Prepare a narrative explanation of the decision logic for any high-risk use case before the regulator asks.
Auditors need the evidence trail from Principle 3 and proof that controls operate as designed. Your decision logs are your primary evidence. Test them before the audit window opens.
Directors need a quarterly one-page view: how many agents are live, at which tiers, what they did, incident and override rates, and any tier changes. If directors cannot see it, they cannot oversee it. Oversight requires visibility.
Companies that treat governance as a brake stay stuck in pilots. No executive will approve expanded autonomy for systems they cannot see, pause, or explain. The debate about risk repeats for every new deployment. Every approval requires a fresh argument. Governance built first changes the calculus entirely: a new agent slots into an existing tier model, identity framework, and reporting line. The approval becomes a routine operational decision, not a novel risk conversation.
The companies building governance infrastructure now are not doing it because regulators require it yet. They are doing it because it is the only way to say yes faster and at greater scale. Governance is not the price of deploying agents. It is the mechanism that makes deployment commercially viable at enterprise scale.
Your agents will make a mistake. The governance question is whether you will know who is accountable, what happened, and how fast you can respond when it does. These five steps move you from exposure to readiness. Each one is executable within your current organization. None requires waiting for a new regulatory mandate or a vendor solution.
Include shadow deployments built by individual teams without central oversight. You cannot govern what you cannot see. This inventory is the foundation of every other step.
For each agent in your inventory, record a business owner and assign a starting tier based on the four diagnostic questions. Provisional is acceptable, but unknown is not.
Audit each agent's constraints. If a financial or access limit exists only as an instruction in a prompt, it is not a control. Move it into the system of record before the next deployment cycle.
Run a kill-switch drill for your highest-tier agents. Document the time from decision to full pause. If you cannot achieve it in minutes, the control does not exist at the speed agents operate.
Start the board's visibility practice now, before an incident forces it. The first report may be imperfect, but it establishes the expectation of oversight and surfaces the gaps you need to close.
When the first incident occurs, you will know who is accountable, what happened, and how fast you can respond. That is the only board answer that matters.
Governing Autonomous Agents at Scale