AI agents - a potential governance nightmare
AI agents are most likely already operating in your organisation. The question isn't if you need governance, but whether you'll implement governance before or after an incident forces your hand.
Over the last couple of weeks I've had the conversation about AI agent governance with a handful of clients.
The big questions are how to ensure that AI agents only do what you want them to do, that they do it with a satisfactory level of precision and predictability, that there is some kind of audit trail, and that they do not cause information security weaknesses. In addition, when the number of agents becomes significant, you also want to avoid that they weave into each other and cause hidden collateral damage.
If you are searching for the Holy Governance Grail, I'm sorry to disappoint; you are on a mission impossible. Not because there are some evil villains inside the AI agents, but because of the inherent characteristics of the transformer technology on which the LLMs, and hence the agents, are built. I'm not going to delve into explanations of the strengths and weaknesses of the transformer technology and language models in this piece, though. This piece is about why governance is particularly important when deploying AI agents and what that governance should entail.
But first, what is an AI agent?
What’s the difference between LLMs, chatbots, and AI agents? These terms are often used interchangeably, which adds to the boardroom confusion. They are indeed different things, albeit connected.
An LLM, or a Large Language Model, is the engine. The term comprises both the mathematical model itself and the surrounding architecture, such as data engineering, guardrails, and RAG (Retrieval Augmented Generation).
A chatbot is an LLM wrapped in a dialogue interface. This enables you as a user to communicate directly with the machine through writing or speaking human language. Chatbots are typically able to connect to tools through MCP (Model Context Protocol), which enables them to solve complex tasks beyond merely text processing.
An AI agent is an independently executed piece of software wrapped around an LLM and connected to the surrounding environment through an MCP (Model Context Protocol). This makes an AI agent into an autonomous software program with the ability to create its own instructions, plan tasks and sequences, chunk tasks into smaller workloads, utilise external tools, and make decisions in ambiguous contexts.
Recent developments
AI agents became mainstream hype in late 2024. Model Context Protocol (MCP) from Anthropic accelerated adoption by standardising how agents connect to external tools, enabling large-scale autonomous actions. In April 2025, Google launched the Agent2Agent protocol, enabling agent-to-agent communication. Within months, agent-building tools became commodity products. This means that anyone in your organisation can now deploy AI agents that make decisions and take actions across your infrastructure. The barrier to entry disappeared before governance frameworks could catch up
The key governance challenges
For the purpose of simplification, the governance issues surrounding AI agents may be divided into three main categories:
1) Decision authority and control
The agent needs boundaries. Who defines what actions it can take? What requires human approval? When agents interact, errors compound—one agent's output becomes another's flawed input. And lastly, without explicit limits, you're delegating authority you don't understand to systems you can't fully predict.
2) Accountability and liability
When an agent discriminates, leaks data, or costs money, who's responsible? The developer who wrote the code? The business unit that deployed it? The executive who approved the budget? Your organisational hierarchy, contracts, insurances, and legal frameworks weren't written for autonomous systems making cross-jurisdictional decisions based on probabilistic models.
3) Security, privacy and trust
Agents create attack surfaces that traditional security can't address. Prompt injection lets attackers manipulate agents through carefully crafted and surprisingly simple instructions. Agents cross data boundaries in ways GDPR wasn't designed to handle. And when agents fail, you need audit trails that probably don't exist.
Governance requirements and actions
European regulators are behind when it comes to AI agents. GDPR and the AI Act predate autonomous agents. ISO standards (42001 for AI, 27701 for privacy) haven't been updated. This creates a problem: you are deploying technology that operates in a legal vacuum. E.g. when an agent processes customer data autonomously, it may violate regulatory principles such as data minimisation, purpose limitation, the right to reject profiling, and the right to explanation regardless of your intentions. You cannot wait for regulators to catch up.
Although the governance challenges may feel overwhelming, there are concrete actions that may be implemented:
Establish unambiguous, machine-enforceable limits on agent autonomy before deployment.
Before deploying any agent, define which systems it can access. Make sure this definition is technically enforced, not just documented. Set financial thresholds that trigger mandatory human review. Implement circuit breakers that automatically halt agents at defined boundaries. Test what happens when agents hit those limits. You can safely assume that they will.
Establish clear accountability chains
Map liability explicitly, such as who's responsible when agents cause harm (discrimination, financial loss, data breaches, operational failures). Document decision authority in contracts between developers, deployers, and users, and avoid situations where developers say they merely coded algorithms, data curators argue no knowledge of usage, and executives insist oversight lay elsewhere.
Perform information security risk assessment, implement mitigating actions in advance, and set up continuous monitoring.
Any LLM-based AI agent with access to external sources will be vulnerable to prompt injections. This vulnerability is inherent to how language models process instructions and cannot be fully eliminated with current architectures. Don't skip fundamental security practices when deploying AI agents. Establish an AI policy, with particular focus on AI agents. Create incident response procedures specific to agent failures, including rapid containment and rollback capabilities. Build resilience, because breaches will happen.
What this means for your board
Best practices for AI agent governance don't exist yet. Market adoption timelines suggest it might take 5+ years to develop best practices. With AI agents, we are still only at 5+ months of adoption. This means that you cannot buy some governance framework for AI agents and be confident that the situation is under control. On the contrary, if you take the path of deploying AI agents in your organisations, you also need to take the demanding parallel path of assessing risks and regulatory compliance throughout your operations.
AI agent governance is not an "IT problem". It's a board-level issue. Agents represent delegated authority operating across your organisation. You need to understand what authority you are delegating, to which actions and decisions, and with which safeguards.
The organisations that will navigate this successfully are those making governance decisions now, not those waiting for frameworks to emerge.
Navigating AI agents requires more than technology decisions.
If your board or leadership team is grappling with how to govern autonomous AI responsibly, get in touch with me to explore what this means for your organisation.
About the Author
Elin Hauge is a business and data strategist, pragmatic futurist, and eternal rebel. With a unique background spanning physics, mathematics, business, and law, she brings fresh and thought-provoking insights into artificial intelligence and other data-driven technologies. Her work connects today’s realities with tomorrow’s possibilities, focusing on demystifying AI, harnessing data power, managing algorithmic risk, and guiding leaders through digital transformation.