AI agents and the King Midas problem

Usually, when someone goes above and beyond our expectations, it’s a sign of outstanding service or impressive personal initiative. When the current generation of AI agents does that, however, the results are quickly becoming much more dramatic.

 

One recent incident in Australia is illustrative of the risks. A member of the public used an AI agent to try and book a slot in a Pilates class, which is a great idea. The session turned out to be full, but instead of informing the user, the agent exploited weaknesses in the studio’s booking system API, cancelled another customer’s reservation, and moved its user into the class.

 

An equally remarkable story from OpenAI involved a research model that uploaded files to a public website in an attempt to create evidence it could cite to help its user. Again, the concern was not simply that the system made a mistake, but that it took an unauthorised action in pursuit of its goal.

 

Many of us will have read reports of GenAI systems inventing case law examples that never existed, but to go further and create published material that can then be presented as evidence is an alarming escalation.

 

As AP reported at the time, “During training of an AI model called 5.6-Sol, the model instructed itself to invent missing data, and an agent wrote a message to remind itself to hide mismatched information.”

 

From a certain perspective, some of these are interesting and largely harmless technology errors. But in reality, they illustrate a much more concerning problem about the limits of machine autonomy and organisational governance. They also have a strong hint of King Midas about them in that the agents granted wishes, so to speak, but in a way that created consequences nobody had wanted. In that famous Greek legend, Midas is granted the power to turn everything he touches to gold; a decision he soon regrets when his food, drink, and even his daughter are transformed.

 

Beyond cyber-security

These are also far from isolated cases. There have now been public cyber-security disclosures from three of the major AI brands (OpenAI, Anthropic and Google) involving agents that compromised systems beyond the environments where they were meant to operate.

 

Of course, agent-related cyber-security breaches like these are of enormous concern, but the potential runs much deeper. In theory, any agent with permission to interact with systems can pursue a legitimate task through an unintended route.

 

That applies just as readily to everyday business systems such as email, where an agent might be authorised to read, draft or even send messages on a user’s behalf.

 

These problems raise all kinds of governance issues. Powerful autonomous technologies are one thing, but when they identify a viable technical route without understanding the consequences for the people affected, they need effective guardrails to prevent them from taking that route in the first place.

 

Agents are often connected to systems through a user account or another form of machine credential. A set of credentials that provides more access than an agent’s task requires can give it far more reach than intended. This creates a form of delegated authority in which an agent can act across connected systems on behalf of a legitimate user.

 

Instead, organisations should establish which actions an agent genuinely needs to take before granting access. For example, read-only access may be sufficient for some tasks, while more consequential actions should require specific approval.

 

Access should also be time-limited where possible, rather than remain available after the relevant task has ended. Whatever process is involved, records of agent activity should make it possible to establish what happened and under whose authority the agent was acting.

 

Solving the problem requires organisations to recognise what contemporary AI agents are capable of. It’s extremely easy to implement them without a clear understanding of how they may behave once connected to real-world systems.

 

For instance, agents can investigate systems continuously and far more quickly than a person working through the same options. In this context, a poorly protected API or weak permission model may once have escaped attention because exploiting it required time and specialist effort.

 

As is increasingly clear, the more capable agents become, the less effort is required to uncover those weaknesses. This means organisations should assume that a weakness which is technically exploitable is increasingly likely to be found.

 

Give agents only the access they need

It might sound like an obvious step to take, but at present, too many organisations simply don’t implement controls that reflect the precise task an agent has been asked to perform. A 2025 survey of IT professionals, for example, found that 80% had seen AI agents act beyond their intended scope, while only 54% said they had full awareness of the data their agents could access.

 

This ‘least privilege’ principle should apply as rigorously to agents as it does to employees.

 

Strong authentication establishes who has authorised an agent to act on their behalf. In practical terms, this means that APIs need to enforce permissions at the system level, rather than rely on an agent to decide which actions are appropriate.

 

Organisations also need visibility of the actions an agent has taken, particularly where it can modify records or communicate externally. Any additional authority should be explicitly granted by the user; a process which itself should fall under predefined governance rules. This is not something to decide on the fly.

 

Email provides a good illustration. As services allow AI agents to access inboxes through APIs and standards such as MCP, agents may be able to read, draft and send messages on a user’s behalf. The same least-privilege principle should therefore apply. An agent asked to summarise correspondence does not need permission to send it, while sending should require explicit authorisation and a clear audit trail.

 

In effect, AI agents should be treated as a new category of insider threat. Even though threat actors are already weaponising agents, they do not need a malicious purpose to create risk if acting with legitimate authority.

 

---

 

Neil Jenkins is Chief Product Officer at Fastmail

 

Main image courtesy of iStockPhoto.com and gremlin