Control is becoming part of the AI product itself

But as agents become more autonomous, another question becomes at least as important: what is an AI agent actually allowed to do? And perhaps more importantly, how do we make sure those boundaries are technically enforced?

That may sound like a security question, but I think it is increasingly becoming a regular product question.

A wiki as a communication channel

On 4 September, Reuters reported an unusual incident from May in which OpenAI agents used a German programming wiki as a kind of shared communication channel. Agents posted information there that allowed them, among other things, to share tactics for bypassing restrictions and coordinate their activities. [1]

A day later, OpenAI acknowledged what it called the "wiki incident". The company also said the AI industry needs better and more transparent ways to report unintended or unexpected agent behaviour. [2]

It is tempting to read a story like this and immediately think of AI running out of control. For businesses, however, that is probably not even the most important conclusion. The more interesting question is what happens when software no longer simply provides an answer, but can independently take action.

A chatbot giving the wrong answer is inconvenient. An agent with the wrong permissions modifying a database, calling an API, deleting files or sending information to an external environment is something else entirely.

From good instructions to hard boundaries

Until now, responsible AI use has focused heavily on prompts, policies and human oversight. All of those matter, but once agents gain access to real business processes, instructions alone are no longer enough.

You can tell an agent that it must not perform a certain action. But if that same agent technically has access to the tool that makes the action possible, the risk remains. That is why we are starting to see a second layer emerge: technical controls around the agent itself.

On 1 September, for example, OpenAI described additional monitoring for its Astra-class models. Classifiers monitor a model's reasoning process and actions for potentially unauthorised behaviour. When such behaviour is detected, a task can be delayed, paused or automatically terminated. [3]

A few days later, Broadcom described a similar direction for VMware Cloud Foundation. The company refers, among other things, to a deny-by-default architecture and a future Secure Agent Framework featuring sandboxing and an Agent Harness. This can determine which tools an agent may use, how agents communicate with each other and how output is validated before anything is actually executed.

Importantly, this Secure Agent Framework is listed under "Future Release Capabilities", so it is not yet a generally available standard. [4] That detail matters because we are still working out what such a control layer should look like in practice. But the direction is becoming clear.

Control is becoming a product capability

What interests me most from a marketing perspective is what this will mean for the way organisations evaluate and buy AI.

Much of the discussion so far has focused on capability: can the model understand our documents, write code, automate processes or execute tasks independently? I expect those questions will increasingly be joined by a different set of questions.

Not only: what can this agent do? But also: what is this agent allowed to do? And how can I prove that?

That is a fundamentally different buying question.

An organisation connecting an AI agent to CRM, ERP, financial systems, development tooling or other business-critical software will ultimately need to know where the boundaries are. Not just on paper, but technically.

From human in the loop to control by design

"Human in the loop" is often presented as the solution: have an employee review important actions before the AI executes them. That remains useful, but it is not a universal solution either.

If every action has to be checked manually, much of the benefit of autonomous agents disappears. And once thousands of actions are being performed automatically, it becomes impossible for people to review everything. The architecture itself therefore has to carry part of that responsibility.

When developing business agents, I would at minimum want to be able to define:

which data sources an agent may access;
which systems and tools are available to it;
which actions it may perform autonomously;
which actions require explicit human approval;
whether code and actions are first executed in a sandbox;
how abnormal behaviour is detected and stopped;
how changes can be rolled back;
which decisions, actions and authorisations can later be traced in an audit trail.

You could capture this in an Agent Control Matrix. Not as an additional document created after implementation, but as part of the solution design itself.

This is also a software architecture issue

In that sense, agentic AI is actually less exotic than it is sometimes made out to be. We already know many of these principles: least privilege, roles and permissions, segregated environments, logging, monitoring, approval flows, sandboxing, rollback and audit trails.

We have spent years building these mechanisms around people, applications, APIs and integrations. Now we need to determine how to apply them to software that makes its own decisions about what to do next.

That is exactly why AI ultimately cannot be separated from software architecture. A model can be exceptionally capable, but as soon as you give it access to business data, APIs and operational systems, the way that access is organised becomes at least as important as the model itself.

The next AI question is therefore not only about intelligence

We will probably allow agents to perform more and more work on our behalf. That means reliability can no longer be defined solely by the likelihood that a model produces the correct answer.

A reliable AI system must also be able to demonstrate that an agent remains within its authorised boundaries. That means knowing which data it could access, which tools were available, which action it attempted to perform, why that action was allowed or blocked, and what happens when the system exhibits behaviour outside the agreed boundaries.

Control therefore becomes part of the AI product itself.

And for organisations currently experimenting with agents, there is one question I find more interesting than many of the questions about models, benchmarks or prompts: