That is a sensible wish, and usually the right place to start. The process you have is one that provably works for you. It is also the moment the architecture of that existing system starts to matter a great deal.
The vocabulary doesn't help. RAG, agents, MCP, orchestration: every month brings new terms, and every vendor has its own list of patterns. Underneath, one question decides almost everything. How much does the model get to decide, and how much stays in your code?
Workflow before agent
On this point, the people who publish most about building with language models agree more than on anything else. Anthropic separates workflows, where your code determines the path and the model does the steps, from agents, where the model chooses the path itself. Microsoft's architecture guidance says to use the lowest level of complexity that reliably does the job, and calls a single agent with tools often the right default for enterprise. Thoughtworks' Technology Radar has moved in the same direction.
For existing software, that is good news. Your system already knows its process. It knows which step follows which, who may do what, and when something is finished. That knowledge is exactly what an agent would otherwise have to guess.
So I think of it as a ladder. Each rung hands a little more control to the model, and asks more of your system in return.
Rung 1: the model answers, your code decides
The model gets a question and returns an answer. Your code decides what happens with it. A summary in a side panel. A suggested category in a dropdown. A draft reply the user can edit.
Two things make this work in production. If code is going to use the answer, ask for structured output: a fixed format your code can validate, rather than a paragraph of prose to parse. And decide whether anyone needs to wait. A summary prepared overnight from a queue asks almost nothing of your application. A button the user clicks needs a screen that copes with an answer that takes seconds, and a plan for when the provider is slow or down.
Either way, you need to know exactly which fields go into the prompt, because they leave your system.
Rung 2: the model knows your data
Now the model answers from your procedures, contracts or tickets instead of from general knowledge. Find the relevant pieces, put them in front of the model, let it answer from those.
It is the most requested step and the most underestimated one. The index has to stay in sync with your data. Retrieval has to respect access rights at the moment of the query: if a user can't open a document in the application, the model must not quote it to them. In many older systems, those rights live in the screens rather than in the data. That is where most of the work hides. The rest hides in the documents themselves. Three versions of the same procedure, and the model has no idea which one is current.
For figures, this is often the wrong tool. A model writing its own SQL against your database tends to produce confident, wrong business numbers. Let it ask a layer that already knows what "revenue" or "open order" means in your organisation.
Rung 3: the model follows your process
Here the model does several steps, but your code sets the order. Classify the incoming request, send it to the right handling, let one step draft and another check. The path is predictable, testable and explainable, because you wrote it.
Many real use cases end on this rung, and that is fine.
Rung 4: the model acts
Now the model can call functions in your system: look up an order, check a stock level, create a ticket. The model proposes the call. Your code decides whether to execute it.
That requires an API, which not every existing system has. It requires a permission model you can reason about, because the model must act with the rights of the user it works for, never more. Start with reading. Writing comes later, with confirmation. And any text the model reads, an email or an uploaded document, can contain instructions. The model may follow them.
A word on MCP, the protocol everyone is adopting for this. It is useful, but wrapping your existing API one-to-one and handing it to a model is not a design. Thoughtworks now explicitly cautions against MCP by default. An interface built for a model should be smaller and safer than the one built for your front end.
Rung 5: the model decides the steps
Only now does the model plan its own route: search, read, call a tool, decide, call another tool. Use it when the next step genuinely can't be known in advance. It needs everything from the rungs below, plus a trace of every step, limits on time and cost, and a clear point where it stops and hands over to a person.
In a system with clean APIs, events and explicit permissions, that is achievable. In a system without them, the agent will find every weakness the architecture has, and quickly.
One door for all model traffic
Whichever rung you are on, put one layer between your applications and the model providers. It handles authentication, logging, cost limits, and fallback when a provider fails. Later it can send simple requests to a small, cheap model and hard ones to a larger one. It is unglamorous work. It is also the place where you will want to answer every future question about cost, privacy and quality.
What the ladder tells you
Look at what the higher rungs require: APIs, events, an authorisation model that lives in the data rather than in the screens. None of that is specific to AI. It is what any system needs to stay changeable.
Most systems we maintain were designed before anyone had heard of language models. That is rarely the real problem. Whether a system can take AI depends on decisions made years ago. Is there an API? Does the system publish what happens? Is it clear who may see what? If the answer is no, your first AI project is in fact an architecture project. The same seams make the system easier to change for every other reason as well.
Let the model do the language. Let your code keep the control.
Where I would start
Pick one task with a clear owner and a measurable "before". Start on the lowest rung that solves it. Climb only when the current rung runs well in production, not in a demo.
Which rung fits your system can usually be worked out in an hour, with one concrete use case and someone who knows the architecture. Bring both. We'll bring the coffee.
