Six months later the first team is tuning prompts every week and nobody dares to upgrade the model. The second team swapped models twice without anyone noticing.
The difference is not the model. It is seven decisions about where the model sits, what it is allowed to say, and what happens when it is wrong. We make these decisions on every integration now. Here they are, in the order we take them.
1. Deterministic first, model last
Most enterprise documents are more structured than they look. A specification follows a numbering scheme, an invoice has a layout, an export has columns. Parse that structure with ordinary code, filter down to the part you actually need, and only then call the model.
We did this recently for a manufacturer of dock equipment. A 299-page building specification contained exactly five posts about their products. A parser found them in two seconds. Sending the whole document to a model would have meant close to a million characters per run, and a model guessing which pages mattered. Sending five posts meant a cheap, focused call per post, with a quote from the source on every answer.
Code is also what you can test. The parser has unit tests; the prompt has hopes.
2. The model emits operations, not free text
Do not ask the model for an answer. Ask it for a list of operations against a catalogue you own: add this feature with that value, remove this item, flag this question. Your code validates every operation against the catalogue before anything is applied. Unknown feature? Dropped. Value out of range? Dropped, and logged.
This turns the model from an oracle into a proposer. It cannot write to your database, run a query or invent a field. It can only choose from what you offered. New capabilities become catalogue changes, not prompt changes, and the catalogue is the place where your domain experts can read what the system is allowed to do.
3. Provenance on every field
A value that came out of a model needs a label saying where it came from. In the dock equipment case every configured feature carries one of four: quoted from the specification, deferred to a drawing, a default an expert would normally apply, or an open question. The reviewer sees sixteen quoted values, four deferred, two defaults and six questions, and knows exactly where to look.
A single confidence number does not do this. Confidence 0.8 does not tell the reviewer whether the model read the value or guessed it well. Provenance does. It is also what makes the output defensible when a configuration drives production and a wrong value becomes a manufacturing error rather than a typo.
4. One door to every model
Put a gateway between your applications and every model provider. Applications talk to an alias, "the extraction model", not to a vendor endpoint. The gateway holds the keys, routes the alias to a provider and version, logs every call, and reports cost per application.
We run one gateway ourselves. When a provider retires a model version, which they do on their own schedule, the alias moves and the applications do not change. When a team wants to try a different model family for one use case, that is a routing rule, not a code change. And cost per handled document, the number a business owner can weigh, falls out of the logs instead of out of a spreadsheet.
5. A path that works when the model does not
Rate limits, outages, a key that expires on a Friday evening. Decide before go-live what the feature does then. In the two systems we built this autumn, the fallback is a keyword parser that produces the same operations as the model, less cleverly, and marks its answers as offline. The screen keeps working. The user sees that the answer is weaker and why.
That fallback is not a luxury. It is the thing that lets you demonstrate the system on a customer's network where your gateway is unreachable, and the thing that lets you ship a model upgrade knowing that a bad day will degrade the feature rather than take it down.
6. Cache by content, replay on demand
Model calls are slow and cost money, and most of them repeat. Hash the input and the catalogue version, cache the output, and a second run on the same document is instant and free. The cache also gives you a replay: the exact output the model gave last month, for the audit that asks why the system proposed a value.
Invalidate on purpose. A new catalogue or a new model version is a new hash, so old answers are not reused by accident. A "refresh" button that bypasses the cache is enough for the cases where a human wants a fresh opinion.
7. Autonomy in stages, with a review screen
Pramod Sadalage and Prem Chandrasekaran describe this well on Martin Fowler's site: the model recommends, then prepares actions a person approves, then acts within guardrails. Each stage produces the evidence for the next one, and each stage is a safe place to stop.
For that you need a review screen that shows the operations, the provenance and the questions, and lets a person accept, edit or reject. It is the least glamorous part of the system and the part that decides whether anyone trusts it. Measure how often reviewers change what the model proposed. If that drops to zero, either the model is excellent or nobody is reading.
What we ask before the first prompt is written
- Structure: how much of the input can ordinary code parse before a model sees it, and what percentage of the volume does that cover?
- Catalogue: which operations is the model allowed to propose, and who owns that list?
- Provenance: for every output field, can the reviewer see whether it was read, inferred, defaulted or unknown?
- Gateway: can we move this feature to another model in a day without changing application code?
- Degradation: what does the user see when the model is unavailable for an hour, and is that acceptable?
- Replay: can we reproduce the exact output the system gave for a given input three months ago?
- Autonomy: which stage does the feature start in, what evidence moves it to the next, and how often do reviewers currently change the proposal?
A model that can only choose from your catalogue cannot surprise you. One that writes free text into your ERP eventually will.
Where to start
If you have a language model in production, or close to it, these seven are worth half a day with your architect. Most systems have two or three of them already and are missing the ones that hurt on a bad day. We are happy to walk through them with you, on your own integration.
