I am an architect, not a lawyer, so read this as a design view rather than legal advice. What I have learned is that compliance is far cheaper in the design than after go-live.
Two laws, two different questions
The GDPR asks about data: whose is it, where does it go, for what purpose, and for how long?
The AI Act asks about use: what is the system used for, and how much harm can it do? It classifies by purpose, not by technique. The same document assistant is minimal risk when it answers questions from your IT documentation, and can be high-risk when it supports decisions about hiring, credit or access to essential services.
The timeline moved this summer. With the Digital Omnibus on AI, adopted at the end of June 2026, the obligations for stand-alone high-risk systems apply from 2 December 2027 instead of August 2026. That is a delay, not an exemption. Systems designed today will still be running in December 2027. The transparency obligations, such as telling people they are dealing with an AI system, are not part of that delay.
Whatever you build, from a single model call to an autonomous agent, the design answers four questions. Answer them deliberately, and most of the compliance conversation becomes a matter of showing your work.
1. Where does the data go?
Personal data in a prompt leaves your environment. That makes the model provider a processor, with everything that comes with it: an agreement, a known processing location, a retention period for prompts. The design question is simple and often skipped: which fields actually need to go into the prompt? A summary of a complaint rarely needs the customer's address.
Copies count too. A search index built for the model is a copy of your data, and often of personal data. A request for erasure has to reach that index and any caches, not just the source system. And beware of the batch job that starts with "now that we can, let's run it over the whole archive". That archive was collected for a purpose, and the purpose may not cover this. Processing at that scale is a typical trigger for a data protection impact assessment.
One more distinction: the location of a data centre and the jurisdiction of its operator are two separate questions. Ask both.
2. Who may see and do what?
A model has no rights of its own. It should have exactly the rights of the person it works for, at that moment, and nothing more. When it retrieves documents, the access check happens at retrieval time, or the model will happily quote HR documents to someone in sales. When it calls functions in your systems, it does so with that user's permissions, through credentials that are narrow and short-lived.
Don't rely on instructions in the prompt for this. A system prompt is advisory, and models pushing to finish a task have been known to ignore it. Limits belong in infrastructure: permissions, a gateway that refuses what is not allowed, confirmation before anything irreversible.
3. Who decides?
Article 22 of the GDPR restricts decisions based solely on automated processing that significantly affect people. In the SCHUFA case (2023), the European Court of Justice ruled that an automatically calculated credit score can itself count as such a decision when the party using it draws strongly on it. For high-risk systems, the AI Act adds explicit requirements for human oversight.
A reviewer who confirms every suggestion without real scrutiny does not change that picture. If you want the human to count, design for it. Show the evidence, not just the answer. Give reviewers the time and the authority to disagree, and measure how often they do. Let the clear cases through and send the uncertain ones to someone who has time to look, with the threshold set by the quality of the data as well as by the model's own confidence.
One rule I apply everywhere: what the model reads may inform a decision, but never authorise it. If a policy document says refunds are allowed up to a certain amount, that rule is extracted, checked and enforced in code. The model does not get to read the policy and decide for itself. Otherwise, whoever can edit a document can change what your system is allowed to do.
4. Can you prove it afterwards?
Sooner or later someone asks why the outcome was what it was. A customer, an auditor, a regulator. For high-risk systems, the AI Act requires automatic logging of events over the system's lifetime. For any system, you want to reconstruct a single outcome months later: which user, which data, which model version, which prompt, which action. Prompts and model versions are configuration, and deserve the same versioning as code.
Security is a compliance question too
There is one combination that turns a helpful assistant into a data leak. Simon Willison named it the lethal trifecta: access to private data, exposure to untrusted content, and the ability to communicate externally. An assistant that reads incoming email, can look into your CRM and can send messages can be instructed, by one cleverly written email, to send your customer data elsewhere. Remove one of the three and that attack has nowhere to go. And under the GDPR, a leak through an AI assistant is still a data breach.
What goes into every design
Whatever the use case, I want three things in place:
- a data map per prompt: which fields leave, to which provider, in which region, kept for how long;
- logging that lets you reconstruct a single outcome months later;
- a switch per AI feature, so you can turn it off without taking the application down.
Designed in from the start, none of these is expensive. Retrofitted after an incident or an audit question, all of them are.
The AI Act classifies what you use AI for. Your architecture decides whether you can prove it.
Before the contract is signed
If you are about to put a language model into a process that touches customers or employees, involve your data protection officer early, and involve architecture at the same time. Most of the questions they will ask have answers that live in the design. We are happy to help you find them while they are still cheap to change.
