Five questions to ask before an AI agent touches your data

Most reviews of an AI agent start with the model: which one, how accurate, whether it trains on your data. Those matter, but the riskier questions are about the access around the model. An agent is software that acts on your systems, so the review should look like one for any other service with broad access.

These five questions take about an hour to answer. They catch most of what goes wrong.

1. Whose identity does the agent act as?

If the agent uses one shared service account, every user inherits that account’s access. Someone who should only see their own team’s files can ask the agent for another team’s.

Ask whether the agent acts as the person who asked, and whether that is enforced by the systems it calls, not only by the agent’s own code. A rule that lives only in a prompt is a request, not a control.

2. What can it see, and who decided?

List every data source the agent can reach. For each one, write down the rule that limits what comes back.

Search and retrieval deserve special care. An index built for everyone answers for everyone. If documents are indexed without their permissions, the agent can quote a file the asker was never allowed to open.

3. What leaves your perimeter?

Prompts, retrieved passages and tool outputs all travel to a model. Ask where that model runs, what is stored, and for how long.

Check whether sensitive values are masked before they leave, and whether your own tools still receive the real values they need. Treat “we do not train on your data” as a contract term to read carefully, not as a technical control.

4. What is logged, and can you replay it?

You should be able to answer “what did this agent do last Tuesday, and on whose behalf?” without asking the vendor. Check that actions, inputs, the rule that applied and any human approvals are logged somewhere you control.

If an incident happens, this log is the difference between a short review and a long investigation.

5. How do you stop it?

Ask how access is revoked for one user, for one agent, and for all agents, and how quickly each takes effect. A good answer is a switch you control, with a result you can verify.

Also ask what the agent does when a rule is unclear or a service is down. Failing closed is safer than guessing.

What to do with the answers

Write each answer down and mark it as one of three things:

• Enforced by a system. The platform will refuse, regardless of what the agent tries.
• Enforced by policy. People are expected to follow a rule.
• Unknown. Nobody can say.

Fix the unknowns first, then move policy-only items into systems where you can. If you would like a second pair of eyes on the review, that is the kind of work I do: book a free 15 minute call.