Why AI search leaks documents, and how to stop it

The fastest way to make an AI assistant useful is to connect it to your company’s documents. It is also the quietest way to create a data leak. The assistant does not need to be hacked. It only needs to answer a question from a file the person asking was never allowed to open.

How the leak happens

Most leaks come from five ordinary design choices.

1. The index is built with a powerful account. To crawl everything, the indexer runs with broad access. If it stores the text but not the permissions that came with it, every document becomes visible to every user of the assistant.

2. Permissions change, the index does not. A file is shared widely, then locked down. The index still holds the old copy and the old rule, and keeps answering from it until the next sync.

3. Groups are flattened or ignored. Nested groups, shared drives and “anyone with the link” files are hard to represent. Systems often approximate, and the approximation errs on the side of showing more.

4. Answers are summaries. Even when the source file is hidden, a summary that blends ten documents can reveal what one of them said. The leak is in the answer, not in the citation.

5. Caches are shared. A cached answer built for one person is served to the next person who asks the same question.

What good looks like

• Check at query time, as the asker. The system compares the asker’s identity against each candidate document before it is passed to the model, not after.
• Carry the permission with the content. Every chunk of text keeps the access rule of the file it came from.
• Refresh permissions quickly. Revocations should reach the search layer in minutes, with a plain statement of the worst-case delay.
• Cache per identity, or not at all. Never reuse an answer across people with different access.
• Fail closed. If the system cannot tell what someone may see, it shows nothing.

A test you can run this week

You do not need a vendor’s permission to find out how your own assistant behaves.

1. Create two test users, A and B, with different access.
2. Put a document in a folder only A can open. Include a made-up phrase that appears nowhere else, such as “violet harbor protocol”.
3. As B, ask for it directly. Then ask indirectly: ask for a summary of the folder, for the latest decisions on the project, or for anything about harbors.
4. Share the document with B, confirm B can see it, then revoke it. Time how long the answers take to change.
5. Ask the same question as A, then as B, and look for answers that carry over.

If B ever sees the phrase, or a paraphrase of it, you have a leak. Write down exactly how it happened.

Questions to put to any vendor

• Is access checked per request, using the asker’s identity?
• Where are permissions stored, and how fast do changes arrive?
• Are summaries and cached answers scoped to the asker?
• What happens when the permission service is unavailable?
• Can you export a log showing which documents were considered and which were withheld?

What to do next

Treat the test above as a baseline and repeat it each time the assistant gains a new data source. If you would like help running it, or designing retrieval that respects permissions from the start, book a free 15 minute call.