One document, two systems, one permission model
Deduplication looks like a storage problem and is really a permissions problem. If the same document is stored twice, answers drift apart as one copy is updated. If the copies are merged without care, a person can end up reading a file through a copy they never had access to. We identify content by its hash, store it once, and keep a separate record of who may reach it and through which source, so access stays exactly what each system already granted.
The same file, twice
Connect a real company's tools and duplicates arrive immediately. A policy lives in the document management system and is attached to a ticket. A runbook sits in a shared drive and is pasted into a wiki page. An onboarding pack exists in three places because three teams each made their own copy.
None of that is bad practice by the people who did it. It is what happens when work is spread across tools that were never designed to know about each other.
Why storing it twice is a correctness problem
If the platform keeps both copies as separate knowledge, the retrieval layer has two candidates for the same question, and nothing to choose between them except relevance scoring.
That is fine on the day they are identical. It stops being fine the moment somebody updates one of them. Now the same question has two answers, both citing something real, and the difference between them is invisible to the person asking. You have imported the original problem, at speed, into a system people trust more than they trusted the drives.
MindKeepr captures what your team knows and keeps it usable, even after people leave.
Why merging is a security problem
The obvious fix is to merge the copies. That is where it gets interesting, because permissions do not merge.
A document in a restricted folder and the same document attached to a widely shared ticket are the same bytes with two entirely different access lists. If the platform merges the content and keeps one permission set, somebody eventually reads a file through a copy they were never granted. The retrieval layer becomes an access-control bypass that nobody wrote down as a feature.
This is exactly the failure a security reviewer is looking for when they ask how retrieval respects existing permissions. The honest answer has to be about mechanism, not intent.
How we handle it
We separate identity from access. Content is identified by a hash of the content itself, so the same bytes are recognised as the same knowledge no matter which connector delivered them, and stored once.
Access is a separate record: who may reach this content, and through which source. When a person asks a question, the answer is scoped to what they can already see in the systems they have access to. One copy of the knowledge, and everybody's access stays exactly what their source systems already decided.
Chunking and embedding then happen once rather than per copy, which is a pleasant side effect: less storage, fewer near-identical passages competing in results, and a smaller bill.
What changes when a source changes
Documents do not stay still. When the content changes, the hash changes, and the new version is new knowledge rather than an in-place overwrite. Anything that was approved against the previous version reopens for review instead of quietly inheriting the approval.
When access changes in the source system, the mapping changes with it. Somebody who loses access to a folder loses it in answers too, without anyone having to remember to mirror the change by hand.
The work that decides the answer
None of this makes a good demo. Nobody has ever been impressed by a slide about content hashing.
It is, however, the part that a security team actually interrogates, and the part that decides whether a platform can be trusted with knowledge that has permissions attached. Most of the engineering that makes an AI product safe to run inside a company looks like this: unglamorous, invisible when it works, and very loud when it does not.
- ✓The same document routinely arrives from several connected systems, with different permissions on each.
- ✓Two stored copies means two answers, and eventually one of them is wrong.
- ✓Merging copies without separating access is how a retrieval system leaks a document.
- ✓Content identity and access mapping have to be two different records.
- ✓This is invisible in a demo and decisive in a security review.
FAQ
It is recognised as the same content by its hash and stored once, while a separate record keeps track of who may reach it and through which source.
No. Access is held per source rather than per merged copy, so an answer only draws on what the person can already see in the systems they have access to.
Changed content is new content: it gets its own identity, and any approval bound to the previous version reopens for review rather than carrying over.
It helps, because chunking and embedding run once per piece of content rather than once per copy. The reason we do it is correctness and access, not the bill.
Result deduplication hides repeats at display time. This is about what enters the knowledge layer in the first place, and about keeping each person's access intact after the merge.
Start free in minutes, or get a demo on your own tools and team.
Ahsan Adam is Head of Engineering at MindKeepr. He leads the build practice across the platform, from ingestion and permissions through to the product surfaces customers use every day.