A small advisory firm engaged New Clarity to evaluate a fundamental problem in their AI agent design approach, specifically, the fractured data available to the agents was causing inconsistent and unreliable results. Most companies running multiple AI agent hits a similar wall, where each agent digs through scattered, unstructured data, and ultimately limits the utility of the agents themselves. New Clarity built this advisory firm one shared, organized foundation of company knowledge, a single source of truth that unifies email, documents, the CRM, call transcripts, and accounting into one place, connects it around the people, clients, and projects it concerns, and serves it to every agent through one interface. Instead of utilizing an expensive data warehouse, New Clarity developed a solution using a low-cost database. The firm's agents now pull structured data from a single source of truth, and new agents plug in on day one instead of triggering another round of custom wiring.
After rolling out their first set of AI agents, the advisory firm quickly realized that the agents were running off of inconsistent, scattered data, and this was the limiting factor in their ability to create reliable workflow automations. In fact, most companies that runs a fleet of AI agents run into the same wall, since most agents are limited by the data it can reason over. That data almost never sits in one place and is exposed through one clean interface. Typically, a small company has data that is scattered across email, file servers, the CRM, call transcripts, and accounting, so before an agent can do anything useful it has to go find the right information, figure out who and what it concerns, and piece together how it all connects. This duplication of work ignores the fact that every agent needs the same fundamental thing, which is one shared, organized body of company knowledge with metadata that defines entities and relationships.
The obvious answer for medium sized to larger companies is a data warehouse or equivalent, since a warehouse is built to organize and relate company data. In practice it's overkill for many small businesses, since they can be expensive and complex, and can become its own project to run. Thus, the real problem for small businesses is more interesting, specifically, how do you give a fleet of agents a single, organized source of truth without standing up an expensive warehouse to do it.
The advisory firm we worked with is a great example of this. It was running several agents, one for drafting proposals, one for managing projects, one for client research, and one for prepping sessions, with no shared data foundation beneath any of them. Each agent reached into the raw sources on its own, and no agent had the full cross-source picture of the company's data. The firm couldn't trust that its agents agreed, and it couldn't add another agent without more custom wiring, so the whole effort came to a halt.
New Clarity built the firm one shared, organized foundation of company knowledge, a single source of truth that every agent reads through the same way, instead of each agent digging through the raw sources on its own. That shared source of company information is the thing that makes the whole fleet of agents work, since it gives every agent the same clean, organized picture to reason over rather than a bunch of individual connectors that each one has to assemble.
The idea is fairly straightforward. Rather than teach each agent to go hunt through emails, documents, CRM, call transcripts, etc, we do that work once, establish a consistent ingestion process that keeps the data fresh, and then hand every agent a view that is organized. Here is what that foundational data layer actually does:
It unifies the scattered sources. All of the firm's knowledge that used to live in separate systems is pulled into one place on a regular schedule, so an agent no longer has to know where a piece of information came from or go chase it down across five tools.
It organizes and connects the data. The information is not left as a raw data, but instead connects logical entities such as people, clients, and projects, and relates them to the sources it came from. So when an agent asks about a client, it gets the full cross-source picture, the emails, the calls, the CRM record, and the documents, all understood as the same client rather than disconnected fragments.
It makes the data intelligently retrievable. An agent asks a question in simple terms and gets back the correct, related information, instead of a list of keyword matches it then has to sort through. In other words, the foundation understands what the agent is really asking for, and hands back all the related data that actually matters.
It keeps the data safe and current. Sensitive information is handled and separated once, at the point the data comes in, so nothing leaks downstream to any agent that doesn't have the appropriate permissions. The foundation then refreshes as new information arrives so the agents are reasoning over what is true now instead of an outdated copy.
It serves everything through one interface. Every agent reads from the same place the same way, so an improvement to the foundation helps every agent at once. Any new agents plug in on day one without triggering another round of custom wiring.
The reason this works well is because we moved the difficult part off the agents and into one shared layer. An off-the-shelf connector only hands back matching content and leaves each agent to figure out who and what it concerns. This foundation instead settles who a piece of information is about, how it relates to other entities, and whether it is safe to provide the data to any particular agent or user. The agents then spend their effort reasoning instead of deriving the same context over and over. That is the difference between a fleet of agents that disagree with each other and a fleet that all speaks from the same set of facts.
The foundation guarantees three primary things about the data before any agent reads it: what data is true, when it was created, and whether it is safe to provide it.
At the center of the data foundation is a canonical entity registry, which is entity resolution across the whole company. It collapses every spelling and variant of a company, project, or person into a single golden record. It leans on an alias graph and name-core normalization to catch the obvious variants. A deletion leaves a tombstone behind, so a later refresh can't resurrect an entity that was already deleted. This establishes a shared identity map, since every agent asks "who is this" through the same map, so one customer never shows up as three, and a single request can fan out across email, calls, the CRM, and documents knowing they all point at the same entity.
On top of the settled entities sits a bi-temporal knowledge graph, which establishes when a fact was true. It tracks the valid time (when something was true) as well as the transaction time (when the system learned it). Facts are never overwritten, so a change supersedes the old value instead of erasing it, and the past stays intact and accessible. This enables an agent ask a point-in-time or "as-of" question, for example who the decision-maker was as of the March kickoff, and get a history back instead of a flat snapshot that only knows what's true today. If you overwrite facts in place, which most systems do, you lose the ability to answer that class of question entirely.
The third piece is guardrails, which means sensitive data and separation rules are applied once, at ingestion, rather than re-implemented inside every agent. The team uses redaction-at-source to scrub secrets and a policy-as-guardrail approach to keep separate client data from cross-contaminating. A single chokepoint means that no downstream agent can leak a redacted figure or blur two companies together, and each agent inherits trust and permissions, so it sees exactly what its tier allows.
The registry settles the entities, the knowledge graph adds the tense, and the guardrails keep the whole flow protected. Together they establish a trustworthy, time-aware single source of truth.
The foundation uses hybrid retrieval, which blends semantic vector search (finding content that's conceptually related even when the wording differs) with trigram lexical matching (catching exact names, identifiers, domains, and acronyms that a meaning-based search tends to miss). The results are merged into one ranking, so the information that multiple approaches agree on takes priority, and passages from the same document collapse into a single result so an answer reflects the consensus of multiple documents. For results that require high precision, a final reranker examines the top candidates against the original question, reorders them, gathers a deep pool, and then applies that step only to the shortlist. This is how we obtained broad recall and sharp precision at the same time.
The single biggest improvement in search quality came from contextual retrieval. An isolated passage often loses its meaning, since a line like "as discussed, we're good to proceed, the signed copy is attached" tells you nothing about which customer, deal, or amount on its own. Thus, every passage gets a short plain-language description of where it came from and what it concerns, with vague references resolved to real names, and that description travels with the passage and makes it findable.
Under the hood the data moves through three separated layers: a faithful raw capture that preserves each source, an embedded corpus that everything searchable runs against, and the entity and knowledge graph on top. Keeping the raw capture separate from the search layer is what keeps the whole thing straightforward (each source fills its own store) while search stays uniform (one collection, one approach), and everything downstream can always be rebuilt if needed. Re-indexing is triggered by changes, so a piece of content carries a fingerprint and is skipped if nothing changed, which makes the system cheap to run constantly.
We have found that the "fail loudly" principle is important in these systems to ensure that failures and fallbacks don't go undetected. A shared knowledge base that fails silently is dangerous, since agents will confidently answer from stale or missing data. If it can't do the job properly it raises an error to be fixed rather than providing a hidden lower-quality result. The context-enrichment and final rerank steps are treated the same way, in that they improve the answer when they run but are never allowed to block ingestion or retrieval if they're unavailable.
The firm now runs its agents on one shared foundation instead of across fragmented and unreliable data. Specifically, here is what the firm can now do that it couldn't before:
Deploy agents quickly and at scale. The shared foundation carries the identity resolution, the history, and the safety rules, so a new agent doesn't rebuild any of it. The firm can now stand up many more agents without the cost and delay of wiring each one to the raw sources.
Trust that their agents will agree. Every agent asks "who is this" through the same map and reasons over the same current facts, so one client no longer looks like three, and any two agents asked the same question give the same answer.
Ask questions across the entire data corpus. An agent can pull the full cross-source picture of a client or project (the emails, the calls, the CRM record, the documents, etc) understood as the same client, and it can ask point-in-time questions that a flat snapshot could never answer.
Keep sensitive data safe by design. Redaction and client separation happen once at the point data comes in, so no agent downstream can leak a redacted figure or blur two companies together, and each agent inherits the access their tier allows for.
In terms of an overall solution, the interesting part of the engagement came down to the economics. All of this runs on an ordinary, low-cost database rather than a data warehouse. Most companies developing more than one AI agent hits a similar wall, and a properly designed data foundation is what moves them past this fundamental roadblock and into the era of productive agents.