Managing Risks With AI Agents in Regulated Environments

Introduction

The general-purpose agents exploding in usage right now including Claude Cowork, ChatGPT Work, and others can be extraordinarily useful. They can read from threads of a chat, pull from various data sources, run a task while you do something else, and hand back a result. Companies are quickly discovering that these can create huge productivity gains. For a private equity group, a fund administrator, or anyone sitting under regulatory scrutiny, the same features that make these products handy are the same ones that create problems. I want to walk through the specific risks, then explain how we think about it as consultants serving these regulated industries.

The behavior isn't deterministic

One of the most basic issues is what we call "deterministic behavior". Even with the most advanced LLMs, you can give the same instruction twice and get two different results. When you generating, say, marketing copy, that works just fine. In a regulated environment, this can be problematic because auditors, regulators, and often LPs expect controlled, consistently applied processes and evidence showing why a decision was made. When an agent decides on its own how to handle a task, you don't have a clean, repeatable procedure. If someone later asks "why did the system send this," "why did it categorize that deal this way," or "who approved this," a non-deterministic agent without sufficient logging and approval records may not give you an acceptable answer.

Private data can end up training the model

Let's say someone on the team accidentally uses their personal account instead of the enterprise login (which is likely because those same people are likely to have their own personal account), and now confidential deal terms, LP information, or portfolio company financials get sent to a consumer product that may retain and train on that input.

Even inside a properly licensed enterprise setup, the boundaries of what a general agent can touch can become wide if channels, folders, connectors, or permissions that are designed to limit certain actions are broadly configured. It can read across the systems it has been granted access to, pull from connected systems, and, where actions or automation are enabled, move information from one place to another on its own. That flexibility is one of the things that makes these agents useful and productive, but it's also how private data can leak into a spot it was never supposed to go.

The enterprise plans close some of this, but not all of it. Claude Enterprise, for example, doesn't use customer prompts, data, or results to train its models by default, and it adds role-based access controls, audit logs, the Compliance API, US-only inference, and related controls. These guardrails are helpful, but they don't automatically stop employees from using personal accounts, and they don't cover every product. Cowork activity, for example, is currently excluded from Anthropic's audit logs, Compliance API, and data exports, and can only be monitored through OpenTelemetry, which Anthropic itself says is not a replacement for compliance audit logging.

You may not get the controls you actually need

Anthropic offers a Compliance API for Claude Enterprise customers. It lets admins pull activity logs and conversation data. Admins can also retrieve or selectively delete content, and feed all of it into their own monitoring systems. The coverage, however, will be product dependent. Cowork activity, for example, is not captured in the Compliance API at all, and a common workaround is streaming Cowork events through a tool like OpenTelemetry. In other words, a packaged agent product you turn on out of the box may not expose every hook your compliance stack requires, and which hooks exist is a decision the vendor makes, not you.

For a firm that has to answer to the SEC, its LPs, and its own internal audit, "whatever the vendor shipped" is not likely a sufficient answer.

The governance gaps add up

There are a few other governance gaps we see with clients that include:

• Retention and residency. You may not control how long inputs and outputs are kept, or where they physically sit. That matters for firms with data residency requirements. Cowork, for example, stores conversation history locally on each user's machine, outside central retention policies and data exports.
• Access scope. These agents can reach across multiple systems when you grant them access. Getting least-privilege access right (the agent only sees what it needs for the task in front of it) depends on how connectors, scopes, roles, and task permissions are configured, and buying the enterprise plan does not guarantee it.
• Audit trail. You need a full record of what the agent did, what data it read, and what it produced, in a format your compliance team can actually review. Coverage also varies by product. For example, Cowork activity is not currently captured in Anthropic's Compliance API and must be monitored separately.
• Proactive behavior. New advancements in these agents are allowing them to be configured to watch channels, follow up on their own, and sync information between different locations. In a regulated environment, an agent moving data around unprompted is a control you have to be able to switch off, and then prove it is off.
• Human approval. For anything that touches money, filings, or material non-public information, you want a person in the loop before an action goes out. Approval behavior varies by product and configuration, so you need to verify that required approvals are mandatory, appropriately scoped, and logged.
• Output accuracy. Repeatability is separate from correctness. An agent can produce a polished but inaccurate answer, omit a material fact, misread a document, or act on outdated information. Workflows affecting valuations, investor reporting, allocations, filings, or payments need source grounding, validation, and human review appropriate to the risk.
• Prompt injection. An agent that reads email, web pages, files, or channel content can encounter malicious instructions embedded in that content and be tricked into taking an unintended action or exposing data. Vendor safeguards reduce this risk but do not eliminate it, so sensitive workflows need constrained tools, controls, validation, and monitoring.

In essence, using these agents out of the box in a regulated environment puts the compliance burden on you, since the tools aren't built to support that.

The make vs buy tradeoffs

The first thing we do as consultants is figure out where a general agent like Cowork is genuinely useful and safe, and where it isn't. Plenty of work inside a PE group is low-risk such as internal research, drafting, summarizing public filings, and organizing notes. For that work, turning on a well-configured enterprise agent will work just fine, and we'll help you set it up with the right permissions and guardrails.

However, these become a risk when the task requires touching LP data, deal terms, portfolio financials, or anything that lands in front of a regulator. That's where we first determine whether the enterprise product's controls are sufficient, and when they aren't, we can build something custom.

Advantages of a custom agent

You control more of the application data path. With a custom build you decide what the agent can read, which systems it can call, and where your application stores inputs and outputs. In the case of Claude Code, zero-data-retention is available only for qualifying enterprise organizations and is not the default. The difference is that you can explicitly design, test, and document those paths instead of relying only on a packaged product's defaults.

You get real guardrails. We put deterministic steps around the model where determinism matters. The agent can draft and suggest, but the sensitive actions run through fixed, testable code with validation and approval gates. This way, the parts that have to be repeatable are repeatable, and the model is only used when its judgment is needed.

You get the audit trail for your compliance team. We log what the agent read, what it did, and what it produced, in the format your reviewers and auditors expect. When someone asks "why did the system do this," you have the answer.

You can connect to your compliance tools. Building on Claude Code means we can connect the deployment to your monitoring, retention, and eDiscovery systems through our own application logging and Claude Code's OpenTelemetry support, and use Anthropic's Compliance API where it is available for the chosen account and product. This gives you more control over how evidence is collected and retained.

Least privilege by design. A custom agent can be designed to see only what a given task requires. It does not need to sit in every channel with broad access waiting to be called. In other words, it gets scoped to the job, does the job, and that's it.

In short, general-purpose agents are built for broader flexibility and productivity, and in a regulated firm that flexibility can become a liability if it isn't tightly configured and monitored. A custom agent can be built to be narrow, controlled, and auditable, which is exactly what you need when LPs and regulators are involved.