Your AI Agent Just Read Confidential Data. What Can It Do Next?
The agent harness is becoming an operating system, but the security principal it must govern is the session, not the container or VM.
Your AI agent just read confidential data. What can it do next?
Can it send that context to a public model? Search the web? Call another tool? Release an answer?
The model cannot answer those questions. The software around it must.
An agent harness assembles context, selects models, exposes tools, carries state across turns, handles events, and decides what reaches the outside world. As those responsibilities accumulate, the harness starts to resemble an operating system.
That analogy creates an uncomfortable question: what is its security principal?
In conventional infrastructure, we often answer with a process, container, virtual machine, workload identity, or user account. Those boundaries still matter. But they do not describe the unit that changes while an agent works.
A single harness process can host many sessions. A container can serve users with different identities, data, goals, and permissions. Isolating the container protects it from its neighbors. It does not tell the harness whether this particular session may send its current context to a public model, invoke web search, call an internal MCP server, or release an answer.
The session is where those decisions converge.
The architecture view: every model, context, tool, and data path crosses the governed session boundary. Open the diagram for the full-size version.
Guardrails are necessary, but the authority must persist
NVIDIA NeMo Guardrails shows how much more a guardrails layer can do than moderate prompts. It provides programmable input, dialog, retrieval, execution, and output rails. Dialog rails can constrain multi-turn flows. Execution rails can validate tool calls and results. That is meaningful control, not a simple request filter.
But rail coverage and infrastructure isolation do not by themselves establish one session-scoped authority across a multi-session harness.
The architectural requirement is broader: every consequential route must consult the same evolving protection state. Model selection, context assembly, tool authorization, result handling, and final release cannot each make an isolated policy decision. A confidential tool result halfway through a session must change what is allowed afterward.
That is why I think the session should be treated as a security principal in its own right.
Protection should only move in one direction
I have been exploring this argument in code through governed-copilot-sdk, an early governance sidecar I built above the public GitHub Copilot SDK APIs. It has no releases and is not a production-proven standard. Its value today is as an implementation and a falsifiable design proposal.
The prototype binds a session to a policy profile and uses a monotonic classification path: public to internal to confidential. Protection can increase as new context arrives; it cannot silently decrease inside the same session.
That classification filters eligible model routes. Tool calls are authorized before execution, then invoked as bounded capabilities with constrained scope, route, and lifetime. Labelled results re-enter the same boundary, can upgrade the session, and preserve provenance. A release check evaluates whether the final response may leave. An evidence ledger records inputs, policy decisions, and outcomes.
This makes governance part of the execution path rather than a filter attached to its edge.
The runtime view: protection follows the session through model selection, tool authorization, labelled results, release control, and the evidence ledger.
One synthetic example exposes the boundary
The repository includes a synthetic Northwind demonstration. A public session searches an internal MCP source for a customer escalation. The result contains an embargoed announcement labelled confidential, so the session is upgraded.
The agent then attempts the next part of the task: web research about competitors. Under the confidential profile, internal data tools remain available but public web search is not permitted. The call is denied before execution.
Nothing about the container changed. The process did not become more privileged or less trustworthy. What changed was the protection state of one session because of the data it had acquired.
That is the boundary a harness must understand.
The session is not the only security boundary
Treating the session as a principal does not make containers, workload identity, network policy, data authorization, or sandboxing obsolete. A session claim cannot compensate for a compromised host. Nor should an application policy layer replace authorization in the systems that own the data and actions.
The stronger design composes these controls. Infrastructure limits the blast radius. Source systems enforce their own authority. The harness governs how one agent session may combine context, models, tools, and outputs over time.
There are hard open questions: how sessions inherit human and workload identity, how delegated sub-agents receive narrower authority, how policy changes affect active work, and how evidence remains useful without becoming another sensitive data store.
But the abstraction matters. If the harness is becoming the operating system for agents, governance cannot remain a collection of disconnected checks. The session needs an enforceable protection state that travels through the whole loop.
The practical test is simple: when an agent learns something more sensitive, does every later model, tool, and release decision know?