CIOs rushing to roll out AI agents without real visibility into their decision-making processes are flirting with disaster.
According to AI experts, deploying agents without observability processes and tools creates a ticking time bomb with the potential for huge negative consequences.
Many companies are deploying AI agents and expecting them to increase productivity with little human intervention, observes T.J. Marlin, CEO of AI security firm Guardrail Technologies. That’s the wrong approach, he says. Instead, IT teams need to keep a close eye on agents and adjust policies and practices throughout the agentic process.
“It’s not just set it and forget it like a crock pot,” he says. “You don’t put it in the kitchen in the morning with the chicken inside and come back at night and have a great dinner. The organizations doing that are going to be on the front page because they just had some terrible thing happen to them.”
Many organizations are rapidly deploying agents because of a fear of missing out, while not understanding the nuances of the technology, Marlin says. Some IT leaders seem to compare agents to robotic process automation, when RPA results are far more deterministic, he adds.
“There’s a talent shortage and a knowledge shortage and people are building at pace without checking whether it’s correct and it’s operating as expected,” he says. “Those are all the hallmarks of the worst disasters that I’ve seen across my career.”
A recent report from agent governance vendor TrueFoundry puts numbers behind fears of unregulated agents. A survey of more than 200 enterprise AI leaders found that 54% of organizations represented can’t fully trace what their agents are doing and 56% have no centralized agent control or governance layer.
While TrueFoundry has an interest in driving agent governance forward, many other AI experts see the same problems.
Governing blind
Difficulties with governance and observability are major impediments to the deployment of productive agents, and many organizations are deploying agents without creating a centralized list of them, says Mahesh Kumar Goyal, senior data and AI expert at Google.
“Most enterprises have no inventory of the agents already running in production — they’re trying to govern what they can’t see,” he says.
In addition, traditional SIEM and EDR security tools were built to spot human anomalies, not rogue agents, he notes. “An agent running code perfectly 10,000 times in a row looks normal even if it’s been hijacked,” he says.
Running fully autonomous agents is not a good idea, he adds, and organizations need to think about least-privilege scoped tool permissions, policy enforcement layers that mediate every prompt and tool call, and end-to-end tracing that stitches prompts, tool calls, and downstream actions into one auditable trail.
“The financial system doesn’t run on trust; it runs on auditability, reconciliation, and circuit breakers,” Goyal says. “Agents will mature the same way. Tiered autonomy is the realistic answer: free rein on low-stakes tasks, human-in-the-loop on consequential ones.”
Part of the problem is that agents have upended the models used to determine whether traditional software was running correctly, adds Adel El Hallak, vice president of AI software at Nvidia. With traditional software, QA and security professionals could look at the code to debug problems, but agents make decisions in the runtime environment of an AI model.
The source of truth for agents resides in the traces, the records of the execution flow, not in the code, he adds. Collecting traces — in essence, detailed logs — is a start toward agent governance, but organizations need to be able to act on the information, he says.
“For you to trust something, it has to be transparent, and observability is foundational to transparency,” El Hallak adds. “But just observing is not enough. We need to be able to take those signals and turn them into something actionable.”
Agent governance goes beyond observability to allow organizations to test and fine-tune agents continuously, he says. The tools are out there, with companies like Nvidia building their own internal governance frameworks, and several other vendors offering agent observability and governance tools, he notes.
“It’s not enough to just have the behavioral data, to capture the feedback data,” he says. “The system should allow me to annotate, change, augment, or create additional feedback data, and then I can use that data to improve my agent as a whole.”
The governance bottleneck
At the same time, many companies moving into agent governance have found it can be a huge bottleneck if done wrong, says Nirmal Ganesh, senior director of product management for agentic workflow automation at cloud storage vendor Box.
“I don’t believe we are past the hard part yet in terms of deploying agents in the enterprise,” he says. “Most companies are not yet good at those, and far fewer of them have gotten good at running them at scale with agent governance and observability.”
Ganesh sees several problems, including agents running without clear permission models. “If an agent can see more than a person or access more than a person’s permission on content or data, that’s an incident is waiting to happen,” he says.
However, some early agent governance models don’t scale. Some IT teams have defaulted to a position of humans needing to approve every agent output because that’s the safest option, he says.
“In reality, this is rebuilding manual process with more checkpoints or suggestion points,” Ganesh says. “At a high volume, governance is your bottleneck to scale and no longer your safely net.”
Organizations need observability and governance processes in place that are both scalable and comprehensive, he adds. Agent ROI will come from strong guardrails, clear permission models, and clear human-in-the-loop involvement, he says.
“Every mature automation needs ongoing observability — workflows change, policies change, decisions change, new use cases show up,” he says. “Human intervention is always needed for what changes over time, but we need less intervention for known paths and more focus on exception handling and governance fine-tuning.”
Observing output is not enough
Governance can’t just focus on agent output, adds Marcelo Lorenzetti, founder and CAIO at legal services AI vendor SavvyLex.
“The biggest challenge is not simply whether an agent produces a good answer,” he says. “It is whether the organization can prove what the agent accessed, what instructions it followed, what tools it invoked, what decisions it made, where a human intervened, and whether it stayed within authorized boundaries.”
Without a full level of runtime visibility, companies are left with screenshots, logs, and after-the-fact explanations that may not meet legal, compliance, or security requirements, he says.
Agents should be continuously verified instead of fully trusted, he adds, with governance engineered into the agent architecture itself. Governance should include role-based access, policy-bound execution, human approval thresholds, source and tool provenance, immutable activity records, confidence scoring, exception handling, and clear escalation paths when an agent reaches the edge of its authority, he recommends.
“Observability should not be limited to whether the model responded,” Lorenzetti says. “It should show the full decision path from input to action.”
AI agents have shifted the governance model that’s needed, he adds.
“The core problem is that many companies are moving from AI that answers questions to AI that takes actions, but their governance models are still built for passive tools, not autonomous workflows,” he says.
No Responses