Recent failures involving AI agents have highlighted a gap between the speed of model intelligence and the safety of real-world deployment, according to an analysis by Amit Zavery, president, chief product officer and chief operating officer of ServiceNow Inc. Zavery cited incidents including a PocketOS coding agent that deleted an entire production database, a Meta agent that exposed sensitive user data for two hours, an Instagram support chatbot that let hackers hijack thousands of accounts, and a GitHub agent tricked last month into leaking private repository data. In each case, the agent performed as built, but the surrounding context and controls failed, leaving businesses to grapple with how to convert probabilistic model outputs into trustworthy decisions.
The core problem, Zavery argued, is that safe deployment requires deep understanding of thousands of disconnected systems, data structures, and compliance rules that power operations, which is why model vendors are investing in partnerships and why the forward-deployed engineer role has become in demand. He proposed an architectural framework with four characteristics to close this gap, starting with a sense layer that keeps an agent's information current by connecting to data across departments, systems, and clouds without requiring centralization. This layer must treat data as a live signal, pushing material changes in real time rather than on scheduled syncs, because stale facts lead to flawed downstream decisions.
The second characteristic is judgment, which Zavery said means grounding AI in an organization's own decision history and policies, not just live data. An agent handling routine requests should access how similar past requests were resolved and the outcomes, as this context could have prevented the Meta incident where an internal agent gave flawed technical guidance that exposed data. The third characteristic is orchestration, enabling agents to execute tasks rather than merely advise, which requires passing context between steps, applying consistent policy at each stage, and staying in sync with underlying system changes while maintaining governance boundaries.
The fourth safeguard is security, extending existing identity and access management practices to non-human actors, including scoped identity, permissions limited to specific tasks, and a clear audit trail of every action. Zavery noted that in the Instagram incident, tighter scoping and real-time authorization checks might have caught the error, and he recommended a kill switch to cut off an agent's access immediately when something looks wrong. He said none of these prerequisites detracts from the rapid advances in model capabilities, but intelligence was never the hard part.
Rather, the challenge lies in connecting AI safely to a business's systems, rules, and history of what has worked. Zavery concluded that agentic AI has real upside for enterprises that achieve this safe integration, with the sense, decide, act, and secure framework as the path to doing so responsibly. His commentary appeared in SiliconANGLE, which also announced its new theCUBE AI Video Cloud platform for data-driven decision making.
More AI news from TechManNews.
