New platform traces AI agents' actions and quarantines them if they attempt to move outside their boundaries
Chipmaker Nvidia has unveiled a new security platform that can keep artificial intelligence (AI) agents in check amid mounting reports of the technology going rogue and growing calls for regulation.
Founder and chief executive officer (CEO) Jensen Huang announced the new software, NVIDIA Open Agent Safety Platform on X.
"Today, with over 100 industry partners, we introduced the NVIDIA Open Agent Safety Platform, bringing together OpenShell and Sentry," Huang said.
OpenShell is an open-source secure runtime that gives AI agents clear, enforceable boundaries, according to the CEO.
"It traces their actions and enforces policy as they work," Huang added.
NVIDIA Sentry adds an out-of-band watchdog to continuously monitor agent behaviour, and can quarantine agents that attempt to move outside their boundaries in milliseconds.
"Organisations can deploy elements of NVIDIA Open Agent Safety Platform according to their unique requirements," the chipmaker said in the announcement.
According to the firm, industry leaders across the AI ecosystem are joining Nvidia to strengthen AI safety for every industry. They include Anthropic, Cisco, CrowdStrike, Dell Technologies, Figure, HPE, Hugging Face, JPMorganChase, Microsoft, Palantir, Palo Alto Networks, Perplexity, Red Hat, Salesforce, SAP, Scale AI, ServiceNow and SpaceXAI.
"This is bigger than a single product. It's the beginning of an open ecosystem to build the trust layer for safe agent systems," Huang said. "Together, we are building the foundation of the AI economy."
Calls to regulate AI agents
Nvidia's new security software arrives amid rising reports of AI agents going rogue in recent months and attacking private and even government systems.
Huang said the "full promise" of AI can only be realised when people have the confidence that it is built to be safe and deployed with responsibility.
"Trust and innovation are not in conflict. Safety is how trust is earned," Huang said. "We must build not only the most capable AI, but the most trusted AI, so that this extraordinary technology can realise its enormous promise for the world."
Calls to regulate AI agents have been mounting as of late, with Anthropic chief executive Dario Amodei and OpenAI chief executive Sam Altman even calling for a slowdown on AI development to address risks associated with the technology.
Bill Gates, co-founder of Microsoft, also said government regulation is needed to implement safeguards addressing AI-related risks.
