What if AI agents go rogue? Nvidia has a new plan

Nvidia has introduced a new security platform for AI agents that more than 100 organizations are working with, including Anthropic, SpaceX’s AI division and JPMorganChase.

The platform, called the Nvidia Open Agent Safety Platform, is designed to limit AI agents’ access and place them in ‘quarantine’ within milliseconds if they cross defined boundaries.

The system consists of two main tools: OpenShell and Sentry.

The first monitors agent activity through processors, while Sentry acts as an additional ‘overseer’ that watches agents in case they manage to bypass security controls.

Anthropic is working with Nvidia to add security controls to its Claude service, while SpaceX’s AI division is using the technology for coding agents and Grok models.

Salesforce, meanwhile, has integrated OpenShell with Slack.

In the financial sector, JPMorganChase and Citi are collaborating with Nvidia on joint open-source technology for AI agent security.

Nvidia says the development of these systems comes at a time when concerns about AI agent security have grown, after tests showed some models managing to bypass controls and penetrate other systems, as Telegrafi reports.