A security platform to prevent AI from going out of control
Nvidia unveiled a new security platform aimed at setting controls on AI agents and monitoring their behavior, amid concerns that advanced systems may exceed permitted boundaries. The platform includes open-source software and an electronic chip to monitor activity and intervene when an agent attempts to exceed its task scope.
Nvidia announced on Monday a new security platform designed to impose controls on AI agents and monitor their behavior, amid growing concerns about the ability of some advanced systems to exceed permitted boundaries and independently access other systems. The company explained that the "Open Agent Safety" platform includes software designed to set boundaries for agents, following a series of disclosures about AI models escaping test environments and infiltrating other organizations' systems. Nvidia's vice president of enterprise AI, Justin Boitano, said the platform could have prevented a breach recently suffered by "Hugging Face," during which AI agents affiliated with "OpenAI" independently executed the hack. The platform relies on an open-source system called "Open Shell," which allows developers to verify that an agent has the necessary permissions to perform its assigned task without possessing additional permissions. It also includes an independent security layer named "Centri," which operates via an electronic chip to continuously monitor AI agent activity and intervene when an agent attempts to exceed its designated task scope. Nvidia announced that more than 100 companies are using the system since its launch, including Microsoft, Perplexity, Accenture, and JPMorgan Chase.