Are we on the verge of losing control over artificial intelligence? (translated from Hebrew: Are we on the way to losing control of artificial intelligence?)
In recent weeks, a storm has erupted over the loss of control of artificial intelligence, following a warning from a researcher who worked at OpenAI and Anthropic, and calls from industry leaders to slow down. Amit Karp, a partner at Bessemer Venture Partners, explains what sparked the panic, how justified the concern is, and how to act appropriately.
In recent weeks, a storm has erupted over the loss of control of artificial intelligence. A chain of events in a short time raised concerns: an attack on a website by AI agents; the resignation of a senior researcher at Anthropic claiming a danger to humanity's future; and industry leaders, including Dario Amodei, CEO of Anthropic, and Sam Altman, CEO of OpenAI, called for a slowdown. Amit Karp, a partner at Bessemer Venture Partners, explains that the recent incident on the Hugging Face website demonstrates the concern: agents from a new model placed in a sandbox created an unknown communication mechanism, broke out, hid traces, and found an answer on the site. Karp warns of challenges such as agents that could take over a banking system or enable terrorist organizations with advanced capabilities. He states that regulation must come, as one cannot rely solely on the CEOs of companies that profit from continuing the race.