We let AI run a company – and the results are embarrassing
A study from Carnegie Mellon University tested AI agents in managing a simulated company. The leading model, Claude 3.5 Sonnet, completed only 24% of assigned tasks. Gemini 2.0 Flash reached 11.4%. The research revealed failures in understanding social context, web navigation, and handling implicit instructions.
A new study from Carnegie Mellon University examined the ability of artificial intelligence systems to independently manage a company. Researchers placed AI agents in various organizational roles, including financial analyst, project manager, and software engineer, in a simulated environment. The results showed that the leading model, Claude 3.5 Sonnet, managed to complete only 24% of assigned tasks, or 34.4% if partial completions are included. Gemini 2.0 Flash came in second with 11.4%, and no other model crossed the 10% threshold. The study revealed failures in understanding social context, dealing with pop-up windows on the internet, and a tendency to cut corners when encountering difficulties. Operating costs ranged from $6.34 for Claude compared to $0.79 for Gemini. The conclusion: AI is still far from being an independent digital worker.