Despite its progress: AI models struggle with complex software modifications

A new test conducted by researchers from 'Alibaba' in collaboration with Asian universities revealed that the best AI models, such as 'GPT-6 Astra' from 'OpenAI', still fail to fully complete complex programming tasks, with a success rate not exceeding 26.4% out of 273 tasks.

A new test conducted by researchers from 'Alibaba' in collaboration with Asian universities revealed that advanced AI models still face difficulties in fully completing complex programming tasks. The test, named 'Code Design Bench', showed that the best-performing model, 'GPT-6 Astra' from 'OpenAI', managed to meet all task requirements only 26.4% of the time, despite achieving much higher results when evaluating requirements separately. The researchers relied on 37,463 unique requests collected from 26,314 actual programming sessions, focusing on cases where users encountered clear difficulties with AI tools, such as rephrasing the request more than once or manually undoing modifications made by the tools.