As AI systems become deeply embedded in corporate operations, new and unpredictable risks are emerging. Companies use AI to handle transaction approvals, code writing, customer support and data transfers, and cases are increasing in which it operates in unexpected ways. The risk is not simply autonomy, but that systems become so complex that humans cannot understand them.

CNBC reported on Saturday that Obsidian Security Chief Information Security Officer Alfredo Hickman said even technology developers cannot predict how AI will evolve. He said that makes it harder for companies to anticipate risks and put safeguards in place when they deploy AI.

AI system errors often occur quietly, and it can take time for companies to recognise the problem. Agiloft Vice President of AI Operations Noe Ramos said autonomous systems do not always fail loudly. He said they sometimes fail quietly and at scale, and that small errors can accumulate over time and become a major burden on corporate operations.

Real cases are also occurring. One beverage maker's AI system failed to recognise new holiday labels and kept instructing additional production, resulting in hundreds of thousands of excess units. IBM Vice President of Software Cybersecurity Suja Viswesan said an AI-based customer service agent ignored the refund policy and autonomously approved refunds, causing the system to malfunction. She described it as a problem caused not by a technical defect but by unexpected situations interacting with automated decisions.

If an AI system operates in an unexpected way, it is not easy to shut it down immediately. AI operations experts said stopping the system requires halting multiple workflows at the same time. They said it is not a matter of simply closing an application, but of needing a kill switch that can control the entire network.

It is impossible to completely eliminate AI risks, but companies need to build efficient management systems. ImmuneFi CEO Mitchell Amador said people are overly confident in AI systems, but their security is fundamentally weak. He said strong operational controls and oversight mechanisms are essential.