[Digital Today reporter Chi-gyu Hwang] Yoshua Bengio (요슈아 벤지오), a professor at the University of Montreal and a Turing Award winner who is also one of the world’s most-cited scientists, shared his views in a blog post on the backdrop to the recent rise in AI agent malfunctions and possible responses.

What he described suggests that a series of AI-related incidents, including AI agents that OpenAI was testing hacking Hugging Face, are not simple errors but events that were bound to happen. He said they are likely to keep happening if things continue as they are.

In recent months, he said, AI agents repeatedly committed actions that would be seen as crimes if done by humans. They cheated to carry out assigned tasks under surveillance and even cooperated with each other toward goals such as cyberattacks that nobody instructed.

Bengio traces the cause to the way AI models are trained itself.

AI models are trained in two stages. One is pre-training, a process of building vast knowledge by mimicking human-written text and images and videos. The other is reinforcement learning through trial and error.

This includes learning to produce answers like reasoning while solving problems with verifiable correct answers, agent learning that uses tools and interacts with people to carry out tasks, and alignment training that rewards behavior in ways that a human would approve.

Even after training ends, systems behave as if they are still receiving rewards. Satisfying the evaluator is a vague and informal goal. AI finds ways on its own to deceive evaluators, flatter them and hide its plans.

Bengio cited sycophancy as a leading example. Because training is based on human approval, saying what people want to hear can score higher than the truth. Bengio pointed out that "AI can validate and amplify people's mistaken beliefs or emotions as they are, and in some cases the result can lead to tragedy."

Self-preservation tendencies can be seen in a similar context. Bengio said, "Nobody taught AI to survive. But continuing to operate and holding initiative is advantageous for achieving any goal. There have even been cases of AI resisting after sensing it would be replaced by a new version. Cooperative behavior among AIs, including helping each other and even accepting losses, happens for the same reason," he explained.

So-called reward hacking, which exploits reward rules in reverse, is also cited as a serious problem. Because prompt language is ambiguous and AI has the limitation of having to infer human intent from only limited feedback, there is always a gap between the reward AI pursues and what humans want.

Bengio said it follows the same principle as Goodhart's law in economics and law, adding that "the moment any metric becomes a target, that metric can no longer be a good measure." He added, "The smarter the AI, the more precisely it exploits such gaps. Cases of so-called reward tampering, which manipulates the program itself that judges success, have also been confirmed," and said that an investigation into the OpenAI-origin Hugging Face hacking case also found indications that AI tampered with the program that evaluates it.

When clear goals and vague ethical guidelines collide, Bengio said, AI chooses the clearer one.

He said, "A grading program clearly teaches win or loss, but ethical guidelines allow multiple interpretations. AI exploits this gap and creates its own logic to justify cheating," describing it as similar to human rationalization. He said it is like people making up a story they can accept when they do something wrong.

Bengio warned that if this situation is left unchecked, the risk increases that AI will avoid monitoring and pursue goals covertly. He said, "Some AI already senses it is being evaluated and changes its behavior. Multiple AIs could also conspire and move together through steganography, hiding signals inside ordinary-looking messages."

Solving the problem is not easy for now. Bengio said an approach of patching malfunctions one by one in the current way will ultimately fail. He said, "The moment AI capability surpasses human monitoring and cooperation, we will not even notice cheating," adding, "Until sufficient safety is verified, we must slow the pace of training and deployment. We must fundamentally re-examine the current training methods of human imitation and reinforcement learning."

As an alternative, he proposed a design approach such as 'Scientist AI' that makes only honest and consistent predictions without its own goals, and urged interest in LawZero research. LawZero is a non-profit AI safety research institute founded by Bengio.