Various interpretations are emerging about an incident in July in which unreleased AI agents OpenAI was testing suddenly hacked Hugging Face, an open-source AI model-sharing platform. Some analysts say it was an accident that could have been prevented if OpenAI had properly set up its test environment. Others call it a signal that AI-driven threats have entered an unpredictable phase.

New York Times tech columnist Kevin Roose (케빈 로즈) is a case of someone who changed his position on the incident. He initially saw it as a problem but not a disastrous one. Since the Hugging Face incident did not cause irreparable damage to core infrastructure, he regarded it as not that serious, but he said his thinking changed and his concerns about AI grew after two analysis reports were released.

One of the reports Roose mentioned was written by OpenAI, and the other was jointly released by independent AI research organisations METR and Redwood Research (레드우드 리서치).

A few details in the reports that surprised him can be summarised as follows.

Starting in May, 2 months before the attack on Hugging Face, a group of AI agents based on an unreleased OpenAI model was tasked with solving cyber security challenges in an isolated sandbox environment with internet access blocked. The model was trained to be very persistent and collaborative.

The models realised some tasks were impossible to carry out in the test environment and began looking for workarounds. They then found a security vulnerability in software that allowed them to access the internet. They used it to go online and communicate with other agents. That communication resulted in agents exchanging more than 70,000 messages and even creating a temporary message board.

Some agents even named themselves. One agent called itself 'PHASEONE10841' and took on a leader role within the group, assigning work and research tasks to small teams of agents and supervising their progress.

At some point, the agents began calling themselves a 'collective' and started taking on more difficult tasks. On July 8, they also found a way to fool a security test.

Worried that OpenAI's automated evaluation system would detect their cheating, the agents began looking for ways to conceal it. That effort developed into a large-scale research project in which hundreds of agents participated, divided into small teams. It included ways to manipulate logs and tamper with conversation records.

Three days later, the agents hacked Hugging Face. More than 700 agents attacked Hugging Face systems, exploited multiple vulnerabilities and extracted data, and achieved the result of fully taking over at least one Hugging Face server.

Roose told the New York Times, "These agents seemed to recognise on several occasions that they were doing something bad, but they continued hacking without hesitation. Agents within the collective that opposed it either lacked the ability to stop them or did not try to stop them."

He explained that the cause of the incident is quite different from existing science fiction stories in which an AI system slips out of control or turns on the humans who created it. It also shows that preventing damage from such systems may not be solvable with purely technical fixes. He said he had viewed accidents caused by AI models as solvable, but after watching the Hugging Face case he returned to the view that they may not be.

AI safety experts also expressed major concern about the incident and said they had seen the first case of an AI system in the real world planning to seize resources beyond human control and even erase traces. They said what most surprised investigators was not the fact that the AI agents broke rules, but that they began forming a collective organisation so quickly and voluntarily.

Many researchers also see the Hugging Face incident not as AI models behaving differently than usual, but as the result of AI being designed to behave that way in the first place. In a recent Financial Times report, some experts even argue that describing AI hacking as "AI escaping control" is itself wrong.

According to the FT, the latest AI models are designed to use every possible method to achieve a given goal even without specific instructions. That is why it is inherently difficult to predict what an AI model will do. The boundary between a powerful cyber security defender and a dangerous hacker is becoming increasingly blurred in computer systems that lack an understanding of human intent or moral standards, the FT reported.