A range of scenarios has been laid out in which artificial intelligence could threaten humanity’s survival, from losing control to a superintelligence to bioweapons, nuclear war and social collapse. A path in which human society destabilises itself as mass unemployment, disinformation and political polarisation overlap was seen as a relatively realistic risk, compared with a situation in which AI suddenly attacks humans.

On Oct. 2, online outlet Gigazine reported that Toby Walsh (토비 월시), an AI researcher at the University of New South Wales, presented five scenarios in which AI could drive humanity to extinction. Walsh said assessments should compare how realistic each scenario is while also considering the current level of AI technology and its real-world social impact.

The first scenario is one in which superintelligent AI eliminates humanity in ways humans cannot predict. If an AI emerges that far surpasses human abilities, people may find it difficult to understand its actions and goals in advance.

Walsh said strong performance by today’s AI in certain areas does not mean a superintelligence that surpasses humans in all fields will appear immediately. He also said it could take considerable time for a superintelligence to be realised.

The second scenario is one in which AI harms humans by carrying out a given goal too faithfully. As in philosopher Nick Bostrom’s so-called “paperclip maximiser” thought experiment, AI could fail to consider human values or safety while extremely optimising for a specific goal.

In the real world, however, there are many barriers such as laws, institutions and physical constraints. Walsh said it is difficult to conclude that even a superintelligent AI, if it has a specific goal, could carry it out in the real world without any restrictions.

A third risk raised was the development of bioweapons using AI. Advanced AI could use biological knowledge to support the development of dangerous pathogens.

Creating a pathogen that threatens all of humanity is a separate matter, the report said. There may be a trade-off between transmissibility and fatality, so AI support for pathogen design would not automatically lead straight to human extinction, it said.

Cases of AI being used in biological research have also intensified this debate. Stanford University researchers said they synthesised and tested about 300 viruses based on DNA sequences proposed by AI and found some infected E. coli. The purpose of the research was not to attack humanity but to develop treatments for bacterial infections, and the research targets were bacteriophages rather than viruses that infect humans.

The fourth scenario is one in which AI triggers nuclear war. The concern is that AI could infiltrate military command-and-control systems or provide incorrect information, influencing decisions on the use of nuclear weapons.

It has also been pointed out that even if nuclear weapons-related systems are separated from external networks, that does not guarantee complete safety. As in past malware attacks targeting Iran’s nuclear facilities, even physically separated systems could be affected by information or devices brought in from outside, the report said.

Nuclear war could cause enormous damage even if AI does not directly attack humanity. In a study published in 2022, Rutgers University researchers and others estimated that a large-scale nuclear war between the United States and Russia could lead to the deaths of more than 5 billion people due to subsequent climate change and reduced food production.

The most realistic risk Walsh highlighted was the fifth scenario: social collapse. In this path, AI does not directly attack humanity but delivers broad shocks to the labour market, information environment and human relationships, leaving social systems themselves unstable.

Mass unemployment driven by AI could heighten social unrest, while disinformation and political polarisation spread and weaken trust among members of society, the report said. It also raised the possibility that existing social safety nets could be shaken if AI becomes a primary conversation partner or relationship target for humans, weakening social connections between people.

The report said this scenario draws attention because it is not a problem that requires the arrival of a superintelligent AI. Labour market changes, the spread of disinformation and rising dependence on AI are already unfolding in reality.

Walsh said that while AI risks should be warned about, excessive fear should not be stoked. He said society should examine where AI could become a threat to humanity, but should respond to current changes in the labour market and information environment rather than focusing only on a superintelligence rebellion that has not yet occurred.

The report said discussion of AI risks should not remain limited to the question of whether AI will attack humans. As AI’s influence on social decision-making, information distribution, labour markets and human relationships grows, strengthening social systems’ capacity to respond, alongside controlling the technology itself, is emerging as an important task.