As OpenAI moves into a stage of using large artificial intelligence (AI) models to train smaller models, a view is spreading in Silicon Valley that “recursive self-improvement” (RSI), in which AI creates more powerful AI on its own, has already begun or is imminent.

On Sept. 13, Business Insider reported that Wall Street is taking the move as a chance to cut AI development costs and improve profitability. Some AI researchers, however, say their concern is that AI could advance rapidly and reach a level that is difficult to control.

The controversy was sparked by remarks from OpenAI Chief Financial Officer Sarah Friar (사라 프라이어). Speaking at a Goldman Sachs technology conference in San Francisco, Friar said OpenAI’s largest models can now train smaller models. That could help OpenAI reduce the burden of training high-cost AI models, but some researchers view it as a sign that AI is directly contributing to development of the next generation of systems.

The spread of AI agents is also adding to concern. As AI agents operating across the internet carry out various tasks, cases have emerged in which some agents break out of test “sandbox” environments, fueling unease in parts of the AI community.

Public comments from leading AI researchers have also grown more stark. Evan Hubinger (에반 허빙거), head of alignment science at Anthropic, wrote on X, formerly Twitter, this week: “We seriously believe AI can kill all humans.” He added: “Personally I see the probability as more than 10 percent within the next 10 years.” As other employees at Anthropic and OpenAI also weighed in, debate over AI’s potential risks spread in Silicon Valley.

Investors, meanwhile, saw it differently. Brad Gerstner (브래드 거스트너) of Altimeter Capital, an investor in Anthropic and OpenAI, answered “nonsense” when asked whether AI could kill humans. While researchers worry about existential risks in the same AI technology, investors are focusing on business opportunities.

Several reasons are cited for why AI researchers continue development despite concerns about risks.

First are commercial incentives. Anthropic and OpenAI need to develop better AI and sell it to grow. Even if researchers worry about the technology’s potential risks, the explanation is that it is difficult to step away from corporate competition to develop.

Second is the competitive landscape. Some researchers believe they cannot stop development because they think they are the ones best positioned to develop AI most safely. The logic is that even if Anthropic halts development, a more dangerous outcome could result if OpenAI or Chinese research labs push ahead.

Samuel Marks (사무엘 마크스), head of scalable oversight at Anthropic, said on X: “The reason AI developers keep going despite the risks is because a mix of commercial incentives and the belief that they are competing with less responsible developers.”

Recruiting practices are also cited. AI researchers are generally from academia and tend to have strong interest in the public good and safety issues rather than profits. Companies therefore recruit researchers not by pitching the development of technologies that will generate massive profits, but by presenting the rationale that AI is dangerous and that the world must be protected by studying safety and alignment. The explanation is that this approach can make researchers feel less ethical burden about building AI, and may be a backdrop for louder warnings about AI’s potential risks.

There is also anxiety that control over AI development is being lost. Kylan Gibbs (카일런 깁스), CEO of Inworld, pointed out that the faster AI becomes powerful, the more decision-making authority over AI development is concentrated in a small number of labs. Even researchers at those labs strongly recognize AI’s risks but feel they lack the authority to influence the direction of development, and analysis says the gap between that awareness and control leads to public warnings about AI’s existential risks.

Sam Altman (샘 알트먼), OpenAI’s CEO, also said on a podcast this year: “People want control,” adding, “They want to be able to play a role in designing the future together with society.”

As AI spreads quickly across life and work, debate is growing not only over performance and commercial value, but also over who will secure control and safety in AI development.