California has unveiled proposed new AI safety rules that include giving the state government the power to halt high-risk AI models in an emergency. [Photo: Shutterstock]

California is pursuing measures to forcefully halt AI models that fall out of control and to regulate the process in which AI develops stronger successor models on its own. It aims to strengthen the government's power to intervene in case advances in AI technology exceed human oversight and safety verification capabilities.

On Nov. 19, Taiwan media outlet IT Home reported that California Governor Gavin Newsom (개빈 뉴섬) recently released a report titled "AI Safety in 2026 and Beyond" that includes those measures.

The report was written by the California Government Operations Agency on Nov. 16 and officially announced on Nov. 18. Its key points are emergency AI shutdown authority, regulation of recursive self-improvement (RSI) and the creation of an independent safety verification system.

It first proposes requiring large advanced AI developers to establish emergency response plans. The plans would allow phased steps including halting AI agents and revoking system access rights, isolating computing resources and suspending model operations.

The state plans to allow the state government to order an emergency shutdown if an AI model is at imminent risk of causing a disaster-level incident or going out of control and the developer is unable to respond. It would order shutdown measures only to the minimum extent and only when no other means can prevent the risk. Companies could challenge the order in court.

Recursive self-improvement, in which AI develops stronger successor models without human intervention, is also included as a regulatory target.

Developers must report the level of automation in AI research and development and the computing resources used for it. If the pace of automation outstrips safety verification capabilities and catastrophic risks rise, the state government could limit the share of computing resources used for such activity.

The report did not recommend immediately imposing caps on computing resources. It said regulatory authority should be exercised only when risks exceed a certain level.

It also seeks to strengthen outside monitoring. Large AI developers must undergo continuous assessments by independent verification bodies from model training through deployment and must meet government-set standards on safety, security and transparency.

It proposed shortening the deadline for reporting major AI incidents to 48 hours from the current 15 days. For urgent disaster risks, it would keep the existing reporting standard of 24 hours, and it would also include loss of model control and attempts to steal model weights as reportable incidents.

California introduced the AI safety law SB 53 last year and passed SB 813 and AB 1405 this year to establish a registration system for independent verification bodies and auditors.

The report's emergency shutdown authority and limits on computing resources are still at the stage of legislative recommendations. Actual implementation would require separate legislative revisions and institutional adjustments.

Keyword

#California #Gavin Newsom #AI Safety in 2026 and Beyond #RSI #SB 53
Copyright © DigitalToday. All rights reserved. Unauthorized reproduction and redistribution are prohibited.