[DigitalToday reporter Chi-gyu Hwang] "We cannot outsource responsibility for what AI does on our behalf. Assurances from model suppliers do not reduce that responsibility. The most trustworthy superintelligence system is not the system that uses the model you trust most. It is the system that makes you trust the model least."

A post Satya Nadella (사티아 나델라), CEO of Microsoft, recently shared on social media platform X (Twitter) under the title "Models as Insider Risks in the Super Intelligence Era" has become a hot topic in the tech sector.

His post argues that in the era of superintelligence, trust structures need to be rebuilt.

He wrote that over the past decades, companies adopting software could trace problems down to specific code paths. With today’s superintelligence systems, that kind of mechanical understanding is not possible. Frontier models that power them outperform conventional software, but it is not possible to determine which training data or weight settings produced a particular output. Even so, companies are granting complex agent systems and models access to the most sensitive data and letting them handle core tasks, Nadella said.

Nadella said, "We cannot outsource responsibility for what AI does on our behalf. Assurances from model suppliers do not reduce that responsibility." He stressed that companies should not treat superintelligence as a black box and merely choose whether to accept its recommendations, answers or actions. He said they must build systems that observe actions, test limits and can confine actions at any time. He was calling for separating the supply of intelligence from the authority to wield it.

He proposed setting aside hard problems such as alignment — getting AI to act in line with human intent — and taking an engineering-first approach to isolation and governance. He said models whose results can vary each time must be supplemented with deterministic system design, human control and reliable operational procedures. He also argued that if existing standards are insufficient, industry standards should be established anew.

One way to do that, he said, is to treat both closed and open-weight models as insider risks — security threats that can be caused by people inside an organisation.

Nadella said it is not because models have malicious intent. Anyone capable enough to access critical systems can make mistakes or be hacked, and isolation and control structures must account for that, he said. "We have already learned a lot about how to handle powerful actors inside companies. This is not new," he said. He said companies are beginning to apply practices refined over decades — such as identity verification, privilege restriction, activity logging and setting isolation boundaries — to superintelligence inside enterprises.

As a starting point, Nadella called for disclosure of the model’s chain of thought (CoT). He described it as a non-negotiable condition and said, "Neuralese — a model’s internal language that humans cannot read — cannot be an excuse for making reasoning opaque."

CoT disclosure alone is neither sufficient nor reliable, because it is not yet known how to ensure model outputs are always factual and transparent.

Another approach is to have models aggressively test and validate one another. Nadella said this method should be used, but he also pointed to its limits. The structure could become an opaque model inside an opaque orchestration layer, watched by yet another opaque model. That would amount to a black box inside a black box.

For that reason, Nadella said control mechanisms that determine what a model can access and what actions it can take must sit outside the model. The idea is based on information security principles from the 1970s that say a program must not be able to bypass or modify the mechanisms that limit its authority. Applied to AI, he said, the approach separates a harness that coordinates a model’s work and an action space that defines what a model can do, and places control and safety mechanisms outside.

Nadella also presented seven design principles centred on observability.

The first is model diversity. Nadella said, "No model should be solely responsible for critical outcomes or self-validate its own work."

The second is to observe everything. He said, "Meaningful model actions must leave evidence that humans can read and that cannot be tampered with." He added, "If you cannot observe it, you cannot trust it. You must be able to reproduce how an outcome was produced without relying on the model’s statements."

The third is verifiability. He said, "You must continuously test the entire system, including not only successful tasks but also failures, attacks, exceptions and system changes."

The fourth is independent control. He said, "An organisation must be able to independently determine the scope of a model’s access and actions."

The fifth is independent audit. He said, "Verification must be separated from the intelligence being verified. No model should hold both evidence of system actions and evidence to judge whether those actions align with original intent."

The sixth is isolation. The core is to assume the model has already been breached and confine it from the start. Citing an emergency brake as an example, he said, "An authorised person must be able to stop or shut down an operating model at any time. As models become more advanced, more advanced isolation technologies will be needed, and they must be standardised."

The seventh is disclosure of incidents. He said, "If a system fails or is breached, you must notify affected parties in time and share with the entire industry what went wrong, which controls failed, and how to prevent recurrence. This must include implementation details that change agent behaviour during execution."