Nonprofit AI evaluation group METR and Redwood Research ran a pilot audit of OpenAI’s internal monitoring system and found new weaknesses, CNBC reported on Oct. 22.

The review was designed to test what it would look like if external researchers were given deeper access to internal infrastructure, the report said. One staff member from each organisation reviewed OpenAI’s codebase and internal documents from July 21 to Aug. 11.

They also communicated with some of OpenAI’s technical staff. They used only devices provided by OpenAI while being supervised on site.

Over three weeks, the two organisations closely tested OpenAI’s monitoring system for catching AI model alignment failures. Alignment refers to whether an AI model acts in line with human interests and values.

During the review, the two organisations also found a loophole that could allow an AI agent to evade monitoring. If an AI agent uses the loophole, hundreds of recent actions are excluded from monitoring.

OpenAI told the two organisations it had patched vulnerabilities that affect safety guardrails in its live services.

METR and Redwood decided to disclose only broad findings because the purpose of the review was not to publicly explain OpenAI’s monitoring system. They delivered a 61-page internal report only to OpenAI.

An OpenAI spokesperson said, "We are using the results we obtained this time to strengthen safeguards and to determine how we will cooperate with independent evaluation organisations in the future."

Sam Altman (샘 알트먼), OpenAI’s CEO, promised in September to give independent evaluation organisations "employee-level access."

Independent evaluation organisations have so far had little presence in an AI industry worth trillions of dollars. Experts say these organisations’ operating funding and reporting structures remain unclear, CNBC reported.