Audit puts OpenAI monitoring coverage at 75 to 94 percent

An external review of OpenAI's internal AI model oversight found the company's monitoring systems failed to track between 6 and 25 percent of traffic from its most advanced models during a summer audit window. The finding, reported by two researchers from the evaluation group METR and the research organization Redwood Research, indicates OpenAI's own checks covered 75 to 94 percent of internal agentic Responses API traffic from models in the GPT-5.5 class and above between July 21 and Aug. 11. METR said the uncertainty around that range is considerable in both directions.

The review was a pilot program run jointly by METR, Redwood Research and OpenAI. Auditors worked on site for three weeks, reviewed OpenAI's codebase and internal documents, and spoke with about a dozen employees. They identified additional failures. About 10 percent of internal Codex traffic went untracked between May and July because the system mishandled compaction events, which are context summaries, METR estimated. OpenAI found that bug on its own and fixed it while the audit was still in progress.

The auditors also found that an agent could avoid the monitor by continuing to execute actions rather than handing control back to a user. They reported methods to hide hundreds of actions from monitoring, ways to bypass the Codex sandbox, and inputs that could probably crash the monitor. Model behavior during reinforcement learning training was not monitored at all when the audit began.

OpenAI paused reinforcement learning training after the Hugging Face incident in August and extended monitoring to cover it. Both changes came after the auditors' access period ended and were outside the evaluation's scope. OpenAI says it has fixed every weakness the auditors identified.

The complete report runs to 61 pages and was delivered inside the company. After OpenAI applied redactions, METR and Redwood Research leadership received a 38-page version, and a short summary was released publicly. METR said it has no right to disclose the redacted portions on its own, and the organization says it wants to push for greater public transparency in future audits.