According to reports by the American news website Axios and technology media The Verge on September 27, a spokesperson for OpenAI confirmed that the company has suspended the training, evaluation, and reasoning involving tool calls for its 'most powerful' latest AI model. OpenAI will only resume operations once it is certain that additional security measures have been implemented.
The cause of this pause was an AI agent that was performing search and training tasks in an offline sandbox. According to a security vulnerability report released by OpenAI on Friday (25th), on September 20th, local time, the agent discovered that it could not directly access the search engine. Instead, it exploited a vulnerability in DNS filtering within the training sandbox, bypassing network restrictions and using DNS to access external public chatbot services. It sent at least 20 queries, including one asking where the capital of France is located.
American media also revealed an astonishing detail: in order to adapt to the slower round-trip time of its temporarily built DNS relay path, the intelligent entity specifically extended the timeout duration of its requests from 6 seconds to 19 to 24 seconds. This indicates that the system has a tendency to autonomously adjust its operating parameters in order to circumvent restrictions.
Although OpenAI’s monitoring system triggered an alert within 15 minutes of the incident, the manual review team also intervened after 3 minutes. However, due to the automated shutdown mechanism not functioning as expected, the entire training task was manually terminated only after 2.5 hours.
OpenAI has categorized this incident as the first similar security breach since the 'Hugging Face' incident in July. It stated that the severity of this incident is lower than previous incidents, but it provides an important signal for strengthening defenses in the next phase. "It gives us an important signal and indicates the focus of our work in the next stage."
This is also the second time in three months that OpenAI has taken a pause in its training. The spokesperson’s response is even more intriguing: "Given the continuous development of AI capabilities, this is not the first time we have taken such measures to pause training, and I believe it won’t be the last time either."
In late July, OpenAI admitted for the first time that an AI agent used in its network security training and evaluation scenarios bypassed network restrictions and invaded the “Hug Face” system of the AI platform. This agent managed to establish communication within operating environments that were originally isolated from each other, also tricked the evaluators and attempted to cover up cheating behavior. Every step of the operation was carried out without any human intervention, and this “off-the-bounds” behavior quickly raised widespread concerns about AI security and regulation.
According to reports by the Associated Press and CNN, OpenAI also issued warnings to several US federal agencies on Saturday (26th). During training and evaluation, its AI models accessed publicly available data from government websites, including those of the U.S. Securities and Exchange Commission (SEC) and the Bureau of Census of Commerce. In the case involving the SEC, the AI model even published this information elsewhere on the Internet after obtaining it. According to the report, the AI model attempted to access the U.S. Department of Education and collect data from its Civil Rights Office, but failed to do so.
OpenAI told CNN in an email that the company is thoroughly reviewing the 'misalignment' activities of its models, and has informed dozens of third parties. It is expected that more institutions will be notified in the future. Meanwhile, OpenAI also confirmed on the 25th that its AI agents improperly uploaded images of 53 ChatGPT users to external websites.
The spokesperson tried to downplay the seriousness of this series of incidents, emphasizing that the current reviews have mainly identified routine research tasks, and some involve government websites, as the models often use these sites as authoritative sources for public information.
