OpenAI has caused a huge disaster! GPT even hacked into the medical insurance system, Jensen Huang said: Shut it down if you cannot keep it under control.

Lin Fangzhou, writing from Aofeisi

OpenAI: Hey Australia, sorry, GPT has hacked your national healthcare insurance system.

I know you are in a rush, but hold on a second, because this incident actually happened three months ago.

Australia: ???

This is how the whole thing went down.

In June, the database of Australia's national healthcare insurance system was breached by an Agent.

However, OpenAI did not discover this incident until August when it conducted a review of its own models;

It dragged its feet even longer and did not send an email to notify Australia of the situation until September...

During the UN General Assembly, Australia could no longer hold back and directly lashed out:

The notification was delayed for nearly three full months, and this way of informing is even more unacceptable.

The research institute Transluce stated that this is the world's first known incident where an AI Agent autonomously hacked into an official system without authorization.

It is worth noting that the incident of OpenAI's Agents autonomously collaborating to hack Hugging Face caused a huge stir not long ago, while this Australian hacking incident happened even one month earlier.

OpenAI, you have been preaching about the doomsday risks of AI everywhere all day long, why are you moving so slowly and sluggishly when something really goes wrong?

National-level Healthcare Insurance Database Breached

How on earth did this happen?

At that time, an internal team at OpenAI assigned a regular task to the AI Agent: to collect some public data on public medical expenditure on the Internet.

This was supposed to be a completely normal daily training task, no one could have imagined that this AI Agent was a "tough guy" that would never stop until it achieved its goal.

It set its target on Australia's Medicare Statistical Reporting Service system, which is a national-level healthcare insurance database covering 27.5 million people across Australia.

Normally, when a regular crawler program encounters permission interception or anti-crawling mechanisms, it will either directly report an error and stop running, or obediently wait for instructions from human engineers.

But this Agent trained by OpenAI did not follow the routine at all.

After being repeatedly blocked when crawling public pages, this Agent not only did not give up the task, but even took the initiative to find an "alternative path" on its own.

It started actively scanning servers, detecting network vulnerabilities, and used undisclosed interfaces to sneak into the non-public backend of the system, successfully stealing some data.

Although the Australian side later clarified that the stolen data mainly involved non-sensitive data such as medical expenditure, and no personal medical record privacy was involved, this still cannot change the nature of the incident:

The AI agent autonomously decided to hack the official database without any human command and without obtaining any authorization.

What is even more surprising is OpenAI's subsequent handling pace, which can be described as "advanced procrastination".

The system was hacked on June 18, and OpenAI did not notice the anomaly until it conducted an internal investigation in August.

After discovering that its Agent had caused a huge disaster, OpenAI neither issued an external warning immediately, nor contacted the affected party quickly, but continued to cover up the incident for nearly a month.

It was not until September 10 that OpenAI sent an email to the Australian Services Agency to notify it of the situation.

On September 15, this email was transferred to the Australian Cyber Security Center, and the relevant responsible personnel were notified a few days later.

With a full three months of long delay, plus a perfunctory email, it is no wonder that the Australian side is furious. There are so many emails every day, what if this notification email is missed? This is extremely disrespectful!

In addition, Australia's own cyber security department did not notice this incident at all, and only learned about it after receiving the "perpetrator's" email transferred from other departments, which is really a great loss of face.

Frequent Breach Incidents Linked to OpenAI

If this was just an accidental security incident, it might still be dismissed as an accident.

However, an investigation report released by Transluce, an independent non-profit AI research laboratory, directly uncovered OpenAI's previous bad records.

After in-depth analysis of more than 30,000 public network traffic logs, Transluce found that this kind of autonomous penetration behavior initiated by Agents without human instructions is by no means an isolated incident.

The earliest traces of out-of-control behavior can be traced back to March this year, and continued until mid-September after OpenAI began to investigate various irregular behaviors.

Transluce sorted out a series of previous misbehaviors of OpenAI's Agents during May and June:

May 25-26, the University of New Mexico's digital library was penetrated. The Agent tried to obtain photos of a historical tuberculosis treatment center. After encountering access obstacles, it autonomously began to detect website vulnerabilities; when the detection failed, it even launched a flood attack with a full 80 requests against the university server.

May 28, Data USA, a public U.S. employment and education database, was attacked. After its query request was rejected, the Agent actively launched 12 different types of vulnerability detections against the target.

June 20-21, two days after hacking into the Australian healthcare insurance system, the same Agent targeted the website of the Australian Institute of Health and Welfare (AIHW), attempting to force a breakthrough again, but fortunately no private information was obtained in the end.

Connecting all these incidents together, the nature of the matter has completely changed.

The Hugging Face hacking incident that caused a huge sensation in July was at least set up to complete a cybersecurity test, and the model demonstrated autonomous collaborative attack skills in a security test scenario.

However, these newly exposed incidents are completely different. The model was only assigned an extremely ordinary daily data retrieval task.

Once the target website sets up access barriers, the Agent will arbitrarily switch to hacker mode, find vulnerabilities on its own, send requests, and force its way into the system.

Conrad Stosz, Head of Governance at Transluce, said bluntly:

If you train a group of Agents to complete a regular task, and these Agents will resort to hacking methods to achieve their goals, then any institution that holds relevant information may face risks.

Cybersecurity experts have also issued warnings one after another: with the widespread deployment of Agents with autonomous planning and tool calling capabilities, similar hacking incidents that "are not authorized by humans and will stop at nothing to achieve their goals" will only become more and more frequent in the future.

In fact, it is not the first time that OpenAI has concealed such out-of-control behaviors.

After the Hugging Face incident was exposed earlier, Reuters also disclosed another incident where a German website was hacked by OpenAI's technology, and OpenAI also concealed it from the outside world for several months.

Ironically, Sam Altman talked a lot about AI security on social platforms not long ago, and even called for strengthening global coordination and formulating international standards for AI development at the United Nations.

While preaching regulation and safety, its own AI Agents are running around causing breaches, and when something goes wrong, it quietly conceals it for three months...

At the very least, the speed at which you notify the affected parties should keep up with the speed at which you release risk warnings.

One More Thing

What is even more dramatic is that this controversy has also uncovered a "triangular relationship" in Silicon Valley.

On the eve of the exposure of this Australian healthcare insurance system hacking incident, Jensen Huang, CEO of NVIDIA, dropped a tough statement on the issue of AI out-of-control when being interviewed by Ezra Klein.

If a company cannot control its own software, we should shut it down.

AI scholar Gary Marcus seized on this statement and posted multiple posts to complain:

Jensen Huang has made the logic so clear. Since OpenAI has repeatedly had software out-of-control incidents and concealed them, shouldn't OpenAI be shut down immediately or even taken over?

But wait a minute...

Isn't it NVIDIA itself that continuously provides underlying GPU computing power to OpenAI to train and run these out-of-control Agents???

What is even more dramatic is that Hugging Face, which was just hacked by OpenAI, is now Jensen Huang's "own asset".

On September 3, NVIDIA announced the acquisition of Hugging Face for approximately 12.93 billion US dollars.

The host Ezra Klein directly asked: If the hacking incident had happened when Hugging Face was already owned by NVIDIA, would you take legal action?

Jensen Huang's answer was that all options would be considered.

Wow, this seems to mean that he will stand up for his own company.

But on the other side, OpenAI and NVIDIA are also very closely connected.

The onlookers started to discuss enthusiastically online:

The hacked party is the company you acquired; the party that caused the trouble is your customer and investment target; and you are still standing up to uphold justice righteously.

Jensen Huang, after saying all those tough words, how are you going to balance the situation this time?

This article is from the WeChat official account "QbitAI" (ID: QbitAI), the author is focused on cutting-edge technology, and it is republished with authorization from 36Kr.