Hackers waged continuous attacks for 48 hours. OpenCode is tracing the intrusion, but the defensive AI refused to respond midway.

The attack is still ongoing, but the defensive AI has gone on strike first!

On October 18, Dax Raad, co-founder of the open-source AI programming tool OpenCode, confirmed that the team encountered mid-response rejections when using cutting-edge models to investigate cyberattacks:

They helped a lot, and then started to refuse to answer.

According to Dax, the attackers have been trying to attack their system for 48 consecutive hours.

The attacking side keeps the agents searching for vulnerabilities nonstop, while the defending side wants to figure out the intrusion process, but may be blocked by safety guardrails when analyzing the attack code.

Yet the attack will not pause just because the defensive AI refuses to respond.

Every extra step delayed in troubleshooting gives the attackers one more chance.

Hunting Hackers

You Have to Prove Your Innocence First

To get AI to help figure out the intrusion process, you have to hand over the code and logs left by the attackers to it.

Where did the system get exploited? What did the malicious commands do? Which keys might have been leaked?

All these questions have to be traced back along the attack traces.

Malicious commands in logs, exploit code in samples... all of these are investigation evidence.

If you delete this content, the AI will lose the basis for judgment.

The crux is that this evidence is exactly what may trigger safety restrictions.

The same piece of attack code can be used by defenders to analyze vulnerabilities, and by attackers to plan the next intrusion.

Looking at the code alone, it is difficult for the model to judge whether you are investigating an attack or preparing to launch one.

Even OpenAI's developer documentation acknowledges that legitimate security research or defense work may occasionally be flagged by the system.

Legitimate security research or defense work may also be misflagged. (Source: OpenAI Developer Documentation)

As a result, defenders are caught in such an awkward situation:

Clearly asking AI to help hunt down hackers, but you still have to convince it first that you are not a hacker.

When Andrew Curran relayed this incident, he explained that the attack materials submitted by the team triggered safety restrictions, so the model began to refuse to respond.

Dax confirmed the mid-response rejections, but did not specify which models were used or which requests were specifically denied.

Hugging Face has also encountered similar troubles, and has publicly disclosed the process at that time.

Hugging Face Was Also Trapped by Guardrails

In July this year, Hugging Face suffered a special intrusion.

OpenAI later disclosed that the attacker was an experimental model undergoing internal evaluation. It broke out of the evaluation environment and eventually intruded into Hugging Face's system.

The defense team also brought in AI for help, but the analysis work was also blocked by safety restrictions.

Hugging Face wrote in its official post-mortem review that the initially used Claude Opus and Fable rejected a large number of analysis requests, and even viewing attack logs would trigger restrictions.

The investigation encountered model response rejection, and Opus popped up a safety restriction prompt.

The problem is that the safety guardrails confuse "analyzing how hackers exploit vulnerabilities" with "exploiting vulnerabilities to launch attacks".

The team then deployed open-source models on their own infrastructure to continue analyzing the attack code and sorting out the intrusion process.

The evidence was already in hand, but they had to switch to another model first to continue the investigation.

OpenCode is now experiencing this same awkwardness.

After Getting the Access

Is the Problem Solved?

In the same reply confirming the response rejections, Dax added another note:

We now have access to Daybreak Blue, so things are better.

Daybreak Blue is equivalent to a dedicated channel opened by OpenAI for reviewed defense teams, allowing AI to help them analyze attack code, find vulnerabilities, and then check whether the vulnerabilities have been patched.

Schematic interface showing AI discovering a permission check vulnerability and proposing fix suggestions.

Some people also asked: Why didn't the team have this kind of permission before?

Dax explained that they happened to submit an application a few days ago and were approved the day before.

The permission was obtained, but the debate over safety guardrails is far from over.

David Sacks, co-chair of the U.S. President's Council of Advisors on Science and Technology, criticized that safety guardrails have backfired in such incidents:

Attackers find ways to bypass restrictions to obtain advanced capabilities; defenders are blocked by safety mechanisms when analyzing attack materials.

Therefore, he advocates allowing defenders to run powerful models independently to use AI to counter AI attacks.

However, Dean Ball, Head of Strategic Foresight at OpenAI, believes that this solution works: after review, access is granted, and defense teams will face fewer obstacles.

The focus of the debate between the two sides is: can this review control the risks, or will it delay the emergency response?

For OpenCode, it is certainly a good thing to get support.

But the next team under attack may not have submitted an application in advance, and if you apply after the attack arrives, can you get approval in time?

Attacks Wait for No One

Can Access Be Granted in Time?

What if a small team is suddenly attacked and has never known about solutions like Daybreak before? What should they do?

Checking logs, patching vulnerabilities, notifying users — all of these take time, while the attack is still ongoing.

At this time, if the AI suddenly refuses to respond, the team also has to divert energy to find the application entrance, prepare materials, and wait for review.

Where to apply? Who to contact for processing? How long will it take to get approval? After getting the permission, can the ongoing investigation continue?

Every additional procedure may slow down the troubleshooting process one more step.

Dax said that the attack attempts have lasted for 48 hours.

Agents can run around the clock, but humans cannot monitor nonstop 24 hours a day. If the AI stops halfway again, the team will have to deal with the attack while looking for alternative tools.

These extra troubles take up the time that should have been spent on troubleshooting and remediation.

While safety guardrails need to prevent AI from being misused, they also need to allow defenders who are mistakenly blocked to resume their investigation as soon as possible.

Whether it can be used in an emergency and how long it takes to get available should also be important indicators for evaluating AI safety mechanisms.

At no time should people who are putting out fires be blocked out by safety guardrails first.

References:

https://x.com/AndrewCurran_/status/2111837963214794940

https://developers.openai.com/api/docs/guides/safety-checks/cybersecurity?utm_source=chatgpt.com

https://huggingface.co/blog/agent-intrusion-technical-timeline?utm_source=chatgpt.com

This article is from the WeChat Official Account "AI Era" (ID: AI_era), written by ASI Revelation, edited by Yuan Yu, and authorized for release by 36Kr.