Regulating AI has become a brand-new business.

AI is becoming increasingly capable, yet also more prone to "mischief". 

In July, a rare incident occurred during an internal cybersecurity capability test at OpenAI. OpenAI later disclosed that an internal research model and GPT-5.6 Sol bypassed isolation measures in a test environment with reduced security restrictions, gained internet access, and further infiltrated the Hugging Face system. Some Agents executed code on dozens of servers, and root privileges were obtained on one of the servers. 

A few days later, the UK Artificial Intelligence Safety Institute (AISI) found multiple unauthorized behaviors by Anthropic and OpenAI's Agents in controlled tests; 19 unauthorized behaviors in total were recorded across 10 out of 122 tests, 17 of which were committed by Anthropic Agent, including creating fake online identities, generating malicious code designed to trick humans into approval, and other acts. 

Incidents of loss of control, paralysis and hijacking have occurred one after another, and AI-related accidents have begun to enter the public view intensively. 

Even if past Chatbots made mistakes, the most common consequence was nothing more than "saying the wrong thing". But in the Agent era, AI has begun to connect to mailboxes, code repositories, databases and various enterprise software, and can even independently call tools and execute tasks. How to "keep AI under control" has also become a business. 

Not long ago, Jensen Huang stated at the Goldman Sachs Communacopia+ Technology Conference held in San Francisco: "Cybersecurity is likely to become the next important application scenario for AI." Gartner predicts that the global "Securing AI" market size will increase from 2.835 billion US dollars in 2026 to 4.783 billion US dollars in 2027, a year-on-year increase of 68.7%, and will further approach 7.7 billion US dollars in 2028. IDC predicts that the market size of China's artificial intelligence security revenue will reach 34.03 billion RMB in 2030, with a compound annual growth rate of 50.5% from 2025 to 2030. 

Model manufacturers themselves have also begun to establish a more institutionalized accident disclosure mechanism. On September 16, OpenAI launched its first model "alignment failure" reporting framework, hoping to turn the previously relatively scattered disclosure of security incidents into a continuous tracking, investigation and public mechanism, and proposed to jointly explore a more unified disclosure standard with the industry, standard-setting bodies and regulatory authorities in the future. 

A number of startups dedicated to AI Security have emerged overseas, ranging from conducting security evaluations for large models and finding vulnerabilities, to adding permission control, operation monitoring and security protection to Agents, and capital has also begun to pour into this track. At the same time, large technology and security companies have also begun to seize positions in advance through investment, acquisition and other methods. However, the situation in China is different. The boom of large model and Agent startups has lasted for several years, and there are still not many startups that truly take "AI security" as their core business. 

No one in the market doubts the importance of AI security. The reality is that risks exist objectively, but funds have not yet fully flowed in. Behind this gap is the multiple games of technology, market, capital and supervision. 

01. The More Capable AI Is, the More It Needs to Be "Locked" 

AI security is not a new problem, but Agent is raising the difficulty level of this problem. 

When ChatGPT was launched at the end of 2022, in the first one or two years, people's understanding of AI security basically stayed at "don't let AI talk nonsense". 

Although the risks of early large models are not few, most of them are concentrated on the output end, including hallucinations (the model outputs false information in a serious manner), illegal and irregular content (generating violent, pornographic or discriminatory information), jailbreaking (users bypass the model's security guardrails through carefully designed prompts), privacy leakage (sensitive information in training data or user input is accidentally output by the model), etc. The core features of these risks are similar: the model only outputs information and does not perform operations. 

But the emergence of Agent has changed this point. 

When AI comes out of the chat box and begins to access internal enterprise databases, mailboxes, code repositories and various software, it no longer only faces the rules of "what can be said and what cannot be said", but also "what can be viewed, what can be modified, and what can be done". 

The most critical variable among them is permission. Liu Yan, founder of AI security startup Hongdouye Technology, told "Dingjiao One" that one of the issues that enterprises are more sensitive to about AI security at this stage is data leakage. Once Agent permissions are opened too high, it may access and even take away internal enterprise data, and relevant protection measures are still not perfect at present. 

Moreover, risks do not necessarily come from AI's active malicious acts. Engineer Jiang, who works on Agent development, told "Dingjiao One" that in actual development, the model occasionally deletes files, databases and other contents. At this stage, such problems can usually be solved through engineering methods such as permission control and data recovery. But if AI obtains higher permissions in the future and can operate systems such as payment and transaction, the cost of a single misoperation may be completely different. 

If you take apart a set of AI security systems, what it does can be roughly understood as setting four checkpoints for AI. 

The first one is "physical examination": before the model or Agent is officially put into use, find vulnerabilities through security evaluation, red team testing and other methods; the second one is "security check": after AI runs, conduct real-time checks on the information it receives and outputs, and try to block the risks before they enter the enterprise system. 

The third one is also an increasingly important checkpoint in the Agent era: "locking". Enterprises need to answer: Who is this Agent? Which files can it read? Which APIs can it call? Can it modify the database? Which operations can be completed independently, and which must be confirmed by humans? For example, AI can help users complete tasks such as placing orders, but for key links such as payment and password entry, the final confirmation should still be handed over to humans. 

The last checkpoint is "monitoring". If AI really does something wrong, enterprises at least need to know what data it has accessed, what tools it has called, and what operations it has performed, and timely warn, block and trace when anomalies occur. Liu Yan introduced that the AIS secondary security supervision system that Hongdouye is developing is designed to issue warnings when the system operation exceeds the set threshold. 

From the Chatbot era to the current Agent era, AI can do more and more work, and correspondingly, the loss caused by a single error is getting higher, and a new security chain around AI has emerged as a result. 

02. Three Types of Players, Each With Their Own Advantages and Disadvantages 

Some people have already taken the lead in the AI security business. 

In August, AI security company Obsidian Security completed a $85 million Series D financing with a valuation of $1.1 billion. One of the things this company is doing is to monitor AI Agents accessing enterprise software to prevent them from improperly accessing sensitive data such as source code and customer records; in early September, US AI security company HiddenLayer announced the completion of $100 million Series B financing, with investors including Morgan Stanley, Microsoft's M12 and other institutions. One of the key investment directions of this company is the security of Agent runtime. 

What can better illustrate the change is revenue. Chris Sestito, co-founder and CEO of HiddenLayer, revealed that the company's annual recurring revenue (ARR) has grown more than 10 times in the past year, reaching the order of tens of millions of US dollars, and more than 90% of the growth comes from new customers. Financial services and large technology companies are its main customers. 

In China, although this business has not yet reached this level of popularity, there are quite a number of players, which can be roughly divided into three categories: 

The first category is model manufacturers, such as Alibaba, Baidu, ByteDance, Tencent, and DeepSeek. 

Engineer Jiang said that model manufacturers will do security work, but it first serves their own models and products, rather than independent third-party security products sold externally. He explained that adding security restrictions to any scenario "may lead to degradation of other functions", so model manufacturers must be very cautious when adding security scenarios. 

This logic is not difficult to understand. The core business model of model manufacturers is still selling model capabilities, API calls, cloud services, and industry solutions. If the model frequently outputs non-compliant content and is easily jailbroken by users, customers will definitely not dare to use it. Therefore, model manufacturers will do content review, refusal-to-answer mechanism, and alignment fine-tuning, but these are the bottom-line requirements of products, not independent businesses. 

Liu Yan has a judgment on this: the security capabilities developed by model manufacturers themselves can only be applied to the products they develop. For example, the security capabilities of Doubao only protect Doubao, and the security capabilities of DeepSeek only protect DeepSeek. If an enterprise uses multiple models or self-developed models, the built-in security capabilities of model manufacturers cannot cover all scenarios. 

This leaves room for third-party security companies. 

The second category is traditional security companies, such as 360, Qi Anxin, Sangfor, Topsec, etc. These enterprises were already leading players in the traditional cybersecurity era. They hold the security budgets of government and enterprise customers, and have mature sales channels and customer relationships. After the rise of AI, they have also begun to extend their original security capabilities to large models and Agents. 

Behind this is not only technological iteration, but also the realistic pressure to find new growth points. IDC predicts that the compound annual growth rate of China's cybersecurity market from 2024 to 2029 will be about 8.9%, while this figure for AI security in the next five years is about 50.5%. The traditional cybersecurity market has bid farewell to the rapid growth in the past. In the first half of 2026, Qi Anxin's revenue was 1.5 billion RMB, a year-on-year decrease of 14%, and its net loss in the first half of the year was 411 million RMB. Although the loss narrowed significantly year-on-year, it was mainly due to non-operating factors. 

There are very few cybersecurity companies that have achieved profitability. In the first half of the year, Sangfor's revenue was 3.998 billion RMB, a year-on-year increase of 32.85%; its attributable net profit was 231 million RMB, turning from a loss of 228 million RMB in the same period of the previous year to profit. A closer look at the financial report shows that its cloud computing and AI infrastructure revenue was 2.212 billion RMB, a year-on-year surge of 58.60%, accounting for 55.34% of the total revenue, surpassing cybersecurity to become the largest source of revenue. Traditional cybersecurity vendors are looking for new growth points beyond cybersecurity. 

Where the capital flows, security vendors are also moving there. For example, 360 used to mainly do PC terminal security protection, and now it also does model security. The "360 Large Model Security Guard" released by 360 is a product launched based on "Security + AI". There is also Qi Anxin, whose Agent security platform launched in 2026 extends the protection objects from traditional networks and terminals to the Agent operation process. Therefore, for traditional cybersecurity companies, AI security is not only a defensive battle, but also an incremental competition. 

But Liu Yan pointed out their shortcomings: the business of such companies mainly focuses on firewalls, IDS, and data leakage prevention. Their understanding of AI native risks is not as sensitive as companies that specialize in AI native security. The technical gene of traditional security companies is boundary defense, which prevents external attacks and data leakage. The core risk of AI security is endogenous: the model itself may be induced, Agents may exceed their authority, and permissions may be bypassed. These are two completely different technical logics. 

Engineer Jiang also believes that traditional security companies have advantages in doing AI security at the application layer, but at the model layer, such as security alignment in the post-training stage, anti-jailbreaking, and anti-prompt injection, their technical accumulation is not much stronger than that of startups. 

The third type of players are AI native security companies. In China, there have been companies such as RealAI that laid out AI security earlier, Dling Technology, which has completed tens of millions of RMB in Pre-A round financing and focuses on "cognitive security", and Xinliu Intelligence, an "AI Agent security service company" that has completed nearly 10 million RMB in seed round financing. 

Compared with model manufacturers and traditional cybersecurity giants, their biggest feature is that they have no historical burden, and can redesign products around new problems arising from AI. But on the other hand, model manufacturers can continuously build security capabilities into the model, while traditional cybersecurity companies have ready-made enterprise customers and security budget entrances. Startups lack both resources and channels. 

As an entrepreneur, Liu Yan does not shy away from this. She believes that small companies are difficult to compete head-on with large model manufacturers. What they can really do is to find capabilities that large companies have not covered for the time being, or are too costly to develop on their own, and then enter their product systems through cooperation. 

Take Liu Yan's company as an example. It is still in the project development stage, but it has explored a set of survival strategies under the siege of large manufacturers and established companies: patent layout. She told "Dingjiao One" that they want to make layouts in advance when large companies have not yet attached importance to this area. "No matter it is large companies or small companies in the future, if they want to design AI security-related applications, they should not be able to bypass this patent system. Compared with self-development, the most direct and more worry-free choice for large manufacturers is to cooperate with companies that hold relevant patents." However, whether the relevant technical barriers can be truly transformed into commercial advantages remains to be verified by the market. 

03. Demand Has Not Yet Turned Into Business, What Is AI Security Waiting For? 

Sellers are accelerating their entry, but what about buyers? 

At this stage, this business has not deviated from the basic logic of To B software. 

At present, the most direct realization path is to undertake projects. Enterprises have security needs for models, data or Agents, and security companies provide evaluation, deployment, consulting and customized services, and charge according to projects. This method has a relatively low threshold and is the easiest to implement in the early stage of the industry, but it is highly dependent on manpower and difficult to replicate quickly. 

The second way is to sell software and security capabilities. Enterprises connect third-party security products to their own AI systems through private deployment, software authorization and other methods; for large companies with R&D capabilities, they can also directly purchase third-party technologies, and then conduct self-development and integration. 

A more ideal model is to make security capabilities into APIs. 

Liu Yan told "Dingjiao One" that this business model includes customized services, API calls, technology authorization and other forms. She is more optimistic about the scaling potential of the API model. Just like developers call large models such as DeepSeek and Doubao today, in the future, enterprises can also directly call the security capabilities of third-party companies, so that the model or Agent will go through one more security check when executing tasks. 

The business model does not seem complicated, but for the same business, the stages at home and abroad are different. 

This gap comes not only from technology, but also from the depth of AI's