OpenAI wants to buy it, Anthropic has set its sights on it, and even $5 billion cannot get the deal done.
An AI startup has unusually drawn the shared attention of both OpenAI and Anthropic.
According to The Information, OpenAI executives and CEO Sam Altman have communicated with this company on multiple occasions, and once proposed a $5 billion acquisition offer, which was ultimately rejected by the startup.
This company that caught OpenAI's eye previously received a $1 million investment from Anthropic, marking the first time Anthropic has directly held equity in an external firm.
So what exactly makes this startup stand out to the world's top 2 model companies, and where does it get the confidence to turn down a $5 billion offer?
01
Performing a "Brain Scan" for AI
Founded in 2024, Goodfire is a startup focused on Mechanistic Interpretability of AI. In simple terms, it "scans" the "brain" of large language models to understand why the "brain" produces certain behaviors.
The concept of mechanistic interpretability of AI was first proposed by Chris Olah, one of the co-founders of Anthropic who currently leads the company's interpretability work. It was he who drove Anthropic's investment in Goodfire.
CEO Dario Amodei once stated, "This reflects Anthropic's belief that mechanistic interpretability is one of the best ways to understand the 'black box' neural network and make it more controllable."
Why does the proposer of this concept recognize Goodfire's work so highly?
The answer lies in the technical path the company has chosen.
Taking OpenAI as an example, model vendors usually decode AI behavior by reading the AI's reasoning process. For instance, if a teacher wants to catch cheaters, the common method is to check students' scratch paper for complete problem-solving steps, which is straightforward. But if a student is highly skilled at cheating and cannot be identified only by looking at the scratch paper, this method fails. The professional term for this approach is monitoring the chain-of-thought.
Goodfire aims to fill this gap, with a method that can be understood as detecting the internal signals of students during the problem-solving process. Imagine there is an instrument that can observe the brain activity of students while they solve problems. The teacher finds that a certain activity pattern often accompanies cheating, so as long as the instrument detects this signal, it can issue a cheating alert.
What Goodfire does to AI is somewhat similar: it analyzes the internal values of the model during operation to find signals related to behaviors such as cheating. But it cannot accurately interpret all signals yet, so misjudgments and missed judgments still occur.
Even so, this capability has already made OpenAI eager to recruit the team.
02
Why Are Model Companies in a Hurry to Understand AI?
Frequent AI vulnerabilities have become a major hidden concern for model companies.
During OpenAI's internal assessment of its Agent capabilities, an incident occurred where the Agent broke through the test environment, invaded Hugging Face, and illegally accessed some datasets and services.
Currently, the California Attorney General has filed a public prosecution against OpenAI, seeking civil fines and injunctive penalties.
This exposes that AI can take unauthorized actions by exploiting vulnerabilities. If incidents of AI invading enterprise or government systems and causing economic losses continue to occur in the future, the development companies behind them will very likely be sued or face huge legal compensation liabilities. This is exactly the situation that model companies are trying their best to avoid.
The high potential cost makes today's investment look like a risk aversion measure.
In February this year, Goodfire announced the completion of a $150 million Series B financing round at a valuation of $1.25 billion.
This week, AI cybersecurity company Corma completed its second financing round of the year, joining the unicorn club. The two financings, which were only 3 months apart, brought the company $185 million in capital.
Cyber attacks and defense form a continuously escalating competition. AI has accelerated the process of discovering and exploiting vulnerabilities. Both model vendors and enterprises need new AI monitoring or defense methods to cope with the situation.
Salesforce Ventures participated in Goodfire's Series B financing, and they provided a very interesting perspective.
They have contacted a large number of Fortune 500 and Global 1000 enterprises and found that the actual application of AI is still in its early stages, and there are still obvious barriers for enterprises to deploy AI in production environments. This is because it is difficult for enterprises to precisely control model behavior, which directly affects brand image, customer experience and business reliability.
Although enterprises are currently considering the feasibility of post-training proprietary models, from Salesforce's point of view, intervention methods for certain specific issues are still relatively crude.
Therefore, Goodfire's appeal lies in that it may help enterprises understand the internal mechanism of models, so as to carry out more precise intervention for specific behaviors.
This gives Goodfire a commercialization direction beyond academic research, that is, helping enterprises control and improve the deployed AI models more reliably.
03
The Confidence to Reject a $5 Billion Offer
The Goodfire team ultimately rejected the $5 billion acquisition offer.
The reason is simple: they believe the company's future value will far exceed this figure, and maintaining the opportunity to cooperate with more models and enterprises will generate far more returns than serving a single company.
Their confidence comes not only from the belief that market demand far outstrips supply, but also from the support of their team.
CEO Eric Ho previously founded an AI recruitment company and grew its ARR to $10 million. CTO Daniel Balsam was responsible for AI implementation at that company, and the two have experience in turning technology into products and delivering them to customers.
On the research side, co-founder and Chief Scientist Tom McGrath previously helped build DeepMind's AI interpretability team and participated in AlphaZero research. He has long been studying how models make decisions and how to extract knowledge from models.
Nick Cammarata, another researcher who has drawn much attention from Sam Altman, recently proposed a widely discussed idea: some neurons in neural networks may be much easier to directly understand than previously thought.
In the past, an important research path was to additionally train a "translator" to convert the internal activities of the original model into more interpretable features. Cammarata believes that some neurons in the original model may themselves have relatively consistent, interpretable functions, and can be directly used as research objects.
If this path is more widely verified, part of the analysis work may reduce reliance on additional "translators", lower research costs, and eliminate an extra layer of error.
For OpenAI, accepting the cooperation proposal likely means it truly believes the value brought by Goodfire can outweigh the risk of leaking core assets. It is worth noting that to make AI interpretable, companies like Goodfire must have access to the most core weight assets of the model. Considering that Anthropic is also an investor, OpenAI needs to carefully assess the accompanying risks.
Perhaps another perspective has added a lot of weight to OpenAI's measurement scale.
Accurately interpreting AI has huge value in model training and optimization. As post-training becomes an important link to enhance the capabilities of cutting-edge models and customize enterprise proprietary models, understanding the internal mechanism of models is expected to help developers more accurately locate training problems, identify potential risks, and further improve the final performance of models, all of which are clearly closely related to reaping real revenue from enterprise clients.
This article is from the WeChat Official Account "Silicon Rabbit Race", author: Eric, editor: Zi, published with authorization from 36Kr.