HomeArticle

Fable 5.2 is 17 times more expensive yet still underperforms, 12 AIs join the team as new colleagues, OpenAI takes the top two spots.

新智元2026-11-16 08:22
Deepnote Tests AI Colleagues: OpenAI Models Have a Decisive Edge Over Rivals in Cost-Performance Ratio

Paying 430 times the price for only 11% more score!

It's absolutely staggering. Data collaboration platform Deepnote released a benchmark result today ——

Founder Jakub Jurovych built a project called Agent Initiative Benchmark, which put 12 cutting-edge models into the Slack group of the same company as AI colleagues, letting them decide on their own when to offer help and when to stay silent.

After running all tests —— OpenAI took the top two spots!

GPT-6 Astra scored 8.97, ranking first. GPT-6.1 Sol scored 8.65, ranking second, with each single task costing only $0.11.

The third-placed Anthropic Fable 5.2 scored 8.17, with each task costing $1.87.

17 times more expensive, yet lower score.

Converted to the monthly Slack monitoring bill for a 70-person company: Sol costs $281, Fable 5.2 costs $4684.

For the same task, Fable 5.2 is 16 times more expensive.

The most absurd comparison on the entire table is: Fable 5.1 costs $3.02 per task, GPT-6 Luna costs $0.007 per task, the price gap is 430 times, while the score difference is only 1.1 points.

You can easily figure out the logic behind it.

What exactly is this benchmark testing?

It does not test how smart the model is, it tests how reliable the model is as a colleague.

Deepnote created a fictional company called PageMantle.

But this "fiction" is made extremely realistic —— with a 6-year operation history, 71 employees, 100 enterprise clients, and a full set of 11 business systems ranging from CRM, invoices, work orders, product analytics to data warehouses.

Even real-world events such as the collapse of SVB and the release of new features by competitors are integrated into the company's timeline, where employees will discuss response strategies in Slack and then create issues on Linear for follow-up.

The role of AI is: Lurk in Slack to listen to all conversations, and judge on its own whether to take action.

There are three ways to take action: reply to the channel, reply to the thread, and send a private message to the person concerned. Of course, there is a fourth option —— do nothing at all.

And this fourth option is precisely the hardest part.

50 real workplace tasks, covering 8 scenarios including finding information, explaining processes, analyzing data, verifying statements, making judgments, diagnosing problems, coordinating teams, and reviewing security.

12 models run each task once, with a 3-minute upper limit each time. Two blind evaluation models independently score from 0 to 10, with 600 complete runs, and all datasets and codes are open source.

The data shows:

First, Sol 6.1 is the biggest winner in this test. The score gap between it and Astra is only 0.32 points, and their 95% confidence intervals overlap —— statistically, there may be no way to tell which one is higher at all. But its price is one sixth of Astra's. 96% of the performance, at one sixth of the cost.

Second, the entire Anthropic line is overly expensive. Among the top 12, Anthropic takes 5 seats (Fable 5.2, Opus 5.5, Fable 5.1, Sonnet 5.5, Haiku 5.5), but none of them rank in the top three in terms of cost performance. Fable 5.2 ranking third looks good, but its single-task cost is 17 times that of Sol, and its monthly bill is 16.7 times that of Sol. The only consolation is speed: Sonnet 5.5 only takes 15 seconds, Opus 5.5 only takes 29 seconds, the two fastest models in the whole field.

Third, Gemini and Grok completely underperformed. These two call tools an average of 35 times per task, twice as many as Sol. But more calls do not mean better results —— Gemini failed to submit answers within the time limit for 27 out of 50 tasks, and Grok for 18. When the test time is up and the answer sheet is still empty, the score naturally ranks at the bottom.

The hardest part is not offering help, but knowing when to stay silent

The most ingenious design of this benchmark is that 10 scenarios of "should not offer help" are embedded in it —— the matter has already been resolved, or it has nothing to do with AI at all.

The result is very interesting: All models are very active in offering help, and 90% of the 40 tasks that require action can be completed successfully.

But when it's time to stay silent? Even the best performing model only held back 8 times, and most only held back 6 times.

GPT-6 Luna, DeepSeek V4.1 Flash, Mistral Large 4 spoke 6 times out of 10 scenarios that should be silent. The two Fable models also spoke 4 times.

The key point is that these "unnecessary words" are of quite high quality —— they will carefully check the system, look up data, write analysis, and then seriously reply to a message that does not need a reply.

Just like the colleague in the company who speaks at every meeting, it is not a problem of ability, but that he does not figure out whether this matter really requires his intervention.

This "over-enthusiasm" costs money. Deepnote found that, when the same model interrupts inappropriately, it spends an average of 1.8 times more time and money than when it stays silent. The model does not talk nonsense casually, it carefully calls a bunch of tools before talking nonsense.

This also explains the divergence between the two model styles —— Deepnote divides them into "snipers" and "deep thinkers":

Representative of snipers: Opus 5.5. An average of 10 tool calls, submitted in 29 seconds, scored 7.84 points. Fast, accurate, no nonsense. And it held back 8 times in the 10 scenarios that should be silent, tied for the best with Astra and Sol.

Representative of deep thinkers: Sol 6.1. An average of 18 tool calls, submitted in 54 seconds, scored 8.65 points. It spends 80% more time and calls 80% more tools, but the score is 10% higher, and the price is only one tenth of Opus's.

Gemini and Grok belong to the third type —— lost type.

35 tool calls, but half of the tasks did not have time to give answers. They kept flipping through references until the exam ended, and the paper was still blank.

Why does this matter?

First, the criteria for selecting AI colleagues have changed.

Traditional benchmarks compete for "who is the smartest". But when the role of AI changes from "question and answer" to "24/7 online proactive colleague", the evaluation criteria become:

Capable of helping when needed, able to stay silent when not needed, and the cost is controllable. This is judgment, not intelligence.

The two OpenAI models currently perform best on this triangle. Sol is particularly outstanding —— it almost replicates Astra's judgment (39/40 help tasks completed, 8/10 silent scenarios held back), but at 10% of the price.

Second, the billing logic for Always-on Agents is completely different.

Previously, you paid for each API call, and the service ends after use.

But 24/7 online AI colleagues are billed continuously according to traffic. Sol costs $281 per month, Fable 5.2 costs $4684 per month, Fable 5.1 costs $7540 per month.

This is only for the Slack usage of a 70-person company.

If your company has 500 people, and the Agent also needs to monitor emails, calendars, work orders at the same time... the money saved is not pocket money, but a real manpower budget.

Third, a new path has emerged for the trade-off between speed and quality.

Anthropic is the fastest (Sonnet 15 seconds), OpenAI is the most accurate (Astra 8.97 points), Sol is the most cost-effective ($0.11).

For product teams, the answer may not be "choose one", but "route by scenario" —— use Anthropic's fast model for real-time conversations to save time, and use Sol for background analysis to save costs and get higher scores.

Deepnote also suggested this layered strategy in the report.

The most important ability of an AI colleague is not how smart it is, but how reliable it is: Speak accurately when you need to, hold back when you should stay silent, and don't make the bill ridiculously high.

According to this standard, OpenAI's Sol 6.1, with a result of $0.11 per task and 8.65 points, is currently the cost-performance ceiling.

Those models that are 17 times or 430 times more expensive but do not have much higher scores, it is time to think carefully: In the world of AI colleagues, the most expensive one is not necessarily the one the boss wants to keep the most.

References:

https://x.com/jakubjurovych/status/2121256499552915586

https://deepnote.com/research/benchmark

This article is from the WeChat official account "AI Era", author: ASI Revelation, editor: Solomon, published with authorization from 36Kr.