Fable 5.1 outperforms humans yet costs 2.5 times more, NeoCognition has done the math for deploying Agents in enterprises.

There is a growing number of large model leaderboards, but it is not easy to tell what the few-point gaps on these leaderboards actually mean in real-world work scenarios.

In evaluations including Artificial Analysis (AA), Terminal-Bench 4.0 and CursorBench 4.0, the scores of Fable 5.1 and Opus 5 only differ by a few points. However, if they are deployed in the same company, using the same software, learning the same business rules, and processing the same batch of bills consecutively, the result turns out to be 72% versus 36%.

The number of completed tasks differs by twice.

This result comes from the newly released ApprenticeBench by NeoCognition. Unlike common single-capability tests, it places the Agent in a simulated construction company, letting it onboard just like a new employee: read the employee handbook and historical documents, use real enterprise software, and gradually figure out the company's business rules based on feedback from supervisors.

ApprenticeBench homepage link: https://apprenticebench.com/

The Agent did not receive specialized training for this specific job. It needs to learn from the company's historical data and the onboarding guidance process while working. In this evaluation, Fable 5.1 and GPT-6 Astra achieved pass rates of 72% and 68% respectively, both exceeding the 51% pass rate of the best-performing human tester in this round.

Fable 5.1 and Astra show a clear leap in capabilities, outperforming the human testers in this round. Source: ApprenticeBench Blog (https://neocognition.io/blog/apprentice-bench/)

This also brings up a bigger question: if Agents can familiarize themselves with software and learn business rules on their own, can the deployment and adaptation work that currently requires FDE (Field Deployment Engineers) to carry out for each client be completed by Agents themselves in the future?

Why can an apparently ordinary accounting job create such a huge performance gap? What do these gaps mean for teams that are training next-generation models?

The company's rules

You only get to know them after joining the company

Knowledge of accounting can be learned in advance, but no general model can or should know how a specific company keeps its accounts and what policy adjustments it has made recently. An Agent needs to go through the same onboarding learning process that a new human employee has to complete.

This is the core design idea of ApprenticeBench: the Agent is placed in a simulated construction company, and uses the same software (Odoo) as human employees to process accounts payable. When onboarding, it is given an employee handbook, software tutorials and six months of historical bills, then processes 100 new bills consecutively. In the first month, the supervisor provides one-on-one guidance for each bill, and later only gives sparse feedback at the end of each month.

From reviewing historical records to gradually handling business independently, the Agent goes through a learning process similar to that of a new employee. Source: ApprenticeBench Blog

When you actually get down to the work, you will find that all the troubles are hidden in the details —— the devil is in the details

In the accounts payable work of a construction company, typos in unit labels, expired insurance, and unclear handovers from previous employees are not uncommon. Multiple issues often appear on the same bill, requiring several rounds of back-and-forth communication. However, existing benchmarks omit all these troubles in real work. Meanwhile, there are many rules that are never explicitly written down. The same expense description may be categorized into different cost codes depending on the purpose of the expense. The Agent needs to figure this out on its own from historical bills, instead of guessing the answer only based on the name.

All these troubles are incorporated into ApprenticeBench. The 100 tasks have been independently verified by two professionals with over 30 years of relevant experience in total, confirming that these scenarios do occur in real work and can be resolved using the information provided in the benchmark.

Emilie F., a construction finance director with 20 years of experience, said: "These scenarios are very realistic. For example, it is far more common than you think that a project manager loses something without realizing it. Before participating in this project, I never thought AI could handle this type of work. If it can deal with these emails, back-and-forth communications and cross-checks, I believe it will become a real helper for the accounting department." Austen K., a business operator with 10 years of experience in construction and project management, said: "These scenarios perfectly match the situations that a construction finance director will encounter. Currently I handle all this work by myself. If AI can take over these tasks, I will have more time to take on more projects and expand my business. For a company of my size, this can easily bring six-figure US dollar value every year. If AI can learn these skills and get high scores in the test, I would be very willing to let it try in my own company."

This is what the team calls "ecological validity". It is also related to a more scalable deployment method: can enterprises use their existing software, materials and onboarding guidance processes to let Agents actually learn to work just like new human employees?

Real-world work

Is where the real gaps between models emerge

When software operations are tested separately, the performance of two models may be close. But to complete a full set of work, the Agent also needs to learn the company's conventions from historical records, adjust its practices according to supervisor feedback and policy changes, and communicate with suppliers to supplement missing information. These challenges also interact with each other. If you are not proficient in using the software, you may not be able to accurately retrieve relevant historical records, and thus cannot understand the company's rules clearly. If you fail to correct the problems pointed out by the supervisor, you will continue to make mistakes in subsequent bills. When all these challenges are combined, the gaps that are not obvious in individual tests will accumulate. The few-point difference between Fable 5.1 and Opus 5 on other leaderboards widens to 72% versus 36% on ApprenticeBench, which is exactly the gap worthy of further research.

AA: Fable 5.1 scores 53, Opus 5 scores 51. Source: Artificial Analysis Leaderboard

Terminal-Bench 4.0: Fable 5.1 scores 57.9%, Opus 5 scores 51.8%. Source: Terminal-Bench Leaderboard

CursorBench 4.0: Fable 5.1 Max scores 51.8%, Opus 5 Max scores 46.6%. Source: CursorBench Leaderboard

It is also worth noting that the gap between open-weight models and closed-source models is very intuitive. Fable 5.1 and Kimi K3 score 53 and 44 on AA respectively, while their pass rates on this benchmark are 72% and 18%. Among the 100 bills, one failed 28 bills, and the other failed 82 bills. If you are the one who has to take over the remaining work, this gap is hardly negligible. This evaluation increases the proportion of complex cases, so it cannot be directly regarded as the error rate of daily business. However, those parts that require human intervention are exactly what model teams need to examine carefully.

The next version of the model

Where should we focus our efforts?

The scores have been clearly differentiated, now let's look at the reasons. Several sets of analysis from ApprenticeBench have already given specific clues.

How much "CUA tax" do models have to pay when using a graphical interface instead of APIs?

How much will using a graphical interface drag down the model's performance? For fair comparison, the ApprenticeBench team specially customized a set of API tools with equivalent functions to the graphical interface based on Odoo's backend interface, allowing the model to complete the same job through the two different methods respectively.

The team proposed the concept of "CUA tax" to measure the success rate loss caused by the Agent switching from API to GUI:

CUA tax = (API success rate − GUI success rate) ÷ API success rate

The relative success rate loss caused by using GUI drops rapidly with the iteration of model generations. Source: ApprenticeBench Blog

This "tax" gradually decreases as models iterate. The GUI/API success rate of Fable 5.1 is 72%/70%, and that of Astra is 68%/65%. This result shows that the CUA tax of the most powerful models today may have disappeared.

This means that Agents are expected to directly use existing enterprise software, reducing the need for specially developed interfaces. However, the operation cost of GUI is still higher, and there is still room for efficiency improvement.

Did the model already know how to do it, or did it learn it after onboarding?

Getting a question right does not necessarily mean the model has learned something new. It may already know the answer, or it may just find an existing answer to copy.

The team set up a "smart novice" control group: only the company handbook and software tutorials are retained, historical bills, supervisor feedback and cross-task memory are removed, to see how much work the model can complete only with its existing knowledge. Fable 5 only passed 11 bills under this setting. When historical bills are added separately, the number of passed bills rises to 31. When supervisor feedback is added separately, the number rises to 35. When both are provided at the same time, the number reaches 43. The experience accumulated in the company can indeed help it do its job better. However, there is still a gap between finding old answers and learning to handle new situations. In the cost code experiment, when there are precedents to refer to, the tested models generally perform well. When it is necessary to infer rules from historical bills, Fable 5.1 and Astra show obvious leading advantages.

The left part shows cost codes with referable precedents, the right part requires induction of implicit rules; the dotted line is the reference performance of the smart novice group. Source: ApprenticeBench Blog

Some models still have limited performance improvement even after reviewing historical bills and receiving supervisor feedback. Some can find answers from old bills, but their accuracy drops significantly when they need to infer rules on their own. These control groups allow us to clearly see the capability gaps behind the scores.

Why do Agents work slower when they gain more experience?

After people get familiar with their work, they usually work faster and faster. But in this test, Agents generally work slower and slower as they process more tasks.

Changes in processing speed and note-taking volume of humans and Agents as tasks progress. Source: ApprenticeBench Blog

Note-taking is one example. Fable 5.1 wrote about 192,000 words of notes, scattered across 122 files. Among the human testers, the one who took the most notes only wrote about 2,800 words. Remembering more information does not mean using it well. As notes accumulate, it takes time to access and maintain them. There is also a very interesting case: when the scanned copy of a bill is too blurry, the Agent even wrote an image processing program on its own, spent nearly two hours on it, and finally asked the supplier for a clear copy. It could have solved the problem by just sending a message, but it took a long detour.

These problems are very difficult to detect only by looking at the success rate. Only by examining the execution process, time consumed and tokens used together can we know where the model gets stuck. Then we can figure out what the next version of the model should learn and what useless work it should avoid, which gives us specific improvement directions.

Performing better than humans

We also need to calculate the cost

The pass rate of Fable 5.1 reaches 72%, exceeding the highest human pass rate of 51%. But the average cost for processing each bill is 18.23 US dollars, while the figure for humans is 7.21 US dollars, a gap of about 2.5 times. Here, the model cost is calculated based on token fees, and the labor cost is calculated based on the salary corresponding to working hours. There are also some costs that are not counted. When processing the same 100 bills, Fable 5.1 sent 1,03