Does One Skill Make DeepSeek V4 Pro Outperform Fable 5? The Popular Plugin Has Been Debunked.

One single Skill makes DeepSeek V4 Pro outperform Fable 5?

A couple of days ago, J-Space quickly went viral with a set of exaggerated test results, but unexpectedly, it completely collapsed today.

The author claimed that by simply integrating one Skill into the Agent environment, V4 Pro can achieve significant improvements across multiple benchmarks, with some scores even surpassing Fable 5, while speed and Token efficiency can also be increased by more than twice.

However, the results of community retests are completely the opposite: not only did the performance not improve, but more tokens were consumed.

Facing doubts from netizens, the author not only failed to disclose complete evaluation records or operation logs, but was also exposed to have deleted the issue that raised doubts.

At this point, the whole carnival turned into a public fraud exposure.

One Skill Makes V4 Pro Outperform Fable 5?

The project that has drawn widespread attention this time is called J-Space Cognition Suite V3.6.

According to the project introduction, this is a set of model-agnostic inference-time control solutions. It does not modify model weights or require fine-tuning, but is added to the Agent operating environment in the form of a Skill.

"DeepSeek V4 × J-Space Capability Realization Report": https://github.com/Tiger3807861189/DeepSeek-V4-J-Space-Capability-Realization-Report

The project author summarizes the common out-of-control situations of Agent into four scenarios.

The first one is working set overload. Too many targets, restrictions and tool information are stuffed into the task at the same time, and the truly important content is submerged instead.

The second one is representation drift. A name, value or task target gradually changes during multiple rounds of reasoning and across multiple files.

The third one is invalid retry. After the tool call fails, the model does not retain diagnostic information, and just runs again along the original path.

The fourth one is premature completion. The model generates a seemingly smooth response, mistakenly thinks that the task has been completed, without verifying whether the result is actually available.

These problems are especially common in long-range Agent tasks.

J-Space tries to organize the entire execution process into a more stable loop: short judgment, execution of operations, in-depth reasoning, result verification, and recovery with diagnostic information when problems are encountered.

It also limits the content that remains active in the current workspace at the same time, and writes key goals, verified information, unsolved problems and next-step operations into an external ledger.

In simple terms, J-Space aims to prevent the model from forgetting what it is actually doing during the working process.

The idea does not sound complicated, but the published results are quite astonishing.

The report claims that after integrating J-Space, V4 Pro's Terminal Bench score increased from 87.9 to 90.1, NL2Repo jumped from 61.5 to 73.4, and DeepSWE increased from 62.7 to 72.0.

According to the public score comparison in the report, V4 Pro even outperforms Fable 5 and Opus 4.8 on some Agent and programming benchmarks.

Fraud Exposure Leads to Post Deletion

The first negative result came from Issue #10 in the J-Space repository.

Link: https://github.com/Tiger3807861189/J-Space-Cognition-Suite-V3.6/issues/10

A developer conducted two rounds of A/B tests using V4 Flash, covering tasks such as mathematical reasoning, code generation, repository development and interruption recovery.

In the second round of experiments, the control group and the J-Space group ran 3 times respectively, for a total of 12 runs.

The results show that there is no significant difference in the final task completion degree between the two groups, while the J-Space group consumed more resources.

In the third-party blind evaluation, the control group scored an average of 8.30, while the J-Space group only scored 7.87.

This experiment has a limited scale, uses V4 Flash and custom tasks, and cannot cover all the benchmarks in the J-Space report, but it also raises some doubts about "simultaneous improvement in performance, speed and Token efficiency".

Subsequently, more conclusive results emerged.

Link: https://github.com/Tiger3807861189/J-Space-Cognition-Suite-V3.6/issues/26

User Jyleaves stated in Issue #26 that he deployed V4 Flash with 8 NVIDIA H20 cards, and loaded J-Space through DeepSeek Harness Standard mode.

Among the 89 questions, the model passed 69, with a final score of 77.5%. The corresponding score given in the J-Space report is 87.1%.

The tester also said that the failed tasks were re-run at least 3 times, but the improvement stated in the report was still not obtained, and thus questioned the authenticity of the project data.

Therefore, the tester asked the author to disclose the complete evaluation environment, test process and sample outputs.

To make matters worse, a developer claimed in Issue #23 that the questioning issue he posted earlier was deleted by the author, so he could only repost to back it up.

Link: https://github.com/Tiger3807861189/J-Space-Cognition-Suite-V3.6/issues/23

Attempting to hide something only makes it more obvious, and the author's action is a self-confession without being forced.

Perfectly Hit the Pain Point of V4 Pro

Clever disguises are rarely fabricated out of thin air. As long as you are good at borrowing existing facts, everything will seem logical.

J-Space just hit the most concerned problem of V4 Pro: it is overly sensitive to the external operating environment.

From the very beginning, everyone found that V4 Pro's performance varies a lot under different users and different call conditions.

On the night of the official version release, someone tested a 3D helicopter game twice with similar prompts. At 0:50 a.m., the result generated by the model was still very rough; less than four hours later, it delivered a much more complete project.

Netizens speculated that the difference might come from outside the model, such as Harness.

In an engineering task called Project2, the same V4 Pro scored 91 points when running in the Standard mode of DeepSeek Harness, 92 points in PTC mode, and after switching to the Minimal mode with fewer tools, it reached 99 points and 96 points in two tests respectively.

The model and the task did not change, but the external operating environment brought a gap of nearly 8 points.

Then some targeted solutions appeared.

Routing Suite tries to select the reasoning mode according to the task type, preventing the model from overthinking when facing simple tasks and acting prematurely when encountering complex tasks.

GitHub: https://github.com/yjh051108/dsh-routing-suite?utm_source=chatgpt.com

Anchored Standard focuses on the first model request: in the first round, use short prompts and a small number of tools to make V4 Pro enter a relatively stable trajectory, and then open up full tool capabilities afterwards.

GitHub: https://github.com/xiaobright/dsh-anchored-standard?utm_source=chatgpt.com

This is the most confusing part of J-Space: it borrows a real existing problem.

Following this logic, a table showing "outperforming Fable 5" can spread across the community in a few minutes, but verifying it is not that easy.

Only the improvement that can be reproduced is a real improvement. For results that cannot provide the complete process, the more astonishing the numbers are, the more we should question them first.

This article is from the WeChat official account "Machine Intelligence", author: Machine Intelligence, published with authorization from 36Kr.