I've figured out the most office-worker-friendly usage of Jev: you can turn your boss into a scenario for the turtle soup reasoning game.

Jev, which has been going viral across the internet in the past few days, is truly a phenomenal creation: it does not chat, nor write articles, but only reads a piece of information, outputs several judgments and their corresponding probabilities, and then sweeps across the entire online community.

A model focused on "decision-making" and "judgment"? It sounds extremely complex, but there is a very straightforward way to understand it: its operating principle is similar to that of the "Turtle Soup" puzzle game.

In the Turtle Soup game, players need to keep asking questions to uncover the truth, but every question is a closed-ended one that can only get two answers: "Yes" or "No". Each answer is not the final conclusion, but can provide direction and guidance for the next question.

JEV is also a model for processing closed-ended questions. More precisely, it is dedicated to structured judgment and can return results of true/false, options, scores and their corresponding probabilities. Different from the Turtle Soup game, it will not only output "Yes" or "No", but also give results like "probably true", "possibly false" and more specific probability figures.

But understanding is one thing, it still feels a little disconnected from real life — in daily work, what exactly are the scenarios that require "decision-making"? Aren't ordinary employees just doing their work and submitting the deliverables to their supervisors...

😯 Right, the people who need to make decisions are none other than the supervisors themselves!

Minimum Scale Test

Obviously, the Turtle Soup analogy is just the most simple and crude way to understand JEV. JEV can also handle multiple-choice questions and scoring tasks. But to first figure out how it actually works, I limited my first test to the true/false scenario: no room is reserved for it to play freely, and I only check whether it can stably judge the same type of problems according to a set of clear standards.

Let's start with a minimal test case. JEV's Playground page is very concise yet misleading. The following shows a very simple conversation case to explain what you need to master to use this model:

State refers to what has happened, questions refer to how to judge what has happened, and JEV returns "the probability that the statement is true".

In the figure, I used an error fragment I encountered in a chatbot and input it into the State window, wrote my judgment rule in the Questions window, and the final result after running is 99%.

Why is this an error fragment? Because when I say "I don't feel well today", the correct response can be suggestions like lying down, drinking water, taking a rest, anything is fine, but there should be no chain-of-thought content such as format check or step planning — if such content appears, the response is judged to be wrong.

Therefore, according to the standards I wrote, JEV judges the probability that the original response is an error fragment as 99%. No retraining is involved here. It just reads the standards in this single run, makes judgments based on them, and outputs the results.

0.99 → Very close to true

0.50 → Uncertain

0.01 → Very close to false

It is not a necessary condition to put only one case at a time. This setup here is only to help better understand the playground cases in the smallest unit, and also to control variables when evaluating JEV: you can put one case in State and four judgment questions at the same time. After the run, replace it with the next case, keep the Questions unchanged, and then compare the results of different cases.

The model capability of Jev is of course important, but a large part of the final effect depends on how the Questions are written, whether you can clearly state your judgment standards, clear enough to leave only closed-ended answers — this is the most similar point between JEV and the Turtle Soup game. If you raise a vague question, you can only get ambiguous results.

If you only write your Question as "Is this email draft suitable for sending", JEV will still return a probability, but since the definition of "suitable" is not specified, this figure is very difficult to verify, and cannot be reused for the next judgment. You need to split the Questions into:

Does it leak the backstage work draft?

Does it introduce unsubstantiated old topics?

Does it confuse the identities of relevant people?

Does it fail to respond to the user's current expression?

Are there any grammatical errors?

The probability obtained in this way is easier to verify, and can also be reused for subsequent judgments.

How to Get High-Impact Results

By the same logic, in daily life, when you submit your finished work, your supervisor will also go through a series of judgments, a set of questions running through their mind: Is this topic good? Will this article go viral? Will this video attract views?

If you can split these judgment standards into specific closed-ended questions and "distill" your supervisor's decision logic, you can effectively improve the pass rate of your work and get high-impact results.

To test the feasibility, I created a small set of test questions, collected dozens of past topics, desensitized the information as the cases in State, and inquired a set of fixed editor judgment questions in batches.

TypeSafe officially provides ordinary HTTP API and Python SDK; the script sends a state and a set of questions to POST /v1/systemone, and obtains structured probabilities. The official also generously gave a $5 trial credit, which is more than enough considering its extremely low price.

From Feishu, I (let Codex) sorted out a set of test topics, and automatically generated "raw version + finished version" test pairs for each topic. The raw version refers to topic inspirations, fragments and partial scattered ideas, while the finished version is the finally formed article. Then I saved them as local JSONL, and asked Codex to generate the test script.

I have been running a relatively simple labeling system in Feishu. In this test, JEV's performance will be compared with my manual judgments. Based on the existing topic data, Codex initially designed a set of questions for JEV — not very practical, and disconnected from real work, but sufficient for testing purposes.

From the returned report, I carefully checked the outputs and scripts, and reviewed the writing of Questions and the judgment logic in the test set. The ranking of several core indicators is roughly consistent with the manual labeling results, except for the "overstepping of assertions" item, which it cannot judge accurately: for some test pairs where the difference between the finished version and the raw version is too large, it cannot tell whether it is a good upgrade or a bad overstep.

It doesn't matter. Bad publicity is still publicity. A hit made by clickbait is still a hit, just get it done. Let Codex finish the last script, and verify it with a brand new topic:

This new topic is only in the draft stage, with only basic facts and descriptions, no arguments and demonstration directions, so its "structural judgment" score is only 22%, and the evidence sufficiency score is only 8%. This is roughly consistent with the discussion result in the topic meeting, and we are one step closer to distilling the supervisor's decision logic.

Finally, package all the components, add a simple front-end, and a topic judgment tool is born!

Just input your ideas and flashes of inspiration, and you can get a series of scores. Even for the same hot topic, different topic writing methods will lead to different scores. It is faster and more intuitive than using large LLM models, and you don't have to worry about the cost at all, the $5 credit can last for ages.

Strictly speaking, we haven't really "distilled" the supervisor out in this test. JEV doesn't know who the supervisor is, nor has it learned the ever-changing demands of any specific leader. Overall, it only compresses the judgment standards that were originally scattered in topic meetings, revision comments and editing experience into a set of fast-processable questions.

However, although I didn't get an electronic supervisor after the distillation, I got a questionnaire that even the supervisor has to answer according to the rules. As for what is still missing to get the real final distillation result, it is probably only to feed in all the "this is not acceptable" feedback from the leaders over the years, and bear the mental damage caused by this process.

This article is from WeChat Official Account "APPSO", Author: APPSO who discovers tomorrow's products, published with authorization from 36Kr.