{"id":7078239,"date":"2026-10-06T14:49:18","date_gmt":"2026-10-06T14:49:18","guid":{"rendered":"https:\/\/fivemor.com\/?p=7078239"},"modified":"2026-10-06T14:49:18","modified_gmt":"2026-10-06T14:49:18","slug":"jev-vs-llm-as-a-judge-the-ai-evaluation-comparison","status":"publish","type":"post","link":"https:\/\/fivemor.com\/?p=7078239","title":{"rendered":"JEV vs LLM as a Judge: The AI Evaluation Comparison"},"content":{"rendered":"


\n<\/p>\n

\n
\n
\"Jev
An LLM judge writes its answer as text. JEV gives a short, ready-to-use answer directly.<\/em>\u00a0<\/figcaption><\/figure>\n<\/div>\n

Many teams now use LLM-as-a-Judge to check AI answers, especially when exact-match tests fail for long or open-ended responses. But every judgement adds cost, delay, and possible bias, making this hard to scale.\u00a0<\/p>\n

Jev, a small decision model from TypeSafe AI, takes a leaner route: it returns a short choice with confidence instead of full written reasoning. In this article, I\u2019ll explain how Jev works, compare it with LLM judges, and test where it helps or falls short.\u00a0<\/p>\n

What is JEV as a Judge?\u00a0<\/h2>\n

A normal chatbot can explain, summarise and write. Jev cannot. It\u2019s designed for handling small decisions and only. TypeSafe refers to it as a \u201cSystem One\u201d model, similar to thinking on the cheap. The company says it trained Jev to give honest confidence numbers. So it has not been open and we haven\u2019t been able to verify these claims. For this reason, a testing with our own data is significant. Rather Langfuse is a well-known AI App tracking tool that is already integrated with LLM judges and code-based checks.\u00a0<\/p>\n

Jev can give three types of answers:\u00a0<\/p>\n

\n
\n\n\n\n\n\n\n
Type<\/strong>\u00a0<\/td>\nWhat you get<\/strong>\u00a0<\/td>\nWhere to use it<\/strong>\u00a0<\/td>\n<\/tr>\n
Choice<\/strong>\u00a0<\/td>\nOne option from a list you give, with a probability for each option\u00a0<\/td>\nWhich answer is better? Which type of error is this?\u00a0<\/td>\n<\/tr>\n
Score<\/strong>\u00a0<\/td>\nA level on a scale, such as low, medium or high\u00a0<\/td>\nHow risky is this action? How good is this reply?\u00a0<\/td>\n<\/tr>\n
Noul<\/strong>\u00a0<\/td>\nThe chance that a yes\/no statement is true\u00a0<\/td>\nIs this answer based on the document? Is this allowed by the policy?\u00a0<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n<\/div>\n

Langfuse, a popular tool for tracking AI apps, already supports Jev next to LLM judges and code-based checks.\u00a0<\/p>\n

JEV vs LLM as a Judge\u00a0<\/h2>\n

Many people compare the two only on accuracy. In real projects, other things matter too. Here is a simple comparison:\u00a0<\/p>\n

\n
\n\n\n\n\n\n\n\n\n\n
Point<\/strong>\u00a0<\/td>\nJEV<\/strong>\u00a0<\/td>\nLLM judge<\/strong>\u00a0<\/td>\n<\/tr>\n
Output\u00a0<\/td>\nShort answer with probabilities\u00a0<\/td>\nWritten text, often in a fixed format\u00a0<\/td>\n<\/tr>\n
Explanation\u00a0<\/td>\nNone\u00a0<\/td>\nCan explain its decision\u00a0<\/td>\n<\/tr>\n
Confidence\u00a0<\/td>\nComes built in, as probabilities\u00a0<\/td>\nThe model just says a number, often 0 or 1\u00a0<\/td>\n<\/tr>\n
Speed and cost\u00a0<\/td>\nVery fast and very cheap\u00a0<\/td>\nSlower and costlier, especially with deep thinking\u00a0<\/td>\n<\/tr>\n
Best for\u00a0<\/td>\nSimple, repeated checks where the proof is in the text\u00a0<\/td>\nOpen questions, hard thinking, written feedback\u00a0<\/td>\n<\/tr>\n
Weak at\u00a0<\/td>\nMaths, code, logic, tricky writing styles\u00a0<\/td>\nHigh cost and delay; can still be biased\u00a0<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n<\/div>\n

The most crucial one is the confidence row. 88 cases were used to test OpenRouter. Jev\u2019s confidence numbers fell nice and between 0 and 1 but the LLM judge was rarely in the middle, falling close to 0 and 1 most of the time, even when it wasn\u2019t sure. The errors scores were 0.043 for Jev as well as 0.054 for the LLM (lower is better). One test is not proof, but it is a reason why not to blindly trust confidence numbers, but rather check them against actual answers.\u00a0<\/p>\n

What the CMU Study Found\u00a0<\/h2>\n

Jev was compared with sixteen other judges on numerous tasks by four researchers from the CMU: Yubo Li, Yidi Miao, Ramayya Krishnan and Rema Padman. There are three observations to be made.\u00a0<\/p>\n

1. Average accuracy, but very low cost\u00a0<\/h3>\n

Jev cost $0.044 for 1000 judgements, and took 0.15 seconds\/judgement. GPT-6 Astra<\/a> took 1.89 seconds with a price tag of $12.182 for the same. So Jev was 277 times less expensive and 13 times faster. The numbers shown are based on the cost of the study\u2019s own test \u2013 actual cost may be higher or lower.\u00a0<\/p>\n

\n
\"Jev
Cost and speed of different judges in the CMU study<\/em><\/figcaption><\/figure>\n<\/div>\n

2. Good when the answer is in the text, weak when it must be worked out\u00a0<\/h3>\n

In a test run, Jev gets 92.5% accuracy while GPT-6 gets the same score on RewardBench. Jev got 87.3% and GPT-6 got 88.4% on HaluEval which evaluates facts based on evidence. On the harder judge bench, Jev\u2019s score was 78.6% while GPT-6\u2019s was 93.1%. On logic puzzles, it was 68.4% against 95.9%. Jev is also a stylish finicky. With the answer being concise and direct and the correct answer being longer and more highly crafted, Jev achieved a score of 76.6%, while GPT-6 scored 90.1%.\u00a0<\/p>\n

\n
\"Jev
How far Jev is from the comparison model on each type of task<\/em>.<\/figcaption><\/figure>\n<\/div>\n

3. Some tasks are hard for every judge\u00a0<\/h3>\n

Without an answer to compare with, all three models, Jev, GPT-4.1 mini<\/a> and GPT-5.4<\/a>, performed near-random selection agnostically and sounded confident. A larger model was not a solution. The lesson to be learned is to present any judge with the evidence or a checklist.\u00a0<\/p>\n

Please note: <\/strong>Some labels may be incorrect, and the authors have not experimented with special fields such as law or medicine. Take these as an indication; and always test on your own data.\u00a0<\/p>\n

The Real Strength: Knowing When It Is Unsure\u00a0<\/h2>\n

A inexpensive judge is useful merely in the event that she finds out when it can be mistaken. The confidence of Jev is just its maximum probability. For instance, it could be 95% \u201cfirst\u201d and 5% \u201csecond\u201d in which case the confidence is 0.95.\u00a0<\/p>\n

This makes it easy to have a two-step check. Set a cut-off, say 0.90. Take Jev\u2019s answer if it is above the cut-off. If it\u2019s below, then send that case to a larger model. [2]\u00a0<\/p>\n

\n
\"Two
The two-step check. Keep the cut-off based on your own data, not copied from a paper.<\/em>\u00a0<\/figcaption><\/figure>\n<\/div>\n

This two-step check was attempted on 1610 new pairs in the study. Jev sent out 68.5% of them single handed. The overall accuracy was 93.4%, a slight improvement over GPT-6 (92.5%) and the cost was just 41.4% of GPT-6\u2019s price. On a new, more challenging task, the system referred 74.2% of the cases to the larger model. That is fine. The concept is to take no chances with the wrong answer, but not to cut corners for the sake of being budget-friendly.\u00a0<\/p>\n

Hands-on: Testing Jev with Tricky Cases\u00a0<\/h2>\n

It would just be another simple demo that would prove that the API works. I was interested to see where Jev could go wrong. Thus I created 12 difficult cases\u2014long but incorrect answers; hidden instructions that attempt to trick the judge; and questions that require calculation. For each case there are 2 answers and I do already know which of those answers is correct.\u00a0<\/p>\n

I ask each judge two times; first asking with A, first with B. This indicates whether the judge just prefers one answer over the other.\u00a0<\/p>\n

It\u2019s only the code that I used, none hidden, everything below. Each file can be duplicated as is. This is an accurate representation of the screen on my computer.\u00a0<\/p>\n

What you need: <\/strong>Python 3.9 or newer, TypeSafe API key (for Jev), and API key for any OpenAI compatible LLM (for the comparison judge).\u00a0\u00a0<\/p>\n

Step 1: Create the folder and install the packages\u00a0<\/h3>\n

Make a new folder, for example \/lab<\/code>, and open a terminal inside it. Then run this:\u00a0<\/p>\n

python -m venv .venv\u00a0\n\nsource .venv\/bin\/activate\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 # on Windows: .venv\\Scripts\\activate\u00a0\n\npip install requests pandas numpy python-dotenv openai matplotlib truststore<\/code><\/pre>\n
Now create a file named .env in the same folder and put your keys in it. Never share this file or upload it to GitHub.\u00a0<\/p>\n
File: .env<\/strong>\u00a0<\/p>\n
TYPESAFE_API_KEY=your_typesafe_key_here\u00a0
LLM_API_KEY=your_llm_provider_key_here\u00a0<\/p>
LLM_BASE_URL=\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 # leave empty if you use OpenAI directly\u00a0<\/p>
LLM_JUDGE_MODEL=your_model_name\u00a0\u00a0\u00a0 # any OpenAI-compatible chat model<\/p><\/pre>\n
After this, your folder should have these files. We will create them one by one:\u00a0<\/p>\n
lab\/\u00a0\u00a0 .env\u00a0\u00a0 judges.py\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 # talks to Jev and to the LLM judge\u00a0\u00a0 raw_call.py\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 # one simple Jev call, to see the raw answer\u00a0\u00a0 cases.py\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 # the 12 test cases\u00a0\u00a0 run_lab.py\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 # runs both judges on all cases\u00a0\u00a0 report.py\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 # prints the results\u00a0\u00a0 plot_frontier.py\u00a0\u00a0 # draws the final chart<\/pre>\n
Step 2: Write the helper file that talks to Jev and the LLM\u00a0<\/h3>\n
This is the most crucial file. It has four parts:\u00a0<\/p>\n