Jev benchmarked on 5,574 real messages: accuracy, calibration and where it fails

Jev launched four days ago and every number in circulation belongs to the company selling it. About 68% accuracy on their own four workflow benchmark, 98.3% on a spam task, 20 to 200 times faster and 40 to 400 times cheaper than a language model doing the same job. Those may all be true. They are also exactly the numbers a vendor would publish.

So I ran it myself on a public dataset with real labels and wrote down what came back. No fine tuning, no examples in the request, two labels, 5,574 messages. This is one task on one day, which I will come back to at the end, but the numbers are real and you can reproduce them in about five minutes.

#The setup

The dataset is the UCI SMS Spam Collection: 5,574 real text messages, 747 of them spam, hand labelled, published in 2012 and used in classification papers ever since. It is a good test precisely because it is old and public. Anything that does well here does well on genuinely messy input, full of abbreviations, missing vowels and 2005 era mobile phone spam.

One honest note about the route. Direct TypeSafe API access is waitlisted, so I went through classifier.dev, a keyless endpoint that serves Jev. Every response came back stamped jev-1.13.0, which is a real version string from the TypeSafe model list, but this is an intermediary. The confidence values may be post processed on the way out, and the timings include somebody else's infrastructure. Read the accuracy numbers as Jev's and the plumbing numbers as indicative.

The harness is the whole program:

const rows = readFileSync('SMSSpamCollection', 'utf8')
  .split('\n')
  .filter(Boolean)
  .map((line) => {
    const tab = line.indexOf('\t')
    return { truth: line.slice(0, tab), text: line.slice(tab + 1) }
  })

for (let i = 0; i < rows.length; i += 500) {
  const slice = rows.slice(i, i + 500)
  const res = await fetch('https://classifier.dev', {
    method: 'POST',
    headers: { 'content-type': 'application/json' },
    body: JSON.stringify({
      labels: ['spam', 'not spam'],
      inputs: slice.map((r) => r.text),
    }),
  })
  const body = await res.json()
  body.results.forEach((r, j) => out.push({ ...slice[j], ...r }))
}

Five hundred messages per call, twelve calls, each one back in one and a half to three seconds. There is a rate limit of 3,000 classifications a minute on that endpoint, so the loop needs a pause every six batches. Nothing else. No embeddings to build, no model to host, no training set to assemble.

#The headline numbers

Seventy one mistakes out of 5,574, with no examples given and no training. The vendor's 98.3% spam claim turns out to be slightly modest. A fine tuned encoder on this dataset can beat it, and will cost you a labelled training set, a training run and something to serve the model on. This cost one HTTP call and no setup at all.

Look at the shape of the errors rather than the total, though. Fifty four false positives against seventeen false negatives means it over flags by roughly three to one. If the action behind this decision is "move to a spam folder", that asymmetry is mildly annoying. If it is "block this account", it is the entire design problem, because the expensive error is the one it makes most often.

#The calibration surprise

This is the part I did not expect, and it is the reason the post is worth writing.

Group the answers by the confidence Jev reported, then check how often each group was actually right. A perfectly calibrated model puts the same number in both columns.

Every single bucket is more right than it claimed. Expected calibration error over the whole set comes to 2.47 percentage points, which is respectable, but the interesting thing is that the error is one directional. Jev is not miscalibrated here in the usual way, where a model says 95% and means 70%. It is underconfident.

And at the top it is exact. Of the 1,357 messages Jev called at 100%, it got every one right. Not approximately. Zero errors in a quarter of the dataset.

Underconfidence is the safe direction to be wrong in, and I would rather have it than the alternative. It still costs you something specific: at a threshold of 0.9 you send items to review that the model would have got right 87% of the time. You are paying humans to confirm decisions that were already sound. If you tune a threshold on the model's own numbers rather than on measured outcomes, you will set it too high.

One more detail worth knowing before you build on these values: the confidences are quantised to two decimals, with 95 distinct values across the whole run and 65% of everything sitting at 0.99 or 1.00. That is a coarse signal at the top of the range. Do not expect to separate 0.993 from 0.997, because you will not be given the difference.

#The threshold is where the design lives

Accuracy is not the number you ship. The number you ship is how much you dare automate, and that follows from where you put the confidence threshold. Auto act above it, send the rest to a human.

Read the first two rows together, because that is the whole argument for a calibrated model. A threshold of 0.9 buys you a drop from 71 errors to 13, a fivefold improvement, and costs you 521 items of human review out of 5,574. That is a genuinely good trade and it is only available because the confidence means something.

Now read the last row. Perfect accuracy is on the menu and the price is reviewing three quarters of your traffic by hand. Nobody is doing that, which is the useful way to think about it: the threshold is not a quality setting, it is how you split work between a machine and a person, and the right value comes from what your review capacity is and what an error actually costs you.

That review queue is a queue in the ordinary sense. It has an arrival rate set by your traffic and your threshold, and a service rate set by how many people you have and how fast they read. If arrivals exceed service the backlog grows without limit, exactly as it would anywhere else, and all the usual arithmetic applies. Choosing 0.99 because it looks safer is choosing to triple the queue, and if nobody sized the queue first, you have built a slow consumer behind a fast producer and it will fall over in a week.

#What it gets wrong

The false positives, the real messages it called spam, are almost all one thing. Chain messages.

<#> %of pple marry with their lovers... becz they hav gud undrstndng dat avoids problems

Feb <#> is "I LOVE U" day. Send dis to all ur "VALUED FRNDS" evn me. If 3 comes back u'll gt married

Send ur birthdate with month and year, I will tel u ur LIFE PARTNER'S name

Forwarded fortune messages, astrology, "send this to ten friends". They have the grammar of spam: an instruction to forward, a promise, no specific human on the other end. A person labelled them ham because a friend actually sent them. The model has no way to know that, because the thing that makes them legitimate is the relationship, and the relationship is not in the text. That is not a model failure, it is a question the input cannot answer, and it is worth recognising the shape because a lot of real classification problems are secretly this.

Then the false negatives, where something interesting turned up:

How come it takes so little time for a child who is afraid of the dark to become a teenager who wants to stay out all night?

That is labelled spam in the UCI corpus. It has no link, no phone number, no shortcode, nothing to buy. So is this one, also labelled spam:

Do you ever notice that when you're driving, anyone going slower than you is an idiot and everyone driving faster than you is a maniac?

Jev called both of them not spam, and I think Jev is right and the dataset is wrong. Which means the real accuracy is a little above 98.73%, and more usefully, that a couple of my seventeen false negatives are not errors at all.

That is the honest ceiling on any benchmark like this. You are not measuring the model against the truth, you are measuring agreement between the model and whoever labelled the data. When you run this on your own data, expect to spend real time on the disagreements, and expect a decent fraction of them to be your labels rather than the model.

#What this does not tell you

One task. English. Binary. Short text of a kind that has been in every spam corpus for fifteen years, which means something in the training data has almost certainly seen messages like these. A decision that is genuinely novel to the model, or one that turns on domain knowledge not present in the text, is a different measurement and there is no reason to expect 98.73% there.

It also tells you nothing about the thing that will actually determine whether this works for you, which is what happens when several of these decisions are wired together behind thresholds and branches. Calibration measured per question does not survive composition, and every number above is a single question answered in isolation.

The method transfers even when the numbers do not. Take a few thousand rows you already have labels for, run them through in batches, build the confidence bucket table, and find your own threshold. It costs about four hundredths of a cent per million input tokens and half an hour. Do that before you wire a decision model into anything that matters, because the vendor's accuracy number was measured on the vendor's data, and yours is the only one that describes your system.

#The one-line summary

On 5,574 real messages with no examples and no tuning, Jev was 98.73% accurate, wrong three times as often toward false positives, and underconfident in every bucket, which makes its confidence safe to threshold on and a poor guide to where the threshold belongs.

#References

• UCI SMS Spam Collection, the dataset, 5,574 labelled messages
• TypeSafe AI, Introducing System One Models and Jev, for the claims being tested
• TypeSafe API reference, question types, context and rate limits
• Arize, Can decision models replace LLM judges?, for the 68% workflow figure and its caveats
• Guo et al., On Calibration of Modern Neural Networks, on reading a reliability table