Jev calibration: why three confident decisions make one unreliable pipeline

You build a moderation pipeline out of three decisions. Is this spam. Is this abusive. Is this off topic. Each one is a typed question to a decision model, each comes back with a probability, and each is calibrated: when it says 90% it is right about 90% of the time. You auto-publish anything that clears all three gates and send the rest to a human.

You do the arithmetic before you ship. Three gates at 90% gives 0.9 times 0.9 times 0.9, so roughly 73% of posts clear the pipeline correctly. That feels low, so you push the gates to 95% and get 86%, ship it, and watch the dashboards.

The aggregate numbers come back better than you predicted. Then, three weeks later, a specific kind of post starts going straight to the front page, and it is the same kind every time.

Both of those things have the same cause, and the arithmetic you did at the start could not have told you about either.

#Calibration is a promise about one question

When a decision model reports calibrated confidence, the claim is specific and worth stating precisely. Take every item where the model said 80%. Across that group, about 80% should turn out positive. That is all. It is a property measured over a population of answers to one question, on the data the calibration was measured on.

It is a genuinely useful property. It is what lets you set a threshold and mean something by it, which is more than you get from an ordinary language model's self-reported confidence. Jev is trained for this specifically, with reinforcement learning against verifiable ground truth scored by a proper scoring rule, and TypeSafe are straightforward that the guarantee is group level.

What the promise does not cover is what happens after the answer leaves the model. The moment you put a threshold on it, branch on it, or feed it into a second decision, you are no longer in the territory the calibration was measured in.

#The product rule is the first thing everybody reaches for

Three independent gates, each correct with probability 0.9. The chance all three are correct is 0.9 cubed, which is 0.729. It is the calculation everyone does, including me the first time, and it rests entirely on the word independent.

Independent means gate two is no more likely to be wrong on a given post just because gate one was wrong on it. Think about whether that is true. All three questions go to the same model, with the same weights, trained on the same data. They are asked about the same input. If that input is unusual, badly formatted, in a language the training set was thin on, or simply a kind of thing the model has never had to judge before, it is hard for all three questions at once.

The errors are not independent. They share a cause, and the cause is the item.

#What correlation actually does

I wanted to see the size of the effect rather than argue about it, so I ran it. The construction holds each gate at exactly 90% accuracy and varies nothing but how much of the error is explained by the item rather than by the individual gate. A shared standard normal factor plus a private one, mixed by a correlation parameter, thresholded so the marginal stays fixed.

import random
from statistics import NormalDist

N, GATES, ACC = 500_000, 3, 0.90
Z = NormalDist().inv_cdf(1 - ACC)

def run(rho):
    a, b = rho ** 0.5, (1 - rho) ** 0.5
    ok = 0
    for _ in range(N):
        shared = random.gauss(0, 1)
        correct = [a * shared + b * random.gauss(0, 1) >= Z for _ in range(GATES)]
        ok += all(correct)
    return ok / N

With the per gate accuracy pinned at 90% and the independent product sitting at 0.729, here is what comes out:

The pipeline gets better as the errors get more correlated. That is not a mistake in the simulation. In a chain where everything has to pass, correlated failures overlap, and overlapping failures waste less of your budget than scattered ones. At perfect correlation the whole pipeline is exactly as accurate as one gate, because the three gates are effectively one gate.

So the product rule is not just imprecise, it is biased, and it is biased in the direction that makes you relax. Your pipeline will beat the number you predicted. You will conclude your estimate was conservative and move on.

#The failures move, they do not disappear

Now bucket the same runs by how hard the item is, meaning by the shared factor, and look at where the failures land. Each column is a decile of item difficulty, easiest on the left.

This is the part that matters, and it is the answer to the second thing that happened three weeks in.

With independent errors the pipeline is wrong about 27% of the time on everything. Unpleasant, but it is noise, it is visible in every sample you look at, and no particular user experiences it more than any other.

With correlated errors at 0.9, the pipeline is essentially perfect on the first six deciles and fails 96% of the time on the last one. The overall failure rate went down. The experience for whoever is generating those hardest items went from bad to total. Every post of that shape gets through, every time, and it keeps getting through, because nothing about the pipeline is random any more.

An aggregate accuracy number cannot show you this. Both rows can average out to something you would sign off on. One of them is a system with a bit of noise and the other is a system with a hole in it.

#Fallback chains break the other way

The moderation pipeline was a conjunction. Everything has to pass. The other common shape is a disjunction: ask the model, and if it is not confident, ask again with more context, and if that fails, ask a bigger model. Three chances to get it right.

Same construction, same 90% per attempt, but now success means at least one attempt is correct:

The independent estimate is 1 minus 0.1 cubed, which is 0.999, and that is the number you would quote in a design doc. At realistic correlation the true figure is 0.943. Your three nines were one nine.

And here the bias runs the other way. Conjunctions come out better than the product predicts, disjunctions come out worse. Whichever you built, the number you calculated is wrong, and you cannot even reason about the direction of the error without knowing the shape of the pipeline.

Retrying the same model with a slightly different prompt is the case where the correlation is close to 1. The retry has the same weights and the same blind spot. This is the same trap as retrying a failing dependency: the second attempt only helps if it fails for reasons the first one did not.

#Where independence actually comes from

Two human reviewers disagree because they have different histories. Two calls to one model do not have that. If you want errors that do not line up, the attempts have to differ in something structural, not in wording.

A rule beside a model is genuinely independent, because a regular expression that catches a phone number has no opinion about tone and fails on completely different inputs. A different model family gets you part of the way, though less than you would hope, since two models trained on overlapping web text share more blind spots than their vendors suggest. A human in the loop is the real thing, which is the whole argument for spending your review budget where the model is least sure rather than spreading it evenly.

What buys you nothing is the same model asked three times in three voices. That is one decision with extra latency and three times the bill.

#What to measure instead

Measure the pipeline, not the gate. Calibration and accuracy per question tell you about a component. Hold out a set of real inputs, run them through the whole thing exactly as it runs in production, thresholds and branches and fallbacks included, and score the output you actually act on. That number is the only one that describes your system.

Then refuse to look at it in aggregate. Slice it by anything that might be the shared factor: language, length, source, customer, time of day, whichever dimension your hardest inputs cluster along. You are looking for a decile that behaves like the last column of that table. A pipeline at 97% overall and 40% on one segment is not a 97% pipeline, it is a broken one with good averages, and only the slice shows it.

Keep the review queue honest, too. If the threshold is doing its job, the items below it should be genuinely mixed. When your reviewers start saying the queue is boring and obvious, the model is underconfident and you are paying people to confirm it. When errors keep surfacing from above the threshold, the calibration you relied on does not hold on your data any more, which is what distribution shift looks like from the inside.

And treat a cheap decision as a thing with a cost. A per call price of four hundredths of a cent makes it tempting to put a gate on everything, but every gate is another place for correlated failure to accumulate, and the review queue it feeds is a queue in the ordinary sense, with an arrival rate and a service rate and all the behaviour that implies. Three gates you understand beat eight you have never sliced.

#The one-line summary

Calibration is a promise about one question on one distribution, and it does not survive being multiplied, thresholded and branched. Correlated errors make conjunctions look better than the arithmetic and disjunctions look worse, and in both cases they concentrate the failures onto one kind of input, where an aggregate number will never show you.

#References

• TypeSafe AI, Introducing System One Models and Jev, on what the calibration guarantee covers
• Anthony Maio, Jev: The Language Model That Won't Talk, which raised the composition problem first
• Glenn Brier, Verification of forecasts expressed in terms of probability, 1950, the scoring rule underneath all of this
• Guo et al., On Calibration of Modern Neural Networks, on reliability diagrams and expected calibration error
• TypeSafe API reference, for the question types and the per request limits