What a Correct Decision Costs

The post that started this

Ably published a demo in which jev played Pong against Claude Haiku 4.5 and GPT-5.6 Sol. One model decision per ball movement, same serve, same rules. In twelve seconds jev made 47 moves. The other two made two each. It is playable.

It is a good chart, and it measures one thing cleanly: how fast a model answers. What it left me wanting was the next question. Speed matters because slow decisions cost money, but wrong decisions cost money too, and so do tokens. If I am routing support tickets, I do not care how many decisions a model can produce. I care what one usable decision costs me.

So I built the version of that test which answers it.

One number that moves with all three things

The score is dollars per 1,000 decisions that are both correct and on time. A decision counts only if it matches the human annotator's label and lands before the deadline, measured from the moment the ticket arrived rather than from when the request went out. Every API call is billed whether it qualified or not.

That one figure moves in the right direction for all three failure modes. Too slow and the decision misses its deadline, so the denominator shrinks. Wrong, and the denominator shrinks the same way. Expensive per token, and the numerator climbs. Nothing hides.

The setup

Tickets come from CLINC150, the intent classification dataset from Larson et al. Every label in it was assigned by a human annotator, which matters: nothing here is a model grading its own homework. I used the 30 banking and credit card intents as the routing taxonomy, because 30 queues is a plausible support desk and the full 150 is not. About one ticket in six is out of scope, meaning none of the 30 queues fit, and every model gets an explicit catch-all option for those.

Five contenders: jev 1.13, Claude Haiku 4.5, GPT-5.6 Luna, DeepSeek V4 Flash and DeepSeek V4.1 Flash, the last two served through Together. Three trials of 300 tickets each, at five arrival rates.

Tickets arrive whether or not you are ready

This is the choice that carries most of the result, so it is worth being explicit about. Arrivals run on a fixed clock into a queue, and a pool of worker slots pulls from it. Time spent waiting in that queue counts against the deadline.

The alternative, sending the next request when the last one returns, lets a slow model set its own pace. The measurement then collapses into per-request latency and the whole exercise becomes the Pong chart again. Production does not work that way. Tickets arrive on their own schedule, and a model slower than that schedule accumulates a backlog. The backlog is what makes latency expensive.

The load is swept, not chosen

A single arrival rate is a number picked by whoever wants a particular answer. Set it low and everybody passes. Set it high and only the fastest survives. So the rate is swept across 2, 5, 10, 20 and 40 tickets per second, and every figure on this page carries the rate it was measured at.

The other four got their best shot

A benchmark where the author's preferred model wins because the others were handicapped is worth nothing. The controls, in full:

• One taxonomy, defined once in a single file, shared byte for byte. No adapter paraphrases anything.
• Every model gets the catch-all option, so out-of-scope performance measures whether a model uses it rather than whether it was denied one.
• Constrained output through each vendor's own mechanism: forced tool use with an enum for Anthropic, JSON schema for OpenAI and Together. No model is penalised for prose it was never asked to suppress.
• Reasoning set to the lowest available setting. Routing is not a puzzle, and leaving reasoning on would inflate both latency and output tokens for work the task does not need.
• Identical slots, deadline, arrival schedule and ticket order. Warm-up calls discarded. Provider order rotated every trial.
• The jev request was linted with wellposed first, because a badly posed question would measure the question rather than the model.

The method was written down before any measurement was taken, and all five amendments carry a date and a reason. Two of them caught configuration bugs that returned HTTP 400 and 404 on every call. Without those fixes, Luna and V4 Flash would have scored zero for a typo.

The result

jev costs $0.048 per 1,000 correct on-time decisions and does not move as load rises. The others start between $0.09 and $1.72 and climb as they fall behind, Haiku reaching $24.48 at 40 tickets per second.

Log scale. The price of a call does not change with load. What changes is how many decisions still qualify.

The mechanism is visible if you plot the share of tickets that came back correct and on time. With 8 worker slots, a model keeps up to roughly 8 divided by its per-call time. Past that the queue grows without limit.

Every cliff lands within a ticket per second of where that arithmetic predicts. V4 Flash breaks first at 10, Luna and Haiku at 20, V4.1 Flash degrades at 40, and jev does not.

The objection I expected

The first thing a careful reader says to this is: you starved the connection pool. Eight slots is not many. Give the others more capacity and the cliff moves.

That is correct, and it is the right objection, so I ran it. Same sweep, 32 slots instead of 8.

jev is still cheapest at every rate under both pools. That is the only thing four times the capacity leaves unchanged. Its lead over Luna at 40 tickets per second falls from 45x to 3.3x, and over Haiku from 510x to 35x. The four models behind it change order at all three rates.

So the honest version of the headline is not 510x. It is about 3x cheaper than the next model once every model has the capacity to keep up, which is the condition anyone running this in production would meet. The larger multiples are real measurements of a starved pool, and they belong in the appendix rather than the title.

Where the time goes

It is tempting to read the cliff as the generative models being slow. They are not, particularly. At 40 tickets per second Haiku spends 671 ms inside the model and 9,822 ms waiting in line. Luna is almost identical. Over 90% of the lateness is queueing, not inference.

Solid is median time in the model, pale is median time waiting for a free slot. The deadline is 2 seconds.

This is queueing theory doing exactly what queueing theory does, and it is the part worth internalising. A model that takes 650 ms per call is not slow in any way a human would notice. It is slow enough that a fixed pool of eight workers saturates at about 12 tickets per second, and past that point the wait is unbounded. The difference between 150 ms and 650 ms does not show up as a 4x gap. It shows up as a cliff.

Before load enters the picture at all

Some of the gap is just price. Stripping out every queueing effect and comparing the cost of one call:

jev is about 35x cheaper per call than Haiku before a single ticket queues. Two things drive that. Output tokens are free on jev and are not on the others, and Anthropic's forced tool use sends roughly 780 more input tokens than Luna's prompt, because the 31 queue names appear twice: once in the instructions and once in the tool definition. That is the cost of the mechanism that makes Haiku's output reliably parseable, and it is a fair cost to count.

The part that argues against my own headline

jev is not the most accurate model here. Luna and Haiku both score 93.6% on in-scope tickets against jev's 92.2%, and jev sends 5.0% of in-scope tickets to the catch-all option against 3.3% and 3.4% for the other two. It over-triggers the escape hatch.

Confidence on that ranking: unclear. The five arrival rates reuse the same 249 in-scope tickets, so the sample behind it is 249, not 1,245, and jev against Haiku comes out at p about 0.09. The correct reading is that accuracy is bunched, not that jev is worse.

Which is the interesting part. Accuracy spans 3.4 points across five models. Cost spans about 500x. On this task the models are roughly interchangeable at the thing people argue about, and wildly different at the thing they do not.

Out-of-scope recall runs 97.3% to 100% for everyone, so it separates nothing. The out-of-scope examples in CLINC sit far outside the domain and are easy to spot once you have somewhere to put them. I had expected this column to be the interesting one, on the strength of earlier work showing that a Choice question with no escape hatch produces confidently wrong answers. Giving every model the hatch removed that failure entirely, which is a result, just not the one I went looking for.

What three trials showed that one would have hidden

DeepSeek V4 Flash ranges from 17.7% to 87.7% at 5 tickets per second across three trials. It also broke at 2 tickets per second, the lowest rate tested, with 14 client timeouts and six server errors. Three days earlier the same model at the same rate scored 92.7%.

Its line on every chart above describes Together's serving on the day I ran it, not the model. I have left it in rather than dropping it, because a benchmark that quietly removes its inconvenient run is not a benchmark. But it should not be read as a property of DeepSeek, and the fix is to go direct to DeepSeek's own API and take the host out of the measurement.

What bounds all of this

What I would change

Three things, in order of how much they would move the result. Run both DeepSeek models against DeepSeek's own API rather than through Together, which removes the largest source of noise in the current numbers. Add a cost view priced at cached rates next to the list-price ceiling, since that is where the result is most vulnerable. Run the zero out-of-scope case, to confirm none of this rests on catch-all traffic. All three come to under $20 of API spend.

The broader thing I would do differently: I designed this expecting the escape hatch to be the story, and it was not. The story was queueing, which I had treated as harness plumbing rather than as the subject. If you are choosing a model for a decision that happens under load, the question worth asking first is not how accurate it is. It is what happens to your queue at the arrival rate you actually see.

Method, code and raw data

Everything is in jev-triage-bench: the preregistered protocol with its dated amendments, the harness, the scorer, and one JSON row per ticket with arrival time, queue wait, service time, token counts and correctness. A second scorer recomputes the headline figure from those rows with no code in common with the first, and the two agree.

If you want to check a number rather than take it, that is the file to open.