Choosing a Model Is a Cost-Latency-Quality Triangle, Not a Leaderboard

The model at the top of the leaderboard is almost never the right default for your product. It is the right answer to a benchmark question nobody in your system is actually asking.

The leaderboard trap

Here is the pattern I see over and over. A team reads that a frontier model tops the reasoning charts, wires it into every call, ships, and then opens the first month's bill with genuine surprise. The model works. That was never in doubt. The problem is that they bought a Formula 1 car to do the school run and are now surprised by the fuel cost and the fact that it takes ten minutes to warm up.

Public benchmarks measure a model against a fixed test: contest math, graduate-level science, an agentic coding suite. Your workload is not that test. If you are triaging support tickets, extracting fields from a scanned form, or routing inbound leads, the marginal intelligence that separates the number-one model from the number-three model may be entirely irrelevant to your task while the price gap between them is very real. On the current lineup, the top Opus tier runs about $5 per million input tokens and $25 per million output; the Haiku tier runs $1 and $5. That is a 5x difference on every single call, and you pay it whether or not the extra capability changed a single answer.

The leaderboard optimizes for one axis: raw capability on hard problems. Production optimizes for three, and they pull against each other.

The triangle: you optimize two

Every model decision lives inside a triangle of cost, latency, and quality, and like most engineering triangles you get to pick two corners to hold firm while the third gives.

Want the highest quality at the lowest cost? You will wait for it: batch the work, run a bigger model, accept that a response takes seconds not milliseconds. Want the highest quality with the lowest latency? Open your wallet, because fast inference on a top-tier model at premium pricing is exactly the corner you are buying. Want it cheap and fast? Then quality is the variable, and your job becomes finding the smallest model that still clears your quality bar for this specific task.

Naming which two corners a given feature actually needs is most of the work. An autocomplete suggestion that appears as the user types is a latency-and-cost problem where "good enough" quality is fine. A contract clause that a lawyer will rely on is a quality problem where a few extra seconds and a few extra cents are noise. Treating both with the same model is how you end up simultaneously overpaying for the autocomplete and under-serving the contract.

Match the tier to the difficulty

Once you stop defaulting to the biggest model, the question becomes concrete: what is the easiest model that reliably does this job? Difficulty is a property of the task, not of your ambitions for it.

Classification, extraction, short summarization, and format conversion are usually well within reach of a small, fast model. When I built Ledger, the invoice-extraction tool, the model's job was narrow and checkable: pull structured fields, then validate them against arithmetic the code owns. That does not need a frontier reasoner; it needs a competent extractor and a math check downstream. Contrast that with Scope, the competitor-analysis agent, which plans multi-step research, weighs conflicting sources, and decides what to look at next. That is genuine reasoning under ambiguity, and it earns the more capable tier.

Modern models give you a second dial beyond the model name: an effort or thinking setting. You can run a mid-tier model at higher effort for a hard subtask instead of jumping to the most expensive model, and often land at better quality-per-dollar than either extreme. The instinct to reach straight for the biggest model skips this entire middle, which is usually where the best answer lives.

Routing and cascades: cheap first, escalate on doubt

The real win is not picking one model. It is refusing to. Route each request to the smallest model that can handle it, and escalate only the ones that need it.

A cascade is the simplest form. Send every request to the cheap model first. Have it return not just an answer but a signal of its own confidence — a structured field, a self-rated score, or a downstream validation the answer either passes or fails. Accept the confident answers; escalate the rest to a stronger model. Take a ticket-triage system handling 12,000 tickets a month at roughly 800 input and 150 output tokens each. Sending all of it to the top tier costs about $93 a month. Sending all of it to the cheap tier costs about $19 — but on the genuinely ambiguous tickets, quality slips. Run the cascade instead: the cheap model handles the 85% it is sure about, and the 15% it flags escalates to the strong model. That is roughly $19 for the full pass plus $14 for the 1,800 hard tickets, about $33 a month, with top-tier quality reserved for exactly the cases that needed it.

The benchmark tells you which model is best at the test. Your eval set tells you which model is best at your job. Only one of those numbers shows up on the invoice.

Cascades have a failure mode worth naming: the cheap model must know when it is out of its depth. A confidently wrong small model is worse than no cascade at all, because it never escalates. This is why the confidence signal has to be grounded in something real — a schema validation, a retrieval hit, an arithmetic check — not the model's unexamined opinion of itself.

Build your own eval set

None of the above is decidable from a public benchmark, because none of it is about the average case on someone else's test. It is about your inputs. So build the only leaderboard that matters: fifty real examples from your actual workload, with the answer you would accept for each.

Fifty is not a rounding error of a number; it is the point. It is small enough that you can assemble it in an afternoon by pulling real cases — including the weird ones, the empty inputs, the malformed inputs, the adversarial ones — and large enough to expose a model that is quietly failing 10% of the time. Run every candidate model, at every effort level you are considering, against that set. Now the comparison is denominated in the units you care about: how many of my fifty did it get right, how much did that cost, how long did it take.

This set becomes a permanent asset. When a new model launches, you do not read the announcement and guess; you run the fifty and get an answer in minutes. When I evaluate whether Lumen, the plain-English analytics tool, should move a text-to-SQL step to a cheaper model, the eval set is a bank of real questions with known-correct SQL. The demo runs on SQLite, but the pattern generalizes to any warehouse: the question is always "does the cheaper model still produce queries that return the right rows," and only the eval set can answer it.

Prompt caching and context discipline

Before you conclude a task needs a bigger model, check whether it just needs a cheaper prompt. A large fixed preamble — system instructions, few-shot examples, retrieved documents — sent on every request is money spent re-reading the same tokens. Prompt caching bills those cached tokens at a fraction of the base input price on reads, against a one-time write premium, so a stable prefix that repeats across requests can cut input cost substantially. The catch is that caching is a prefix match: a single changed byte near the front — a timestamp injected into the system prompt, a reordered JSON key — invalidates everything after it. Freeze the stable part, put the volatile part last.

Context discipline is the other half. In Verba, the document assistant, the temptation is always to stuff more retrieved passages into the prompt "just in case." But every irrelevant passage is tokens you pay for, latency you add, and noise the model has to work around. Retrieving five sharp passages beats retrieving twenty loose ones on all three corners of the triangle at once. Sometimes the fix for a struggling model is not more model; it is less prompt.

Plan for churn: isolate the choice behind an interface

Whatever you pick today will be wrong within a year, and that is fine if you built for it. New models ship constantly, prices move, a tier that was the sweet spot gets undercut. The teams that suffer are the ones with a hardcoded model string sprinkled across forty call sites.

Put the choice behind a thin seam. Each task names what it is; a single config maps that name to a model, an effort level, and a vendor. Call sites ask for the task, not the model.

// One place owns the mapping. Call sites never name a model.
async function classify(ticket: string): Promise<Triage> {
  return llm.run({
    task: "ticket-triage",   // config resolves this to model + effort
    input: ticket,
    schema: TriageSchema,
  });
}

With this seam in place, adopting a new model is a config change and a rerun of your fifty examples, not a refactor. Swapping a cascade in behind ticket-triage touches one file. The model becomes what it should have been all along: a replaceable component, not a load-bearing wall.

Stop shopping for the best model. Start measuring the right one — on your task, at your budget, against your own fifty examples — and let the leaderboard belong to whoever is still impressed by it.