Build the evaluation set before you pick the model

A vendor demonstrates a document assistant at 94% accuracy. Eleven weeks after go-live the same system still measures 94% and the client wants it switched off. Both numbers are honest. Only one was measured on the client’s documents.

The eval set outlives the model, the prompt, the chunker and usually the vendor. Build it before any of them, from the corpus that will actually arrive, with someone who knows which mistakes cost money.

The pages a public benchmark quietly excludes are the ones you will fail on

Public benchmarks are curated in a specific direction: one canonical version per document, digital text rather than scans, one script per page, questions written against passages known to exist. Every one of those conveniences is removed on contact with a real share drive.

What arrives instead is revision 4 and revision 5 of the same SOP in adjacent folders; annexes scanned at 200 dpi in 2011; delivery notes photographed on a phone at an angle, with a thumb; and — routinely here — a Bulgarian maintenance form with Cyrillic field labels and Latin part numbers, the occasional Cyrillic С standing where a Latin C belonged, which no string comparison will reconcile.

Sample from the mail, not from what is easy to label. Stratify by the things that actually vary — layout family, scan quality, language, document age — and record the case count per stratum so the score can be read per stratum rather than blended. For a production gate, 150–400 labelled cases is the working minimum, and composition matters more than count: if 40% of monthly volume is one supplier’s clean digital PDF, that layout should not occupy 40% of the set, or the headline is dominated by the case that was never at risk.

The opposite error is just as real: a set built entirely from the ugly tail scores low, stays low, and says nothing about throughput. Keep both, and never average them without saying so.

Accuracy is one number covering two errors that do not cost the same

A false accept and a false reject are not the same event. In extraction, a wrong VAT number written into the ERP surfaces as a reconciliation exception weeks later, after payment, and costs someone a morning. A document sent to the review queue costs under 60 seconds when the source region is highlighted for the reviewer.

Where the ratio is 50:1 or worse, a system two points less accurate but biased toward abstention is the better system, and accuracy ranks the two exactly backwards. Report the confusion matrix, and state the target as a false-accept rate at a fixed review load: “under 0.5% false accepts with no more than 20% of documents queued” is a specification. “96% accurate” is a mood.

Abstention deserves its own tile. On a real corpus 5–15% is healthy; under 2% means the system is guessing and someone tuned it to look decisive; over 25% means retrieval is broken and the generator is taking the blame.

The failure taxonomy comes from reading traces, not from a framework

Ragas and DeepEval will compute faithfulness, context precision and tool-call accuracy for you. Neither can tell you what goes wrong in your domain, because that is a fact about your documents. So the first week is open coding: read real failed interactions, name each failure in your own words, cluster afterwards rather than before. It converges faster than people expect — after roughly 30 traces, new failures stop appearing. The near-universal first finding is that 60–80% of user-visible failures are retrieval misses, not generation failures: Barnett’s FP1–FP3 (the answer never reached the context) against FP4–FP7 (the model had it and mishandled it). The fixes share nothing — chunking and reranking on one side, prompt and context construction on the other — and conflating them is the most reliable way to waste a quarter.

Labels come from one named domain expert: binary pass/fail with a written critique on every fail. Not a 1–5 scale — a 3 is uninterpretable, and two annotators will not mean the same thing by it.

This is the phase clients least want to pay for: several days of a senior person reading transcripts, producing no demo. It is also the highest-value work in the engagement.

Near-duplicates are the test, not the noise

Two revisions of the same SOP differ by 2% of their text and sit almost on top of each other in embedding space. Ranking cannot separate them and should not be asked to. The eval item is not “does it find the SOP” but “does it find revision 5, on a question whose answer changed in revision 5” — and the fix is a metadata filter on effective date and site that makes superseded revisions unreachable rather than lower-ranked. Budget for the side effect: restrictive filters fragment an HNSW graph, which is why filterable HNSW and ACORN exist.

The same near-duplication bites the set itself. A document appearing both in the prompt’s few-shot examples and in the eval set inflates the score, and it will not reproduce. De-duplicate across that boundary by document ID, not by hash — the scans will not hash alike.

A judge has to be scored before it is allowed to score

LLM-as-judge is the only way to run a few hundred cases on every commit, and it is trusted far past its evidence. MT-Bench and Chatbot Arena established that strong judges reach over 80% agreement with human preferences — roughly the level at which humans agree with each other. That is the ceiling, not the starting point, and it comes with position bias, verbosity bias, self-enhancement bias and weak reasoning on anything arithmetic.

So measure the judge the way you measure the system: true-positive and true-negative rate against the human labels, per failure mode, targeted at 0.85 or better on each. Raw agreement is worthless under class imbalance — if 8% of outputs fail, a judge that passes everything scores 92% and detects nothing.

Two consequences people skip. Pin the judge’s model ID as strictly as the production model’s, because a silent provider upgrade moves your scores without touching your system. And avoid letting a model family grade its own output; self-enhancement bias is measured, not theoretical. Judge runs are latency-insensitive, so send them through the Batch API at 50%.

A set is only a gate if something blocks on it

Deterministic assertions — schema validity, citation present, forbidden terms, latency — run on every commit and cost nothing. The judged subset must finish in under ten minutes to be allowed to block a merge, with the full set nightly. The threshold is a regression delta: no more than a 1–2 point drop, on every prompt, model, index and chunking change, not only the ones that feel risky.

For anything agentic, single-pass scoring is the wrong instrument. τ-bench found frontier models above 50% single-pass and below 25% at pass^8 in the retail domain — succeeding on all eight independent trials of the same task. Inconsistency, not incapability, is what production exposes. Run each case five to ten times and report pass^k.

If the system serves Bulgarian, it needs its own set and its own reported score. Models benchmarked mainly on English typically run 5–15 points lower on Bulgarian, and a blended multilingual number hides that completely.

Where this stops applying

A frozen eval set rots. Every production correction is a candidate case, but a set fed only on failures drifts adversarial and last quarter’s score stops being comparable. Version the set, keep a frozen core for trend, and date every addition.

And an eval suite answers whether the system works, never whether it was worth building. That question is cost per resolved task against the process it replaces, and no golden set contains it.

Finally, the boundary about us. Palamed has four deliverable engagements: a European car marketplace with 300,000+ listings, the Ministry of Education and Science digital dictionary at beron.mon.bg, email automation for a beauty brand where the client measured roughly a 60% cut in outreach time by their own method, and an NLP module that classifies incoming trading questions from email and Instagram and drafts the answer from the product and pricing data the team already maintains, where the client counted roughly 85% less manual typing on repeat questions and a person approves every reply before it sends. We have not run a labelled eval suite through a client’s CI. The above is the published pattern — the 0.85 alignment bar, the 30-trace convergence, the pass^k finding — and the standards under it. Anyone quoting 99% accuracy is telling you their eval set was written by whoever wrote the prompt.