FOXNUT AI

Definition

How to evaluate LLM output with an eval set

You evaluate LLM output by running real cases with expected outcomes after every change. This catches failures that human review misses.

By the Foxnut team · Updated

How do you evaluate LLM output?

You evaluate LLM output with an eval set: a fixed collection of real cases from your own operation, each with an expected outcome, run against the system on demand and re-run after every change. Evaluation is then a comparison between what the system produced and what the case says it should produce. Whether the output is good stops being an opinion and becomes a list: which cases pass, which fail, and what changed since the last run.

We are Foxnut AI, a senior-only practice in Bengaluru and Paris that builds AI systems and hands them over. There’s a link to talk to us at the end of this article. Let’s get your evaluation working first.

Confusion to avoid: two things feel like evaluation and are not. Reading a handful of outputs and judging them by eye is sampling, and the next section is about the failures it reliably misses. A public benchmark score is a fact about a model on someone else’s cases, not about your system on yours. A model that tops a leaderboard can still fail your invoice formats every day.

The eval set is also one of the tests a client team must be able to re-run alone, inside the handover method these tests belong to. A team that cannot evaluate the system’s output cannot safely change the system.

What failures does human review miss in LLM output?

In our experience, the failures humans most often miss are authentication and data-sanity failures. The case for evals over human review is not speed. The two see different failures, and human review sees the wrong ones.

A person reads LLM output for plausibility: tone, fluency, whether the answer looks right. LLM output is fluent even when it is wrong, so plausibility is exactly the property that survives a real failure.

In these failures, the system is quietly working against the wrong account or with expired access, or the data feeding it is stale, incomplete or from the wrong source. The output reads as well as it ever did. A reviewer approves it because nothing about the text says anything is wrong. The defect is upstream of the words.

How does an eval catch what human review misses?

An eval does not read for plausibility. It checks the output against an expected outcome computed from known inputs, and an answer built on the wrong account or bad data does not match, however well it reads.

This is why our position is that strict, well-designed evals covering the edge cases are what keep a production system running. In our view they are the only reviewer that never gets talked into a fluent wrong answer.

What does a handed-over eval docket contain?

A typical eval docket covers all the major and minor use cases plus the edge cases, and it is backtested on verified company data. Backtested means the expected outcomes are checked against records the company knows to be true, so the bar is real before the system is measured against it.

An eval set worth handing over is unique to the company it tests. It is built from that company’s needs and workflows: its documents, its customers, its failure history. That is why you cannot buy a docket generic or copy one from another engagement, and why receiving one gives you something no benchmark supplies.

Layer of the docketWhat it coversWhat a failing run there means
Major use casesThe workflows the system runs every day - the reason it existsThe system is broken where the business feels it first
Minor use casesThe occasional paths: the month-end report, the rare request typeA quiet break that would otherwise surface weeks later, in production
Edge casesThe inputs that predictably go wrong, authentication and data-sanity checks among themExactly the failures human review misses
Backtested expected outcomesResults verified against real company data, not invented examplesThe bar itself is wrong, and the docket needs correcting before the system does

The last row is the one that makes the other three trustworthy. An eval set whose expected outcomes were never verified against real data tests the system against a guess, and a system that passes it has proven nothing.

What are LLM evals?

LLM evals, as practitioners use the term, are the runs described above: the system’s output on known cases compared against expected outcomes, repeatable at will.

LLM evaluation in the academic sense, meaning benchmark suites, leaderboards and capability scores, answers a different question: which model to reach for. It answers that about models in general, not your application in particular. The two are routinely confused because they share a word.

How do you test an LLM app?

You test an LLM app more widely than either of those, because an application is more than its model. It authenticates against accounts, pulls data from sources, calls tools, and hands results to people or other systems.

You test the deterministic parts of that pipeline the way software has always been tested. The model-driven parts are where conventional testing breaks down: the same input can produce differing output, and there is no compiler to object to a bad change. The eval set fills that gap.

In a working system the two meet in one habit: nothing that changes the system’s behaviour goes to production without the docket run against it. That is what makes change safe.

Who owns the eval docket after handover?

The team that owns the system owns the docket. Handing it over means the client team receives the cases, the expected outcomes and the means to run them in their own environment, not a report of the last run.

Within a complete AI handover the eval set is one artefact among several, and the full list sits with the artefact inventory this one belongs to. It is the one we treat as inseparable from the system itself, because a system without its docket can be operated but not safely changed.

Does the eval docket keep growing after handover?

Yes. Workflows change, new edge cases arrive with new customers and new data, and each production surprise that was not in the docket belongs in it by the following week. A docket that keeps growing is the record of the team using its ownership.

What should you ask an AI vendor about evaluation?

Ask three things before any build starts: how the system’s output will be evaluated, who will hold the eval set at the end, and what data the expected outcomes will be verified against.

A vendor with real answers is offering a system you can run. A vendor without them is offering output you will have to keep taking on trust.

At Foxnut AI the people who sign the proposal do the work. There is one tier, and we designed, made and launched our own products first. We build AI systems and hand them over with the docket, so your team can change the system without us.

The first step is a free expert audit: a senior practitioner reviews your setup, with no RFP and no pitch deck. You can read what Foxnut Consulting does first. Ask us these three questions before your next AI vendor call, ours included. We will answer them live.

Foxnut AI takes on work like this from Bengaluru and Paris. If you want to see how we do it before you talk to anyone, here is how an AI engagement is scoped and priced.