Definition
How do you evaluate AI vendors before you buy?
To evaluate AI vendors, check training data, bias tests, privacy terms, integration, production accuracy and customer references before you sign.
By the Foxnut team · Updated
How do you evaluate AI vendors?
To evaluate AI vendors, check six things: where the product’s training data comes from, how it has been tested for bias, what the data-privacy terms let the vendor do with your data, how it connects to the systems you already run, how it behaves in production on your work and not in a demo, and what current customers say when you call them. You are evaluating a product you will have to run after the sales team has moved on. So these checks measure what owning it will be like, not how impressive it is.
We are Foxnut AI, a senior-only practice in Bengaluru and Paris. We build AI systems and hand them over for a living, which in a vendor evaluation puts the studio on the buyer’s side of the table. There’s a link to talk to us at the end of this guide. Let’s get your evaluation done first, with the questions we ask when a client is weighing a purchase, written down so you can ask them alone.
What is an AI vendor?
An AI vendor sells a product: software with model behaviour inside it, priced as a licence or a subscription, run by your team after onboarding ends. A consultant or studio sells judgment and a build: a person or firm you hire, whose output your organisation ends up owning. The two purchases fail differently, so they are vetted differently.
A firm is vetted on artefacts, mechanisms and what it refuses to do. The checks for vetting the person or firm you hire are their own list, and almost nothing on it transfers here. A product is vetted on what is inside it, what the contract says about your data, and how it holds up in production.
Confusion to avoid: buyers who apply the consultant test to a product end up judging the salesperson. Buyers who apply the product test to a consultant end up asking a two-person studio for a SOC 2 report on software that does not exist yet. Know which purchase you are making before you evaluate anything.
How do you vet an AI vendor?
You vet an AI vendor by asking for each check below in writing before a contract is signed. The middle column is the artefact to request. The right column is the failure the check exists to catch.
A vendor that cannot produce the artefact is not necessarily hiding something. But the absence is itself the answer, and it belongs in the risk column of your decision, not in a follow-up call after signature.
| The check | What to ask for | What it protects you from |
|---|---|---|
| Training data provenance | A written account of what the model was trained or tuned on, and whether your data joins that pool | A product whose behaviour nobody can explain, and your records training a competitor’s next version |
| Bias testing | The tests run before release, on populations like the ones your system will decide about | Discovering in production that the product fails systematically for a group of your customers |
| Data-privacy terms | The clauses on data use, retention and deletion, read by whoever owns privacy internally | Terms that quietly grant the vendor more of your data than the product needs |
| Integration specs | The documented interfaces, and a named answer to “what does connecting this to our stack take” | A purchase that becomes a six-month integration project priced after the fact |
| Production accuracy and latency | Numbers measured on work like yours, with the conditions stated - not a benchmark from the launch post | A demo that was the product’s best day rather than its average one |
| Reference calls | Two current customers of similar size, chosen by you from a list, not by the vendor | Buying on the strength of the one customer for whom everything went well |
What questions should you ask an AI vendor at the contract stage?
Ask for a training and re-training plan, and ask what the system does as the organisation grows. Our own record of vendor and build negotiations shows the same pattern on almost every deal: at the contract stage, clients ask for extended support. It is the wrong ask. Support does matter, but it is already built into the commercial agreement, so negotiating hard for it wins nothing that was not already won.
The first thing clients should ask for, and almost never do, is a training and re-training plan for the employees who will use the system every day. That means a plan and not a launch workshop, because the people trained at go-live change roles and leave, and the system’s usefulness walks out with them.
The second is the potential expansion of the system against the growth the organisation actually expects: what the product does at twice the volume, in the second market, with the next team onboarded, and what that costs. A vendor’s answer to the expansion question is also the cheapest preview of what being their customer is like in year two.
Cost questions get asked, and one deserves sharper wording than it usually gets. Clients ask for advice on reducing AI token costs, and any competent vendor or builder covers that as a matter of course. The version worth putting to a vendor directly: what in the product’s own design keeps token consumption controlled.
Uncontrolled development AI can balloon token costs out of proportion, and a product built that way passes the bill to its buyer. A vendor who can name the mechanism has thought about your bill. One who answers with a pricing page has not.
Is evaluating an AI vendor the same as evaluating LLM output?
No. Evaluating LLM output, meaning whether a model’s answers are correct, safe and stable over time, is testing work. It is done with structured test material against a running system, and it continues for as long as the system runs. This page’s question comes earlier and is about the supplier: whether this company, this contract and this product are the thing to buy at all.
The two meet at exactly one point, and it is worth using. Ask the vendor how you would test this on your own cases before signing. The response tells you more than most reference calls. A confident product comes with a way to be tried, and a fragile one comes with reasons a trial is not representative.
What can a vendor evaluation not tell you?
A vendor evaluation cannot tell you whether you should be buying at all. That depends on the problem being real, the data being reachable and someone internally owning the outcome, which is your own readiness and not the vendor’s quality.
It also cannot tell you what running the product will cost your organisation in attention, retraining and change. No vendor document states that, and every operating team discovers it.
An evaluation done honestly ends in one of three places. Buy, with the artefacts filed and the two contract asks made. Do not buy, with the failing check named. Or not yet, because the evaluation kept surfacing questions about your own side of the table and not the vendor’s. The third result is the most common one in our experience, and it means the evaluation is working.
Who can help you run an AI vendor evaluation?
We can, and we are the other kind of purchase on this page, a firm you hire and not a product you license. We build AI systems and hand them over, so we care about what your team has to run after the sales call. The people who sign our proposal do the work. There is one tier, and we designed, made and launched our own products first.
If you are somewhere in the middle of this evaluation right now, we will walk through it with you live, on a call, at no cost. A senior practitioner reviews your setup, with no RFP and no pitch deck. You can read about Foxnut Consulting’s senior-only practice first, or book the time.
Foxnut AI takes on work like this from Bengaluru and Paris. If you want to see how we do it before you talk to anyone, here is how an AI engagement is scoped and priced.