How to choose an AI development partner that ships, not one that demos

By the second sales call, every AI development company has shown you a demo, and the demos look alike. A chat window answers a question about a sample document, or an agent fills in a form while someone narrates. None of it tells you what you need to know, which is whether this team will put something into your business that is still right in month three. These are the questions we would ask if we were the buyer, and the answers that should worry you. We say briefly where we stand on each, but the questions work on anyone, including us.

Ask	A good answer	Worry if
What can we use, and when?	A date in days or weeks, on your real cases	The first deliverable is a report
How will you measure quality?	An evaluation set of your cases, scored on every change	The answer is all about the model
Who owns the code and prompts?	Your repository from week one, with prompts and evaluation sets named in the contract	You get the code at the end
Which of our data leaves our systems?	Every copy named, and permissions widened one step at a time	The agent starts with every permission
What does the price fix?	A short first stage on your data, then a fixed price	A fixed quote before anyone has seen your data
Who runs it after launch?	A monthly fee with monitoring, or a handover with a runbook	Nobody is named for model changes

Ask for a working version on your data, early

A demo runs on inputs the vendor chose. It shows that the model can handle the easy case, which is rarely in doubt now. What is in doubt is your data: the scanned PDF with a coffee ring on it, the customer email in two languages, the record that exists in two systems under different names. The only test of that is a working version running on your own cases.

So ask what you will be able to use, when, and on what. A good answer names a date in days or weeks rather than a phase, and names the cases it will run on. It also says what the vendor needs from you to get there, which is usually a sample of real data and an hour with the person who does the work today. If the first deliverable is a discovery report or an architecture document, you are paying the vendor to learn your business before you have any evidence they can build for it.

Judge the early version on the right things. It will be rough. The question is not whether it looks finished but whether it ran on your cases, whether the vendor can tell you which ones it got wrong and why, and whether the failures look fixable. A team that shows you its failures in the first week will show you them in the tenth.

Our own answer: a first working version on your real cases within a week, and a fixed price for what comes next once you have used it.

Ask how they will measure quality

An AI feature’s correctness varies by input, so “we test thoroughly” means nothing until you know what it is tested against. The answer you want is an evaluation set: real past cases from your business with answers someone on your side has checked, a harness that runs the feature against them, and a score per field or per decision, recorded on every change to the prompt, the model or the code.

The follow-up questions show whether the answer is real. Who labels the cases, and does anyone from your team review the labels? How many cases are there, and how are the awkward ones chosen? What score has to be reached before the feature goes live, and who decides? What happens to the score when the model provider ships a new version? If they can show you an anonymised evaluation report from earlier work, ask to see it. How we build and maintain one is in building an evaluation set from past cases.

Be wary of answers that are all about the model. “We use the latest model” or “we fine-tune on your data” describes an input, not a measurement. In the work we have done, the choice of model has mattered less than the definition of a correct answer, and the definition is what an evaluation set writes down. Before an AI feature of ours goes live, it is measured against real cases from your business, so its accuracy is a number rather than an impression.

Who owns the code, the prompts and the evaluation sets

In an ordinary software project the asset is the code. In an AI project it is also the prompts, the tool definitions, the evaluation set, the labelled examples and the configuration that says which model runs where. Those took the longest to get right, and they are what a vendor is most tempted to keep. A contract that assigns “the deliverables” to you while the prompts stay in the vendor’s platform gives you code that does not work without them.

Ownership has to be written down. Under US copyright law, software written by an independent contractor is generally not a work made for hire: the definition covers employees and a short list of commissioned works, and software is not on the list. Ownership moves to you through a signed written transfer. Other jurisdictions have their own rules, so have a lawyer read the clause, and check that it names prompts, evaluation data and configuration explicitly.

Two practical checks matter as much as the clause. The first is where the code lives. If it is in a repository in your organisation from the first week, you can see what is being built and nobody has to remember to hand it over at the end. If you are promised the code at the end, you will get it at the end, in whatever state it is in. The second is whose name is on the model provider and hosting accounts. If they are the vendor’s, your data flows under the vendor’s terms with the provider, and the access is theirs to switch off.

Ask too what is licensed rather than assigned. Some agencies build every client project on an in-house framework. That can be fine if you know which parts you will own, which you will license, and whether the licence survives the end of the relationship.

With us, the code lives in your repository from day one, and you own it, including the prompts, the evaluation sets and the infrastructure configuration.

How they handle data and permissions

Ask which of your data leaves your systems, where it goes and under whose terms. A typical AI feature sends text or documents to a model provider and may keep some of it in a vector index, a log or an evaluation set. Each is a copy you should know about. If you are subject to the GDPR, a vendor processing personal data on your behalf needs a contract that meets Article 28, which requires, among other things, your authorisation before they engage sub-processors such as a model provider, and that they delete or return the personal data at the end of the work, at your choice. Ask where the evaluation set will live afterwards, because it is made of your real cases.

Then ask what the system will be allowed to do. An agent that can read your inbox, write to your CRM and send email is a bundle of permissions, and giving a model more of them than the job needs is Excessive Agency, one of the OWASP top ten risks for LLM applications. A careful answer widens permissions in steps, as below, and logs every action with its inputs. We described that sequence in putting an AI agent inside a workflow you already run. If the agent reads text written by outsiders, such as customer emails or web pages, ask how it handles instructions hidden in that text; prompt injection defences for tool-using agents covers what a real answer contains.

The vendor’s own access belongs in the same conversation. Ask what they need on day one, whether read-only access to a sample will do for the first version, and how their access is removed when the work ends.

Fixed price or time and materials

Both pricing models fail in predictable ways, and one shape works for most AI projects.

Pricing	What happens
Fixed price before the vendor has seen your data	Padded to cover what they do not know, or an argument about change requests the first time the data is messier than the proposal assumed
Time and materials with no milestones	All the uncertainty sits on your budget, and the vendor has no reason to push back on scope
A short first stage, then a fixed price	The first stage produces a working version on your data and answers the questions that make early fixed prices dangerous: how messy the data is, how hard the integrations are and how accurate the AI part can get. After that, a fixed price for a scope both sides can now describe can be an honest one

That is how we price: fixed, after the first week.

Whatever the model, read what the price actually fixes. Is there a written scope with exclusions? Is acceptance defined, ideally as a score on the evaluation set rather than a demo on the day? How are changes priced? And what will the system cost to run each month, which is your bill rather than theirs? What an AI agent costs to run shows how to estimate that, and what an AI MVP costs in 2026 goes through the line items inside a build estimate.

Handover, and who runs it after launch

An AI feature does not stay finished. Suppliers change their templates, customers start writing differently, and model providers retire versions on a schedule. Anthropic, for example, gives at least 60 days’ notice before retiring a publicly released model, and OpenAI keeps a similar deprecations page with shutdown dates. Someone has to run the evaluation set against the replacement model and fix whatever drops. Ask who that will be.

There are two honest answers. Either the vendor keeps running the system for a defined monthly fee, with monitoring they can show you, or they hand it over, and the handover is a set of artefacts rather than a meeting. The artefacts are the code, the prompts, the evaluation sets with the harness that runs them, the monitoring and a runbook for deploys, incidents and model changes. A good test before the final invoice is to have someone on your side deploy a small change using only the runbook.

Handover can be the plan from the start. With EroPlay.ai, we built the MVP as technology partner to find out whether people wanted the product; once the demand was clear and it needed a full-time CTO, we handed it over, as planned. A partner should be able to tell you which ending it expects for your project before the work begins. Ours is either of the two: the code, prompts, evaluation sets and runbook in your repository, or we keep operating the product for a monthly fee.

Red flags when hiring an AI development company

Red flag	What it tells you
A strategy deck written without access to your data	If it maps AI opportunities across your business and nobody has looked at a single real case, it describes AI in general. An assessment is worth paying for when it includes a small evaluation on your data, which is what we think AI consulting should consist of
Evaluation described in adjectives	Accurate, reliable, production-grade, with no number and no set of cases behind any of them
A demo as the deliverable	Weeks of work, and what you receive is a recorded walkthrough or a hosted prototype you cannot run yourself
A different team after the contract	Ask to meet the engineers who will write the code, and who is responsible when code written with AI tools is wrong
Agreement with everything	A partner who never tells you a feature will not survive launch, or that your data is not ready, is selling hours

Most teams use AI coding tools now, and we use agents for much of our own research, code, tests and reviews. What matters is that an engineer reads every result and answers for it.

A test you can run before you sign

The cheapest way to compare two candidates is to give both the same twenty real cases from your business, anonymised if they need to be, with the answers you already know, and ask what each can show you a week later. What comes back, and the questions each team asks about the cases on the way, will tell you more than either pitch. If you would like us to be one of the two, write to us with the cases or a description of them. The custom AI apps page describes how the work runs after that first week.