To choose an AI development partner, judge them on the unglamorous parts rather than the demo: how they pick the problem, how they prove the system works, where your data goes, who owns what at the end, and what it costs to run once they leave. Ask every shortlisted agency the same twelve questions below and compare the answers side by side. Strong candidates give specific, sometimes slightly uncomfortable answers. Weaker fits tend to give confident generalities.
The demo is the least informative part of any AI pitch. Almost anyone can get a large language model to produce an impressive result on a hand-picked example in an afternoon. What separates a system you can rely on from a pilot that quietly dies is everything around the model: evaluation, integration, error handling, cost control and handover. The questions below are designed to bring those things to the surface before you sign anything.
What you are actually buying
When you hire an AI studio or consultancy, you are not really buying a model. Models are rented by the token from a small number of providers, and the leading options change every few months. You are buying three other things: judgement about which problem to aim at, engineering that wraps a probabilistic component in deterministic guard rails, and a working system your own team can operate afterwards.
That framing changes what a good answer sounds like. A partner who talks mostly about which model they prefer is answering a question you didn’t ask. A partner who talks about your process, your data, and what happens when the output is wrong is answering the right one. Keep that distinction in mind as you go through the list.
Questions about the problem and scope
1. Which problem would you solve first, and why that one?
The best first project is usually narrow, high-volume, rule-heavy and cheap to get wrong occasionally: invoice coding, ticket triage, extracting fields from supplier documents. A good partner will ask about volumes, current handling time and error rates before recommending anything, because those numbers decide whether the project pays back. If you want to do this homework yourself first, our guide to identifying which business processes to automate first walks through the scoring.
2. When would you tell us not to use AI?
Plenty of problems are better solved with a rules engine, a better form, a direct integration or an existing SaaS product. A partner worth hiring can name situations where they would advise against an LLM, and ideally will spot one in your own shortlist. If every problem you describe comes back as an AI problem, treat that as a signal to probe further. Our comparison of custom software versus off-the-shelf for Irish SMEs covers the same trade-off from the buying side.
3. What will exist at the end of the first four to six weeks?
The answer you want is a thin vertical slice running on your real data: one document type, one queue, one workflow, end to end, with a measured accuracy figure. A strategy deck and a roadmap have their place, but they are not evidence that anything works. Ask for the concrete artefact and how you will be able to inspect it yourself.
Questions about quality and failure
4. How will we know it works?
This is the most revealing question on the list. A serious answer involves an evaluation set: a few hundred real examples from your business with known correct outcomes, scored automatically every time a prompt, model or piece of code changes. It should also involve a baseline, meaning how accurate and how fast the current human process is, so that the comparison is honest. Frameworks such as the NIST AI Risk Management Framework describe this measurement discipline in more formal terms, but you don’t need the paperwork to ask the plain question: show me the number, and show me how you calculated it.
5. What happens when it is wrong?
Every AI system will be wrong some of the time. What matters is whether errors are caught, routed and learned from. Look for confidence thresholds that send uncertain cases to a person, an audit log of every decision, and a simple way for staff to approve, correct or reject an action before it touches a customer or a ledger. We wrote up how we make our own AI agents easy to steer from a phone in Human-in-the-Loop Action Tags; whatever the mechanism, a partner should be able to show you theirs.
6. What happens when the model changes underneath us?
Providers retire model versions, change defaults and ship new releases on their own schedule, as the GPT-6 rollout across ChatGPT tiers illustrates. A system built carelessly can change behaviour overnight without anyone touching it. The answer you want covers pinned model versions, re-running the evaluation set before any upgrade, and a thin abstraction layer so that switching provider is a configuration change and a test run rather than a rewrite.
Questions about data, security and compliance
7. Where exactly does our data go?
Ask them to trace every hop: your systems, the partner’s development environment, the model provider, any vector database, and any logging or monitoring tool. For each one you want to know the region, the retention period, whether data is used for training, and which subprocessors are involved. Under GDPR you remain the controller, so this is your problem as much as theirs; the Data Protection Commission publishes guidance on processor relationships. Our article on data security and privacy in AI projects lists the controls worth insisting on.
8. How does this use case sit under the EU AI Act?
The EU AI Act (Regulation 2024/1689) is risk-based. Most internal back-office automation falls into lower-risk categories, but uses such as screening job applicants or assessing creditworthiness are listed as high-risk and carry real obligations, and customer-facing chatbots have transparency duties. A competent partner should be able to give you a first-pass classification of your use case and explain what follows from it, while being clear about where you need your own legal advice. If you want a management-system structure around this, ISO/IEC 42001 is the relevant standard.
9. How will it integrate with what we already run?
Most of the effort in a real project sits in the integration, not the model. Ask whether they will connect through APIs, database access, file drops or screen automation, and why. A sensible pattern is read-only access first, then write access behind an approval step once accuracy is proven. Our piece on integrating LLMs into legacy enterprise systems goes into the options and their failure modes.
Questions about ownership, cost and handover
10. Who owns the code, prompts, evaluation sets and accounts?
The contract should assign the code to you, and it should explicitly include prompts, evaluation data and configuration, because in an AI system those are as much a part of the product as the code. Just as important, the model API keys, cloud accounts and repositories should live in your tenant and be billed to you. If the system runs on the partner’s accounts, leaving becomes a migration project rather than a handover.
11. What will it cost to run per month, not just to build?
Ask for a per-transaction estimate: roughly how many tokens each document or ticket consumes, multiplied by your volume, plus hosting, monitoring and an honest figure for maintenance time. Ask for that at today’s volume and at three times today’s volume. Inference costs can be modest or surprisingly large depending on design choices like model size, caching and how much context is sent each call, and a good partner will have thought about which of those levers matter for you.
12. What does handover look like, and who maintains it?
You want runbooks, a dashboard showing accuracy and cost over time, an automated test suite including the evaluation set, and a named person on your side who has run the system before the partner steps back. Ask what a maintenance arrangement looks like if you want one, and what it looks like if you don’t. Systems handed over without tests and documentation tend to accumulate exactly the kind of debt described in our guide to reducing technical debt with automated systems.
How to read the answers
Run the twelve questions as a structured call with the engineers who will actually do the work, not only the person selling it, and write the answers down so you can compare candidates on the same page. The table below is a rough guide to the difference between an answer that needs a follow-up and one that suggests the partner has done this before.
| Area | Answer that needs a follow-up | Answer that suggests real depth |
|---|---|---|
| Problem selection | Recommends a solution before asking about volumes or costs | Asks for volumes, handling times and error rates first |
| Evaluation | “The model is very accurate” | A labelled test set, a human baseline and a reported score |
| Failure handling | Errors are rare, so not much is planned | Confidence thresholds, review queues and an audit log |
| Model changes | Always uses the latest model automatically | Pinned versions, regression tests before upgrades |
| Data | “It’s all secure” | Every hop named with region, retention and subprocessors |
| Ownership | Runs on the partner’s accounts for convenience | Your tenant, your keys, IP assigned including prompts |
| Running cost | Only quotes the build | Per-transaction estimate at current and higher volume |
| Handover | Documentation provided at the end | Runbooks, tests, dashboards and a trained owner on your side |
Two practical points round this off. First, a short paid discovery phase is a cheap way to test the relationship before committing to a larger build; you learn how they work and you keep whatever they produce. Second, a partner who answers some of these questions with “we’d need to find out” is often more trustworthy than one with a polished answer to everything, provided they explain how they would find out.
Build, buy or wait: a quick decision check
Before you shortlist anyone, it is worth checking whether a partner is the right route at all. These are the questions we would run through ourselves:
- Buy if an existing product covers most of the workflow, your data can go to that vendor, and the remaining gap is something your team can live with.
- Build with a partner if the process is specific to your business, high in volume, and you can describe what a correct output looks like well enough to label examples.
- Build in-house if you have engineers with real spare capacity and the system is core enough that you want the knowledge held internally from day one.
- Wait if you cannot measure the current process, the data is scattered across inboxes and spreadsheets, or the process itself changes every few weeks.
- Wait if the use case looks high-risk under the AI Act and nobody in the organisation yet owns compliance for it.
If you land on wait, that is a useful outcome rather than a failure; the prep work is usually data clean-up and process measurement, both of which are covered in our article on AI automation challenges to plan for before you build.
D3X Solutions LABS is an AI engineering studio in Ireland that builds AI agents, automation and orchestration systems, and advises teams putting AI into production. We start by measuring the current process and building an evaluation set from your real data, put accounts and keys in your tenant from the outset, and treat handover as part of the build rather than an afterthought. If you are weighing up partners, we are happy to answer all twelve of these questions about our own approach, and to tell you plainly if we think you should buy, build in-house or wait. Book a free 30-minute call using the form below.
Frequently Asked Questions
How much does it cost to hire an AI development agency?
It varies widely with scope, integration complexity and how much of the work is discovery versus build. A sensible way to manage the risk is a short, fixed-price discovery phase before committing to a larger build. Always ask for running costs per month as well as the build price, because inference, hosting and maintenance continue after the project ends.
Should we hire an AI consultancy or build an in-house AI team?
If the system is core to your business and you have engineers with spare capacity, building in-house keeps the knowledge internal. If you need to move faster than you can hire, or the project is a first proof of value, a partner can build it and hand over to your team. Many businesses do both: a partner for the first system, with an internal owner trained during handover.
How long does an AI project take to reach production?
A thin working slice on real data is often achievable within a few weeks, but reaching reliable production usually takes longer because of integration, evaluation and approval workflows. Projects that touch legacy systems or regulated decisions take the most time. Ask any partner what will exist and be measurable at the end of the first four to six weeks.
What should be in a contract with an AI development partner?
At minimum: assignment of IP covering code, prompts, evaluation data and configuration; a data processing agreement naming subprocessors and regions; confirmation that accounts and API keys sit in your tenant; and clear exit and handover terms. If a maintenance arrangement is included, define response times and what counts as a change versus a fix.
Does our AI partner need to be based in the EU?
Not necessarily, but GDPR still applies to your data, and international transfers add contractual and compliance work. An EU-based partner familiar with the EU AI Act and the Irish Data Protection Commission’s guidance can reduce that friction. Time zone and the ability to meet in person also matter more than people expect during the first months.
Want this working in your business?
D3X Solutions LABS builds AI agents and automation that pay back fast. Bring one process to a free 30-minute call and we will sketch what automating it would take.
Get in touch
