The reliable way to integrate a large language model with a legacy enterprise system is to leave the legacy system alone and put a thin, well-tested layer in front of it. That layer exposes a small set of safe operations, such as “look up this order” or “create a draft credit note”, and the LLM works only through those operations. Around it you add asynchronous messaging so slow model calls never block the old system, retrieval over the legacy data for questions, human review for anything consequential, and an evaluation set that tells you whether the whole thing still works after each change. Screen automation is the last resort, not the starting point.
Below are those patterns, the order to build them in, and the cost, latency and security questions that decide whether an integration survives production. It is aimed at teams whose core systems are ten or twenty years old and staying put.
Why the legacy system is the hard part
Connecting a model to an ERP, a policy administration system or a mainframe-backed core platform is where projects stall. Legacy systems share a few traits. They are the system of record, so mistakes are expensive. Their interfaces were designed for humans or nightly batch jobs, not for software making hundreds of calls an hour. Business rules live partly in code, partly in stored procedures and partly in the heads of long-serving staff. And they are often fragile under load, with no sandbox that resembles production.
None of this rules them out. The integration work is mostly ordinary software engineering: interfaces, data contracts, idempotency, testing and monitoring. We look at how to assess this before committing to a use case in our guide to the AI readiness audit.
Pattern 1: an API façade in front of the legacy system
Build a small service that sits between everything new and the legacy system, and expose only the operations the use case needs. Behind the façade, the service talks to the old system in whatever way it understands: a SOAP endpoint, a database view, a stored procedure, an import file or a vendor SDK. In front, it offers clean, documented, versioned operations with strict input validation.
Some design rules make this layer worth having:
- Start read-only. Lookups and searches carry little risk and deliver value quickly.
- Make every write idempotent, with a key per request, so a retry after a timeout cannot post the same transaction twice.
- Enforce business rules in the façade, in code, rather than trusting the model to remember them.
- Return structured data with clear error codes the calling system can act on.
The same operations then become the tools an AI agent can call. Because the façade defines what is possible, the model cannot invent an operation, write to a table it should not touch or skip a validation. This is also the layer that compounds: once it exists, the next automation reuses it rather than starting again, which is the core argument of how to build software systems that compound over time.
Pattern 2: events and queues instead of synchronous calls
LLM calls are slow and variable compared with a database query. A response can take a few seconds, or much longer under load, and providers occasionally fail or rate-limit. If a legacy transaction waits on a model call, that variability lands on your core system and its users.
Decouple the two. When something happens in the legacy system, publish an event to a queue: a new order, a changed claim status, an inbound document. A worker picks the event up, calls the model, and writes the result back through the façade when it is ready. Where the legacy system cannot publish events, there are established options. You can poll a change table on a schedule, use change data capture on the database log, or pick up the export files it already produces overnight. For writes, the transactional outbox pattern keeps the legacy database and the queue consistent without distributed transactions.
Queues also give failures somewhere to go: items that fail validation land on an exception list for a person, rather than vanishing or blocking the next item.
Pattern 3: retrieval over legacy data
Many of the most useful early uses are questions rather than actions. What did we agree with this customer last year? Which open orders are waiting on this part? Retrieval-augmented generation (RAG) answers these by fetching relevant records or documents at query time and giving them to the model as context, rather than training a model on your data.
Two kinds of retrieval are worth distinguishing. Unstructured content, such as contracts, emails, procedures and notes, goes into a search index, often combining keyword and vector search. Structured data in the legacy database is usually better served by well-defined query operations on the façade, or by text-to-SQL against a read replica with a restricted schema, than by turning rows into text and embedding them.
The details that decide whether RAG works in an enterprise setting are rarely about the model. Keep the index fresh, with an incremental sync rather than a quarterly rebuild. Carry access permissions into the index and filter results by the user’s rights before they reach the prompt. And return sources with every answer, so people can check the record behind it.
Pattern 4: screen automation as a last resort
Some systems have no API, no accessible database and no import function. Robotic process automation, where software drives the user interface, can bridge that gap. It is also the most brittle option: a screen change, a slow page or an unexpected dialog box breaks it. If you must use it, wrap it behind the same façade so it can be replaced later, keep the scripted steps short and deterministic, and do not let a language model decide where to click. A thin, tested service layer in front of the legacy system almost always beats letting a model drive a screen.
Keep people on the consequential decisions
Most legacy integrations should start as draft-and-approve. The model prepares a credit note, a reply, a reconciliation or a field update, and a person approves it in a queue before it is written back. Once you have evidence on accuracy for a category of work, you can let low-risk, high-confidence items flow straight through and keep review for the rest. Route by rules you can explain, such as value thresholds, customer type or validation failures, not only by the model’s own confidence score. Our article on automating manual data entry covers the exception-review step in detail.
Evaluation before and after go-live
Build an evaluation set before you build the integration: a few hundred real examples with known correct outcomes, drawn from historical records and checked by the people who do the work today. Run every change to prompts, models or retrieval against it. Before going live, run in shadow mode, where the system produces outputs that are compared with what staff actually did but never written back. After go-live, sample outputs for review and feed corrections into the evaluation set. Without this, a model upgrade or a prompt tweak can quietly break behaviour that worked last month.
Cost and latency
Model costs scale with tokens, so they scale with how much context you send. Retrieval that returns twenty long documents when two short records would do is slower and more expensive. Practical levers include:
- Trimming context to the fields the task needs.
- Routing simple classification and extraction work to smaller, cheaper models and reserving larger ones for harder cases.
- Caching responses for repeated queries where the underlying data has not changed.
- Using providers’ batch interfaces for overnight work that does not need an immediate answer.
Set timeouts for every model call, decide in advance what happens when the provider is unavailable, and track cost per processed item from the first week.
Security and data protection
The integration inherits every security question the legacy system already has, plus some new ones. The service account the façade uses should have the narrowest permissions possible, and user-facing features should act with the permissions of the person using them. Any system that reads untrusted content, such as inbound email or uploaded files, and can also call write operations needs protection against prompt injection: scoped tools, validated arguments and human approval for sensitive actions. Decide where the model runs and how long the provider retains data before you send it anything. Our guide to data security and privacy in AI projects sets out the controls in detail, including EU hosting and zero-retention settings. These measures support your GDPR obligations, but compliance still depends on your lawful basis, contracts and processes, not on the architecture alone.
A sensible order of work
- Pick one use case with clear volume and a measurable outcome.
- Build the evaluation set from historical data.
- Build a read-only façade and a retrieval layer, and ship an internal question-answering or lookup tool.
- Add the queue and a single draft-and-approve write operation.
- Run in shadow mode, measure against the evaluation set, then go live with review.
- Relax review for categories where the evidence supports it, and reuse the façade for the next use case.
Regulated firms often have their hardest data locked in old core systems, which is why the same pattern underpins compliance reporting automation. Engineering teams nervous about touching legacy code may also find our notes on reducing technical debt with automated systems useful, particularly the section on characterisation tests.
Working with D3X Solutions LABS
D3X Solutions LABS is an AI engineering studio in Ireland. We build AI agents, automation and LLM integrations for businesses in the EU, UK and Ireland, and a large part of that work is connecting new capabilities to systems that were never designed for them. Our approach follows the order above: we map the legacy system’s interfaces and data, agree one use case and its evaluation set, build the façade and queue, and put the model behind them with review steps and logging from day one.
You keep the code, the evaluation set and the documentation, and the façade stays useful long after the first project. If you have a core system that people work around rather than with, book a free 30-minute call and we will talk through what an integration could look like.
Frequently Asked Questions
Can an LLM be connected to a system with no API?
Usually, yes. A façade service can reach the system through a database view, stored procedures, import and export files or, as a last resort, screen automation. The model then works only through the façade’s defined operations, so the method of access can be improved later without changing the AI side.
Should we fine-tune a model on our legacy data?
Rarely as a first step. Retrieval gives the model current, permission-filtered data at query time and lets you cite sources, while fine-tuning bakes in a snapshot and makes deletion requests harder to honour. Fine-tuning can help later for narrow formatting or classification tasks once you have a solid evaluation set.
How do we stop the AI writing bad data into the system of record?
Put validation and business rules in the façade, in code, and make every write idempotent. Start with draft-and-approve so a person signs off each write, and only allow straight-through processing for categories where the evaluation results justify it.
What does LLM integration cost to run?
Running cost depends mainly on volume, the amount of context sent per request and the models used. Trimming context, routing simple tasks to smaller models, caching and batch processing all reduce it. Measure cost per processed item from the start so you can compare it with the manual cost it replaces.
Do we need to replace our legacy system before adopting AI?
No. Wrapping the existing system behind a clean interface lets you add AI capabilities now and makes a later migration easier, because new components depend on the façade rather than on the old system directly.
Want this working in your business?
D3X Solutions LABS builds AI agents and automation that pay back fast. Set up a free 30-minute call and we will map your first win together.
Get in touch
