Designing Scalable AI Pipelines for Legacy Systems

Designing Scalable AI Pipelines for Legacy Systems

Teams often add AI pipelines to legacy systems after the core transaction logic is already in place. The new work has to read from tables that were never designed for analytics, and it has to write results without breaking nightly batch jobs that still run the business.

We have seen this pattern across several clients who run mainframes or large ERP instances. The AI work starts as a side project and then collides with data formats that have accumulated twenty years of exceptions.

Data extraction sets the real pace

Legacy tables rarely expose clean timestamps or change logs. Teams spend the first months writing custom extract jobs that poll for deltas instead of relying on CDC tools that the source system does not support.

We usually begin by mapping which fields actually change in production and which ones only appear to change because of audit triggers. That mapping decides whether daily files or hourly queries are even feasible.

The practical steps are:

• Identify the smallest set of tables that carry the signals the model needs
• Build a narrow export process that runs outside the main transaction window
• Store the exports in a separate schema so the original application code stays untouched

Once that extract stabilizes, downstream machine learning work can start without constant coordination with the ERP team.

Separate inference from the transaction path

Putting model calls inside existing service methods creates latency spikes that the legacy code was never sized for. We keep inference on its own compute and call it through simple queues or file drops.

This separation also makes it easier to roll back. When the model returns low-confidence scores, the queue consumer can route the record to a manual review table instead of updating the main ledger.

Data engineering work happens before model training

Most legacy data contains nulls, overwritten fields, and business rules that only exist in old stored procedures. Cleaning happens in the export layer, not inside the training notebook.

We create a small set of typed views that the data engineering team owns. The machine learning team then pulls from those views rather than negotiating directly with the source system owners every time a column changes.

Evaluation must include the original business process

Accuracy numbers on a hold-out set do not show whether the new predictions reduce manual corrections in the warehouse or finance department. We tie evaluation to the same tickets or exceptions that the legacy workflow already tracks.

This requires logging both the model output and the final human decision for the same record. Over time the comparison reveals whether the pipeline is actually moving work or just adding another review step.

Ownership stays with the system that already runs

The team that maintains the legacy database keeps control of the extract jobs. A separate, smaller group owns the model training and the queue consumers that write results back.

Splitting ownership this way prevents the common situation where an AI feature is declared complete but then drifts because no one is watching the nightly extract.

One decision before any code is written

Choose the narrowest slice of data that already has a stable export and a measurable business outcome. Build the pipeline for that slice only, then decide whether the next slice is worth the same effort.