Serving ML models at scale: batch vs real-time inference

Serving ML models is the moment a machine learning project stops being a notebook and becomes a system with an SLA. A model that scores well offline still has to answer one question, for one user, inside a latency budget somebody already promised to the business. That single constraint decides how the model is packaged, where its features come from, how much compute the company pays for every month, and how fast a bad prediction turns into a visible incident.

Most of that complexity collapses into one deceptively simple fork: batch or real time. Batch inference runs on a schedule, writes predictions into a table, and lets the application read them like any other dataset. Real-time inference keeps the model behind an endpoint and computes the answer while the caller waits. Both patterns are proven, both are unremarkable when implemented well, and choosing the wrong one costs money in two very different ways.

Teams usually discover the mistake late. A batch pipeline gets retrofitted with hourly runs, then fifteen minute runs, then a queue, until it becomes an expensive imitation of an online service. The opposite failure is just as common: a real-time endpoint provisioned for a use case nobody queries more than once a day, quietly burning GPU hours to feed a dashboard. The decision tree below prevents both outcomes, and it rests on the same MLOps foundations that hold up any modern data architecture.

What actually changes when you start serving ML models

Training optimizes for accuracy on historical data. Serving optimizes for a contract: an answer, within a time limit, at a predictable cost, for as long as the feature exists in the product. Those goals pull in opposite directions often enough that the serving layer deserves its own design review instead of an afterthought at the end of a data engineering roadmap.

Three things change immediately. Features stop being a dataframe and become a lookup, which is why the online and offline halves of a feature store have to return the same values, otherwise the model silently degrades through training and serving skew. Dependencies stop being a requirements.txt and become an image that has to boot fast, pushing most teams toward the container patterns they already use for Docker and Kubernetes deployments. Monitoring stops being a validation score and becomes production telemetry, because the only way to know a model is failing is to instrument it like any other service, using the tracing practices in our guide to OpenTelemetry for AI systems.

Governance arrives with the first prediction that touches a customer. Somebody has to know which model version produced which output, on what input, at what time. That lineage requirement is the same one behind data traceability with OpenLineage, and building it into the serving layer up front costs far less than reconstructing it after an auditor asks.

The decision tree for choosing your inference pattern

Work through these five questions in order. The first one that returns a hard answer usually settles the architecture, and the remaining questions then size it.

1. Does the prediction depend on data that only exists at request time?

If the input includes something the system cannot know in advance, such as the contents of a shopping cart, a text the user just typed, or a transaction being authorized right now, real-time inference is the only honest option. Fraud scoring, dynamic pricing and any RAG-style application fall here. When every input is already sitting in the warehouse, keep reading, because you probably do not need an endpoint at all.

2. How stale can the prediction be before it becomes wrong?

Freshness is a business question disguised as a technical one. A churn score recalculated nightly is fine, because customer behaviour does not shift meaningfully in six hours. A delivery ETA recalculated nightly is useless. Write the acceptable staleness down in hours or seconds before anyone opens a cloud console, the same discipline that keeps warehouse costs under control.

3. Is the set of entities you need to score finite and known?

Batch inference works beautifully when you can enumerate the population: every active customer, every SKU in the catalogue, every open ticket. Pre-score all of them, write the results to a table, and the application does a primary key lookup instead of a model call. This pattern makes predictions available to BI tools and product features at the same time, which is why it pairs so well with a governed semantic and metrics layer.

4. What is the latency budget for the action the user is taking?

An interactive experience typically needs the whole round trip, including network and feature lookup, to land in the low hundreds of milliseconds. A background approval can tolerate seconds. Heavy document processing can tolerate minutes, which is exactly why asynchronous inference exists as a distinct option in most managed platforms, including Microsoft's data and AI stack. Document extraction jobs rarely need a synchronous endpoint at all.

5. What does the traffic curve look like, and what is the cost ceiling?

Steady, high-volume traffic justifies a dedicated endpoint with autoscaling and dynamic batching. Spiky or sparse traffic favours serverless inference, where you accept a cold start in exchange for not paying for idle capacity. According to Amazon's documentation, SageMaker exposes these as four distinct inference options: real-time, serverless, asynchronous and batch transform. Azure Machine Learning draws a similar line between online and batch endpoints, and Vertex AI separates online from batch prediction, so the pattern survives whichever cloud your data platform runs on.

Batch, streaming and real-time inference compared

The four patterns below cover almost every production case. Running two of them in the same product is normal, and often correct, provided each one has a documented reason to exist, the same coexistence discipline required when scaling data modeling across teams.

Batch is the cheapest per prediction and the easiest to debug, because a failed run can simply be replayed. Streaming sits in between, trading a persistent consumer for freshness measured in seconds, and it inherits the operational habits of durable workflow orchestration. Real-time carries the heaviest operational burden, since every deploy is a live deploy and every dependency sits on the critical path.

For large language models the same tree applies with one extra variable: token generation makes latency depend on output length, not only on input size. Continuous batching, implemented in serving engines such as vLLM, keeps GPU utilization high by admitting new requests into a running batch instead of waiting for the slowest sequence to finish. Teams that skip that layer usually learn about it through the cloud bill, a pattern documented in our review of agentic AI costs in production.

What breaks at scale, and how to keep it from breaking

The three most common production failures have nothing to do with model quality. Training and serving skew tops the list, where the transformation applied at inference differs subtly from the one applied during training, and it is the strongest argument for computing features once and reading them from a shared store, the same discipline that keeps a governed transformation layer trustworthy. Silent staleness comes next: a scheduled batch job fails, the table keeps yesterday's values, and no alert fires because the query still returns rows. Version drift closes the podium, when the endpoint runs an artefact nobody can trace back to a commit.

Each has a boring fix. Instrument the pipeline so a missing partition raises an alarm rather than serving old numbers, an approach borrowed from the practices that protect a data stack end to end. Compare live prediction distributions against a training baseline so degradation is caught by drift detection before a business user reports it. Tag every deployed artefact with the training run that produced it, the same lineage habit that makes LLM governance workable at enterprise scale.

At BIX Tech we work with multiple data, cloud and machine learning stacks, so the recommendation stays situational, exactly as it does when comparing warehouse-native AI options. A recommendation engine for a catalogue of fifty thousand products may be perfectly served by a nightly batch job and a lookup table, while a payments risk model in the same company needs a sub-second endpoint with an online feature store behind it. The right answer follows the latency contract, the freshness requirement and the traffic curve, rather than the maturity of the tooling.

Serving ML models well is mostly a matter of matching the pattern to the contract and then refusing to drift away from it. Answer the five questions honestly, write the answers down, and the architecture stops being a debate and becomes a consequence. If your company is putting machine learning models into production and needs a serving architecture that holds up under real traffic and real budgets, our specialists can help design it for your context. Talk to our team and move your data maturity forward. ⬇️

FAQ: frequently asked questions

What does serving ML models mean? Serving ML models means making a trained model available to produce predictions for an application, either by running it on a schedule and storing the results or by exposing it behind an endpoint that answers requests. It covers packaging the model, supplying features, scaling compute and monitoring outputs in production.

What is the difference between batch and real-time inference? Batch inference computes predictions for many records at once on a schedule and writes them to a table the application reads later. Real-time inference computes a single prediction on demand while the caller waits, usually over HTTP or gRPC. Batch costs less per prediction and is easier to replay, while real-time delivers freshness measured in milliseconds.

When should you use batch inference instead of an endpoint? Use batch inference when the entities to score are known in advance, the inputs already exist in the warehouse, and a delay of hours between data arriving and the prediction being available is acceptable. Churn scores, propensity models, inventory forecasts and lead scoring usually fit this profile and avoid the cost of an always-on service.

How do you reduce the latency of a real-time ML endpoint? Cut the largest contributor first, which is usually the feature lookup rather than the model itself. An online feature store, dynamic or continuous batching, a smaller or quantized model, warm replicas that avoid cold starts, and an endpoint deployed in the same region as the caller all help. Measure the full round trip, never model execution time alone.

What is training and serving skew? Training and serving skew happens when the features a model receives in production differ from the ones it saw during training, because the transformation logic was reimplemented in the serving path. The result is a model that performs well offline and poorly live. Computing features once and reading them from a shared online and offline store is the standard prevention.