Difference
The terms “online prediction” and “batch prediction” can be confusing. Both can make predictions for multiple samples or one sample at a time and can make predictions asynchronously(when online prediction leverages a real-time transport to send prediction requests to your model).
Here is a simplified overview of both architectures.
