Model Serving: Definition and Examples

Model serving is the step that transforms a trained AI model into an operational service capable of responding to production requests. Concretely, it involves loading the model into memory, exposing an interface (typically a REST or gRPC API), and managing the infrastructure needed to handle inference requests reliably and efficiently. This stage is often considered the bridge between the research phase and production deployment. A model may achieve excellent results in the lab, but without a proper serving infrastructure, it cannot be used by real applications. Model serving therefore encompasses issues of latency, throughput, autoscaling, and model version management. Model serving solutions range from open source frameworks like TensorFlow Serving, TorchServe, or Triton Inference Server to managed platforms offered by cloud providers (SageMaker, Vertex AI, Azure ML). The choice depends on request volume, latency constraints, model type, and available budget. In the context of large language models (LLMs), model serving presents specific challenges related to model size, GPU memory management, and optimization techniques such as quantization, continuous batching, and PagedAttention. These optimizations make it possible to serve models with billions of parameters at reasonable costs.