A short arrival summary · 5 min

Before the request reaches a GPU

Authentication, routing, admission control, and batching prepare a request for inference.

Essential path

Plain language, the core causal flow, and required checks.

The short runway before GPU execution

A request is authenticated, routed to an appropriate model deployment, checked against capacity, and admitted to a queue. The tokenizer converts its text into token IDs before GPU work begins.

The scheduler may briefly combine compatible work into a batch. Larger batches can improve throughput, but waiting too long increases user-visible latency. Continuous batching revisits this decision as active sequences finish and new requests arrive.

One useful separation is control plane versus data plane. Configuration, placement, health, and policy prepare the service. The admitted prompt and its generated tokens are the request’s active data path.

This chapter follows the request from the application endpoint through routing, admission, tokenization, and batching. The next chapters show how the model is loaded and how the admitted request executes on the GPU.

Interactive explanation

Prepare a request

Core + Expert

These are sequential stages, not alternative choices. Each gate changes the request state before GPU work may begin.

Current request stateAuthenticate

Verify identity, quota, and request policy.

Batching controls appear only after authentication, routing, admission, and tokenization have succeeded.

Serving decisionVerify identity, quota, and request policy.

GPU workNot dispatched

The compute layers underneath the model server

Every deployment retains the same inference core: model artifact, inference server, accelerator runtime, device driver, and accelerator. Containers, Kubernetes, and virtual machines add placement, isolation, lifecycle, and ownership layers around that core. They do not replace it or change the model's token computation.

Interactive explanation

Build the compute platform stack

Core + Expert

Containers, Kubernetes, and virtual machines change placement and ownership. They do not replace the model server, accelerator runtime, driver, or GPU.

01Model artifactweights + config + tokenizer

02Inference servervLLM, TensorRT-LLM, TGI, or similar

03Accelerator runtimeCUDA, ROCm, or oneAPI user-space libraries

04Host GPU driverdevice control and command submission

05GPUphysical accelerator executing kernels

Selected viewCommon inference core

Invariant pathModel → server → runtime → driver → GPU

After this chapter, you can:

Name the essential pre-GPU phases.
Explain why batching changes throughput and wait time.

Follow the serving decisions that turn an incoming prompt into GPU work: authenticate, route, admit, tokenize, and form a compatible batch.

Core check

Before the request reaches a GPU

The short runway before GPU execution

Prepare a request

Move one prompt through five serving gates

The compute layers underneath the model server

Build the compute platform stack

Keep the inference core visible while infrastructure layers are added

Why might a serving system briefly queue compatible requests?