Part 2 · Inference System

Follow one prompt through model readiness, single-GPU inference, and coordinated multi-GPU execution.

Cold start to warm worker · 8 min

Load and ready the model

The runtime loads model shards into GPU memory, initializes execution, warms up, and declares the worker ready.

Essential path

Plain language, the core causal flow, and required checks.

From stored files to a warm worker

At startup, the serving system resolves an exact artifact, reserves device memory, and moves weight shards from storage through host memory into GPU HBM. Multi-GPU workers also establish their communication groups.

The runtime then initializes kernels, memory pools, and cache structures. Warmup requests exercise important execution paths before the worker is marked healthy and admission opens.

Loading happens once per worker lifecycle; the resident weights are then reused across many inference requests.

Interactive explanation

Move model files into GPU memory

Core + Expert

Cold start · Phase A

Move stored model files into GPU memory

interactive explanation

This phase moves and verifies bytes. Its endpoint is weights resident in HBM—not a worker that is ready to serve.

Current operationResolve an immutable model version and verify its manifest.
Current stateResolve an immutable model version and verify its manifest.
Remote read floor11.7 s
Device transfer floor2.9 s

Readiness is a sequence, not a process state

After bytes arrive in HBM, the worker may still need to initialize memory pools, compile or select kernels, establish multi-rank communicators, allocate cache structures, and exercise representative shapes. Admission should open only after that usable execution path succeeds.

Interactive explanation

Ready an inference worker

Core + Expert

Cold start · Phase B

Turn resident weights into a usable inference worker

interactive explanation

This phase begins exactly where data movement ends. Admission remains closed until the runtime can execute representative model work.

Weights already resident in HBM
S0S1S2S3S4S5S6S7
  1. 1Allocate runtime
  2. 2Connect GPU ranks
  3. 3Initialize kernels
  4. 4Warm up
  5. 5Open admission
Request admissionCLOSED

All weight shards arrived during Phase A. The remaining gates prepare execution.

Current evidenceEvery expected shard is mapped to the correct HBM region.
Worker stateWeights resident; runtime cold
After this chapter, you can:
  • Distinguish cold and warm inference.
  • Trace weights from storage to GPU HBM.

Core check

What distinguishes a warm worker from a cold one?