Part 2 · Inference System

Follow one prompt through model readiness, single-GPU inference, and coordinated multi-GPU execution.

Prompt to streamed tokens · 12 min

Inference on one GPU

Tokenization, prefill, KV-cache creation, iterative decode, sampling, and streaming turn a prompt into output tokens.

Essential path

Plain language, the core causal flow, and required checks.

Read the prompt, then extend it

Tokenization produces input IDs. During prefill, the model processes the prompt positions in parallel and creates attention keys and values for every layer. Those reusable values form the KV cache.

Decode is iterative. Each active sequence produces one next-token prediction, the sampling policy chooses a token, its new key and value are appended, and the sequence returns to the scheduler unless a stopping condition has fired.

Time to first token includes queueing and prefill. Inter-token latency describes the cadence after generation starts. They can have different bottlenecks.

Interactive explanation

Follow single-GPU inference

Core + Expert

Single-GPU inference

Follow one prompt until a complete response is streamed

interactive explanation

The prompt, output text, KV cache, and GPU work share one state. Decode repeats until a stop condition completes the response.

Prompt

How do GPUs work ?

How#4100do#4137GPUs#4174work#4211?#4248
GPU operationWaiting for token IDs
KV cache · 0.0 MB in this illustration
Streamed response

No output token yet

Waiting for first token

The serving system hands one admitted prompt to a ready worker.

Current actionThe serving system hands one admitted prompt to a ready worker.
Time to first tokenStill accumulating
Time per output tokenMeasured after generation begins

What one decode iteration does

The scheduler selects active sequences. Their latest token IDs enter the model. Layers read the fixed weights and the previously cached keys and values. The final layer produces logits, the decoding policy selects a token, and the new key/value entries are appended. Finished sequences leave; unfinished ones return for another iteration.

After this chapter, you can:
  • Trace the complete single-GPU inference lifecycle.
  • Distinguish time to first token from inter-token latency.
  • Tokenization converts text to token IDs.
  • Prefill processes the prompt in parallel.
  • The KV cache preserves reusable attention state.
  • Decode predicts one next token per sequence iteration.
  • Sampling chooses and streams the next token.

Core check

What does the KV cache avoid recomputing during decode?