Part 3 · Inside the GPU

Zoom from a kernel launch into blocks, warps, schedulers, execution units, and memory.

Device to streaming multiprocessor · 9 min

Zoom into GPU anatomy

A GPU contains many streaming multiprocessors that execute large numbers of related threads while sharing a memory hierarchy.

Essential path

Plain language, the core causal flow, and required checks.

Why the GPU is built differently

A CPU devotes substantial silicon to sophisticated control, large caches, and fast execution of a small number of threads. A GPU accepts more latency per thread and devotes far more of its area to parallel arithmetic. That trade only pays when the workload exposes enough independent, regular work.

Interactive explanation

Compare CPU and GPU execution

Core + Expert

Latency versus throughput

Run the same work on a few wide CPU cores and many GPU lanes

interactive explanation

The workload changes the dependency graph. Many lanes help only when enough independent, regular work is ready at the same time.

Work dependency
1 →2 →3 →4 →5 →6 →7 →8 →9 →10 →11 →12 →13 →14 →15 →16 →17 →18 →19 →20 →21 →22 →23 →24 →25 →26 →27 →28 →29 →30 →31 →32
Each operation waits for the previous result.
CPU · four wide cores
core 1core 2core 3core 4
Strong fit for one dependency chain
GPU · launch overhead + many lanes
Most lanes idle because only one operation is ready
CPU siliconControl, prediction, and large caches reduce latency for a few threads.
GPU siliconMore arithmetic lanes and thread state increase throughput.
Conceptual allocation—not a physical die-area scale drawing.
Illustrative CPU time32 normalized cycles
Illustrative GPU time140 normalized cycles, including launch
InterpretationExtra lanes cannot break a true dependency.

A hierarchy built for throughput

High-bandwidth memory holds weights, KV-cache blocks, and working tensors. Cache and memory controllers feed many streaming multiprocessors, usually called SMs.

An SM contains warp schedulers, execution pipelines, registers, and shared memory. Many warps can remain resident so the scheduler has other eligible work when one warp waits on a dependency or memory.

Two hierarchies meet inside the GPU. The hardware hierarchy is GPU → SM → scheduler → execution and memory pipelines. The work hierarchy is kernel grid → block → threads → warps. Hardware admits a complete block to one SM, groups its threads into warps, and lets a scheduler choose eligible warps for instruction issue. Blocks and warps are assigned work, not permanent physical layers.

Exact unit counts vary by GPU generation. The durable mental model is how the work hierarchy maps onto the hardware hierarchy.

Interactive explanation

Zoom into a GPU

Core + Expert

GPU anatomy

Zoom through hardware while following the work mapped onto it

interactive explanation

Hardware and work are related but different hierarchies. Blocks and warps are assigned work; SMs, schedulers, pipelines, and memory are hardware.

Hardware at this zoomGPU package
Work mapped hereModel operation
GPU compute dieSMs · caches · memory controllers
HBM stacks
stackstackstack
Count and layout vary by accelerator.
Accelerator package · illustrative, not a specific productThe GPU die connects to multiple high-bandwidth-memory stacks; the three stack symbols and ellipsis mean “several,” not exactly three.
Selected levelGPU package
Mapping ruleThe device contains many SMs.
After this chapter, you can:
  • Locate HBM, cache, streaming multiprocessors, and execution units.
  • Explain why GPUs favor throughput.

Core check

Where do resident thread blocks execute?