Part 3 · Inside the GPU

Zoom from a kernel launch into blocks, warps, schedulers, execution units, and memory.

Hide latency with eligible work · 10 min

How warps are scheduled

When one warp waits, a scheduler can issue instructions from another ready warp, keeping execution units busy.

Essential path

Plain language, the core causal flow, and required checks.

Ready work hides waiting

A warp contains 32 CUDA threads. At an instruction-issue opportunity, a warp scheduler chooses an eligible warp whose operands and execution resources are ready.

If a warp is stalled on memory or a dependency, another resident warp can issue. This rapid interleaving is how GPU concurrency hides latency without making the original memory operation itself faster.

High theoretical occupancy only creates the opportunity to hide latency. Dependencies, divergence, and instruction mix determine achieved utilization.

One instruction stream, different lane decisions

CUDA threads hold individual state, but the threads in a warp issue together. When a branch divides the warp, hardware masks the lanes for one path, then the other, before reconverging. Divergence is expensive when both paths contain substantial work.

Interactive explanation

Simulate warp divergence

Core + Expert

SIMT and divergence

Watch one warp execute two branch paths under masks

interactive explanation

Threads keep individual data and branch decisions, but a warp issues one shared instruction stream. Divergent paths therefore run one after the other.

0•1•2•3•4•5•6•7•8•9•10•11•12•13•14•15•16•17•18•19•20•21•22•23•24•25•26•27•28•29•30•31•
Scenario setup
Path A · 16 lanes · 5 instructions
Path B · 16 lanes · 8 instructions
No branch decision yet; all 32 lanes move together.

All 32 lanes issue the same instruction together. The scenario stays fixed while the stages reveal what executes.

Issued cycles0 branch-path cycles
Active-lane utilization100% now · uniform

Interactive explanation

Schedule eligible warps

Core + Expert

Warp scheduling

Watch one scheduler partition hide a memory wait

interactive explanation

This trace follows one simplified scheduler partition. A memory request can remain in flight while an independent warp uses an arithmetic pipeline.

One simplified scheduler partitionOne issue decision is highlighted per stage. Work already sent to memory can continue concurrently.
Warp 0ready · LDload then add
Warp 1ready · FP32independent arithmetic
Warp 2dependencywaiting on prior result
Scoreboard + schedulerInspect eligibility

The scoreboard sees two eligible warps.

New issue this stageLD/STFP32Tensor
Memory systemNo request in flight
Arithmetic pipelineNo arithmetic active

Two warps are eligible; no instruction has issued.

C0—C1W0 · LDC2—C3W1 · FP32C4W0 · dataC5W0 · ADD
Current lessonThe scoreboard sees two eligible warps.
Eligible after this eventWarp 0 and Warp 1
Real GPU contextReal SMs may have multiple scheduler partitions, so different eligible warps can issue to available pipelines in the same cycle.
After this chapter, you can:
  • Explain eligible versus stalled warps.
  • Describe how concurrency hides latency.

Core check

How does a GPU hide a long memory wait?