Move less, organize it, reuse it · 10 min
Keep the GPU supplied with data
A GPU stays busy when related data arrives together and reusable data remains close to the execution units.
Plain language, the core causal flow, and required checks.
How can a fast GPU still wait?
A GPU can perform an enormous number of calculations, but its execution units cannot calculate with data that has not arrived. This chapter asks one question:
How do we keep the GPU working instead of waiting for data?
Think of the GPU as a large kitchen:
- HBM is the storeroom. It holds a lot of data, but repeatedly visiting it costs time and bandwidth.
- L2 and L1 are nearby caches. Hardware may retain recently used data there automatically.
- Shared memory is a workbench. A kernel deliberately places a reusable group of values there for threads in one block.
- Registers are ingredients in a worker's hands. They hold the values one thread is using right now.
These are not always a rigid sequence that every load must visit. A normal load may be served from a hardware cache. Shared memory is different: a kernel explicitly stages data there when coordinated reuse is worthwhile.
Interactive explanation
Follow and coalesce a memory load
Keeping the GPU supplied
Move less data, move it together, and reuse it nearby
Think of HBM as a storeroom, shared memory as a workbench, and registers as ingredients already in each worker's hands.
Thousands of GPU threads can calculate quickly, but only after their data arrives. Distance and organization determine how long they wait.
Large, shared by the GPU, and expensive to revisit repeatedly.
A kernel explicitly stages a reusable tile here for one block.
The values a thread is using right now.
Two actions reduce HBM traffic
1. Organize the first delivery
Threads are already grouped into warps. A kernel does not normally create a new warp just to improve memory access. Instead, it maps neighboring threads to neighboring data. The memory system can combine those requests into fewer deliveries. This is called coalescing.
Scattered requests still return the requested values, but they may require many more deliveries.
2. Reuse data after it arrives
If several calculations need the same values, threads can cooperate to load a tile once, place it on the shared-memory workbench, and reuse it. Frequently used individual values can remain in registers. Kernel fusion can also avoid writing an intermediate result to HBM only to read it back immediately.
The practical rule is:
Move related data together, keep reusable data nearby, and avoid unnecessary round trips to HBM.
In AI inference, the same principle applies to model weights, activations, intermediate tensors, and the KV cache. Application developers usually receive these optimizations through frameworks, compilers, and tuned libraries. Kernel engineers work directly on layouts, coalescing, tiling, fusion, and register use.
- Distinguish HBM, hardware caches, shared memory, and registers.
- Explain how organized delivery and reuse reduce trips to HBM.
Core check
Which access pattern is usually easier to serve efficiently?
Terms in this chapter
- Coalescing
- Combining nearby memory requests from threads in a warp into fewer memory transactions.
Primary references
- CUDA C++ Best Practices Guide ↗NVIDIA
Primary reference for global-memory coalescing and performance considerations.