Launch one kernel end to end

What is a GPU kernel?

A GPU kernel is a small program applied to many data elements in parallel. For vector addition, every thread runs the same rule—C[i] = A[i] + B[i]—but computes a different index i. The kernel is the program; the launch describes how much parallel work to create.

A normal model operation may invoke one kernel, several kernels, or a fused kernel. Matrix multiplication, attention, normalization, activation, and sampling all become accelerator work. Libraries may supply precompiled implementations, while compilers such as Triton or graph compilers may generate or fuse specialized kernels. This translation is normally prepared ahead of a request or during warmup; it is not rewritten from scratch for every token.

From a model operation to thread work

On NVIDIA hardware, a launch specifies a grid made of thread blocks plus arguments, stream ordering, and block dimensions.

Hardware assigns complete blocks to SMs when registers, shared memory, thread slots, and block slots are available. Threads in each block are partitioned into warps for instruction issue.

The CPU launch can be asynchronous. A later dependency or synchronization boundary determines when another operation must wait for its results.

Interactive explanation

Launch one kernel

Core + Expert

Kernel programming and launch

Follow one vector-add kernel from source operation to completion

interactive explanation

A kernel is a small program applied across many data elements. The launch creates blocks of threads; hardware maps those blocks onto SMs.

01Kernel program02Host enqueue03Driver command04Grid created05Blocks dispatched06Block admitted07Warps formed08Instruction issued09Memory serviced10Kernel completed

Example operationC[i] = A[i] + B[i]Each thread computes one valid index i.

CPU streamkernel not submitted

Runtime + driverwaiting

Grid · representative blocks

B0B1B2B3B4B5B6B7

Available SMs

SM 1SM 2SM 3SM 4

Selected block · 4 warps

W0W1W2W3

threads grouped in hardware issue units

Completionwaiting for every block

Stage 1Kernel program

One small device program says what each parallel thread does.

Core uses small representative numbers so every grid, block, warp, and SM can be followed visually.

Grid shape8 blocks × 128 threads

Selected block128 threads → 4 warps

The portable stack and vendor-specific implementations

The portable idea is:

Model or framework → operator/compiler → accelerator runtime → device driver → accelerator

The concrete names differ:

• NVIDIA GPU: CUDA libraries/runtime, NVIDIA driver, and NVIDIA GPU.
• AMD GPU: ROCm/HIP libraries/runtime, AMD driver, and AMD GPU.
• Intel GPU: oneAPI/SYCL or Level Zero and Intel GPU.
• Google TPU: XLA/PJRT and the TPU runtime; it does not use CUDA's grid/block/warp model in exactly the same way.

Google Cloud also offers NVIDIA GPU machines; those still use CUDA. Google’s own accelerator is the TPU. The rest of this chapter uses NVIDIA/CUDA as a concrete example while identifying which concepts are portable.

Core check

What is a grid made of?

AThread blocksBModel replicasCWeight checkpoints