Inference across multiple GPUs

More capacity has more than one meaning

Independent replicas each hold a complete model and serve different requests. They increase aggregate throughput but do not help an oversized model fit on one device.

Model-parallel approaches make several GPUs cooperate on one request. Tensor parallelism divides operations within layers; pipeline parallelism divides layers; expert parallelism routes tokens among expert partitions.

Cooperation introduces communication and synchronization. The useful question is not simply “How many GPUs?” but whether they are replicating work or sharing one model execution.

Interactive explanation

Compare multi-GPU strategies

Core + Expert

Multi-GPU inference

See exactly when GPUs work independently or communicate

interactive explanation

Every strategy uses the same GPU cards. The request placement and communication paths change to match the selected execution model.

01Independent replicas02Tensor parallel03Pipeline parallel04Expert parallel

rank 0GPU 1complete model · request 1

rank 1GPU 2complete model · request 2

rank 2GPU 3complete model · request 3

rank 3GPU 4complete model · request 4

request 1 ↓ GPU 1request 2 ↓ GPU 2request 3 ↓ GPU 3request 4 ↓ GPU 4

GPU ranks4

Execution modelEach GPU holds the complete model and serves a different request.

Communication patternNo inter-GPU collective on the token path

Best fitThroughput and availability

Core check

Which setup makes GPUs communicate while serving one model?

ATensor parallelismBIndependent replicasCClient-side tokenization