10.2b Warps: groups of 32 threads executing in lockstep — the SIMT model¶

🧭 Context Introduction¶

When you write a program for an NVIDIA GPU, you typically launch thousands or even millions of threads. But the GPU hardware doesn't actually run all those threads independently at the same time. Instead, it groups them into warps — the fundamental unit of execution inside a Streaming Multiprocessor (SM). Understanding warps is essential for engineers working with AI infrastructure, because warp behavior directly impacts performance, memory access patterns, and how efficiently your models train or infer.

⚙️ What Is a Warp?¶

• A warp is a group of 32 threads that execute together on a single SM.
• All 32 threads in a warp run the same instruction at the same time — this is called lockstep execution.
• This model is known as SIMT (Single Instruction, Multiple Threads).
• Each thread in the warp has its own registers and own program counter, but they all follow the same instruction sequence.

🕵️ The SIMT Model Explained¶

SIMT is NVIDIA's approach to parallel computing. It combines the efficiency of SIMD (Single Instruction, Multiple Data) with the flexibility of multithreading.

Example in plain language:
Imagine 32 workers standing in a line. A foreman shouts "Lift the box!" — all 32 workers lift their own box at the same time. Each worker's box might be different, but the action is identical. That's a warp.

📊 Why 32 Threads?¶

• The number 32 is a hardware design choice by NVIDIA.
• It balances parallelism (many threads running at once) with resource constraints (registers, shared memory, and scheduling logic).
• Each SM contains multiple warp schedulers that can issue instructions to different warps every cycle.
• A typical NVIDIA data center GPU (like the H100 or A100) can have 64 to 128 warps active per SM at any given time.

📊 Visual Representation: SIMT Warp Execution Model¶

This diagram displays how a single instruction is executed in lock-step across a group of 32 threads (a warp) in the SIMT model.

🛠️ Warp Divergence — The Performance Trap¶

When threads inside the same warp take different paths in code (e.g., an if-else statement), the warp must execute both paths serially.

Example scenario: - Threads 0–15 take the if branch. - Threads 16–31 take the else branch. - The warp executes the if branch first (threads 0–15 active, threads 16–31 idle). - Then the warp executes the else branch (threads 16–31 active, threads 0–31 idle).

Result: The warp takes twice as long to complete.

🚨 Key Rule for Engineers¶

• Minimize warp divergence in your kernels.
• Keep conditional logic aligned with warp boundaries (e.g., use thread IDs modulo 32 to decide branches).
• Use predication (compiler-generated) or data reorganization to avoid divergent paths.

📈 Warp Scheduling and Latency Hiding¶

• The SM's warp scheduler rapidly switches between warps to hide latency (e.g., memory access delays).
• While one warp waits for data from global memory, the scheduler picks another ready warp to execute.
• This is called zero-overhead context switching — switching warps costs no extra clock cycles.
• For AI workloads, this means the GPU can keep its compute units busy even when memory bandwidth is saturated.

🧩 Practical Implications for AI Infrastructure¶

🔍 Quick Reference: Warp vs. Thread Block vs. Grid¶

✅ Summary for New Engineers¶

• Warps are groups of 32 threads that execute in lockstep — this is the SIMT model.
• All threads in a warp run the same instruction at the same time, but on different data.
• Warp divergence hurts performance — keep conditional logic aligned with warp boundaries.
• The SM uses warp scheduling to hide memory latency and keep the GPU busy.
• Always design your kernels with warp size (32) in mind for optimal AI infrastructure performance.

🧠 Remember: The warp is the smallest unit of execution that the hardware actually schedules. Understanding warps is the key to unlocking GPU performance in AI workloads.