10.2a SM internal components: CUDA cores, Tensor Cores, schedulers, and register file¶

Welcome, new engineer! This topic dives inside the Streaming Multiprocessor (SM) — the tiny but mighty engine that does all the heavy lifting in an NVIDIA GPU. Think of the SM as a factory floor inside a massive data center chip. To understand how AI workloads (like training large language models or running inference) actually get computed, you need to know the four key components inside each SM: CUDA cores, Tensor Cores, schedulers, and the register file. Let's break them down simply.

🧠 Context: Why Should You Care About the SM?¶

Every time you run a deep learning job or a scientific simulation on an NVIDIA GPU, your code is broken into thousands of tiny threads. These threads are grouped into warps (typically 32 threads), and each warp is assigned to an SM for execution. The SM's internal components determine how fast and how efficiently those threads get processed. Understanding these parts helps you optimize your AI workloads and troubleshoot performance issues.

⚙️ CUDA Cores — The General-Purpose Workers¶

CUDA cores are the standard arithmetic units inside an SM. They handle general-purpose calculations like addition, multiplication, and logic operations for floating-point (FP32) and integer (INT32) data.

• What they do: Execute one operation per clock cycle per core for standard math.
• Why they matter: Most traditional GPU workloads (graphics, basic simulations) rely on CUDA cores. For AI, they handle non-matrix operations like activation functions (e.g., ReLU) or data preprocessing.
• How many per SM: Varies by GPU generation. For example, an NVIDIA A100 SM has 64 CUDA cores; an H100 SM has 128 CUDA cores.

Key takeaway: CUDA cores are the "jack-of-all-trades" inside the SM.

🚀 Tensor Cores — The AI Accelerators¶

Tensor Cores are specialized hardware units designed specifically for matrix multiply-accumulate operations — the core math behind deep learning (e.g., convolutions, fully connected layers). They can perform a matrix multiplication in a single clock cycle, which would take many CUDA core cycles.

• What they do: Execute mixed-precision matrix math (e.g., FP16 input, FP32 accumulation) at extremely high throughput.
• Why they matter: Tensor Cores are the secret sauce for modern AI. They deliver up to 60x higher throughput than CUDA cores for matrix operations.
• How many per SM: Again, varies. An A100 SM has 4 Tensor Cores; an H100 SM has 4 Tensor Cores (but each is more powerful).

Comparison: CUDA Cores vs. Tensor Cores

🕵️ Schedulers — The Traffic Controllers¶

Schedulers are the logic units inside each SM that decide which warp gets to execute next. They manage the flow of instructions from the GPU's memory to the execution units (CUDA cores and Tensor Cores).

• What they do: Every clock cycle, the scheduler picks a warp that is ready to run (i.e., not waiting for data) and issues its next instruction to the execution units.
• Why they matter: Without schedulers, execution units would sit idle. Schedulers keep the SM "fed" with work, maximizing utilization.
• How many per SM: Typically 4 schedulers per SM in modern GPUs (e.g., A100, H100). Each scheduler can issue instructions to multiple execution units per cycle.

Key takeaway: Schedulers hide memory latency by quickly switching between warps — this is called latency hiding.

📊 Visual Representation: Streaming Multiprocessor (SM) Core Blocks¶

This diagram maps the internal compute, scheduling, and cache structures contained within a single Streaming Multiprocessor (SM).

📦 Register File — The Ultra-Fast Scratchpad¶

The register file is a massive, high-speed memory bank inside each SM that stores temporary data for active threads. Registers are the fastest memory in the GPU hierarchy (faster than shared memory or global memory).

• What it does: Holds variables, intermediate results, and thread-specific data during execution.
• Why it matters: The register file size limits how many threads an SM can handle simultaneously. If a kernel uses too many registers per thread, the SM can launch fewer threads, reducing parallelism.
• How large: Typically 64KB to 256KB per SM, depending on the GPU generation. For example, an A100 SM has 256KB of register file.

Key takeaway: Register pressure (using too many registers) can hurt performance. Engineers often tune kernel code to balance register usage and thread count.

🛠️ How These Components Work Together¶

Here’s a simplified flow inside an SM:

1. Scheduler picks a warp that is ready to execute.
2. Scheduler issues an instruction (e.g., a matrix multiply or a scalar add).
3. The instruction goes to either CUDA cores (for scalar math) or Tensor Cores (for matrix math).
4. Intermediate results are stored in the register file for fast access.
5. The scheduler immediately switches to another warp while the first warp waits for data — this keeps the execution units busy.

Example scenario: When training a neural network: - Tensor Cores handle the matrix multiplications in forward/backward passes. - CUDA cores handle activation functions (e.g., ReLU) and element-wise operations. - Schedulers keep both types of cores busy by rapidly switching between warps. - Register file stores the weights and intermediate gradients for each thread.

📊 Quick Reference Table: SM Components at a Glance¶

✅ Final Thoughts for New Engineers¶

• Don't overthink the hardware — just remember that Tensor Cores are for AI math, CUDA cores are for everything else, schedulers keep things moving, and the register file is the fast scratchpad.
• When optimizing AI code, focus on using Tensor Cores (via mixed-precision training in frameworks like PyTorch or TensorFlow) and avoid excessive register usage.
• The SM is the heart of the GPU — understanding these four components will help you diagnose performance bottlenecks and write better AI workloads.

You now have a solid foundation for SM internals. Next, you can explore how warps are scheduled or how memory hierarchy (shared memory, L1 cache) interacts with these components. Happy learning!