8.1b GPUs optimized for throughput: thousands of simple cores, SIMD execution¶

🧭 Context Introduction¶

When you first look at a CPU and a GPU side by side, the difference in design philosophy is dramatic. A modern CPU might have 8 to 16 powerful cores, each capable of handling complex tasks independently. A GPU, on the other hand, packs thousands of simpler cores. This isn't a design flaw — it's a deliberate optimization for throughput. In AI workloads, we don't need to run one complex task extremely fast; we need to run millions of simple, identical calculations simultaneously. This is where the GPU's architecture shines.

⚙️ The Core Philosophy: Many Simple Cores vs. Few Complex Cores¶

The fundamental difference between CPUs and GPUs comes down to how they allocate silicon real estate:

• CPU Design: Few cores (8–16), each with large caches, complex control logic, and out-of-order execution capabilities. Optimized for latency — finishing one task as quickly as possible.
• GPU Design: Thousands of cores (2,000–10,000+), each with minimal control logic and small caches. Optimized for throughput — processing many tasks simultaneously, even if each individual task takes slightly longer.

🕵️ SIMD Execution: One Instruction, Many Data¶

The GPU achieves its massive throughput through a technique called SIMD (Single Instruction, Multiple Data). This means:

• One instruction is fetched and decoded once.
• That same instruction is executed across many data elements simultaneously.
• All cores in a group (called a "warp" in NVIDIA terminology) execute the same operation at the same time, but on different pieces of data.

How it works in practice:¶

Imagine you need to add 5 to every element in an array of 1 million numbers:

• CPU approach: Loop through each element one by one, adding 5. This takes 1 million sequential operations.
• GPU approach: Divide the array into groups of 32 or 64 elements. Each group is processed by a warp of cores, all adding 5 simultaneously. This takes roughly 1 million / 32 = 31,250 parallel operations.

This is why matrix multiplications — the core operation in neural networks — are incredibly fast on GPUs. Every element in a matrix multiplication follows the same formula (multiply and accumulate), making it a perfect candidate for SIMD execution.

📊 Visual Representation: Throughput Execution: Thread Hiding and Scheduling¶

This diagram shows how GPUs achieve massive throughput by scheduling instruction warps in parallel, hiding long memory latencies behind thread switching.

📊 Why "Simple Cores" Matter for AI¶

The simplicity of GPU cores is not a limitation — it's a feature that enables massive parallelism:

• Less control logic: Simple cores don't need complex branch prediction or out-of-order execution. This frees up silicon space for more arithmetic logic units (ALUs).
• Lower power per core: Simple cores consume less energy per operation, allowing the GPU to run thousands of cores within a reasonable power budget.
• Higher density: You can pack more simple cores onto a single chip than you could complex cores. An NVIDIA H100 GPU has over 18,000 CUDA cores.

What this means for AI workloads:¶

• Matrix multiplications: The backbone of neural networks. GPUs can process entire matrix blocks in a single SIMD operation.
• Convolutions: Used in image processing and CNNs. Each filter application is identical across different image regions.
• Element-wise operations: Activation functions (ReLU, sigmoid) apply the same formula to every neuron output.

🛠️ Real-World Example: A Simple Vector Addition¶

To see the difference in practice, consider adding two vectors of 1,024 elements each:

CPU approach (sequential): - Loop through index 0 to 1,023. - For each index, add the two values and store the result. - Total operations: 1,024 sequential additions.

GPU approach (parallel with SIMD): - Launch a kernel with 1,024 threads (one per element). - Group threads into warps of 32. - Each warp executes the same add instruction on 32 different pairs of numbers simultaneously. - Total operations: 32 warps × 1 instruction each = 32 parallel steps.

The GPU completes the task roughly 32 times faster than a single CPU core — and modern GPUs can launch thousands of such warps simultaneously.

🧠 Key Takeaway for Engineers¶

When designing or optimizing AI infrastructure, remember:

• GPUs are not faster CPUs — they are fundamentally different tools optimized for different problems.
• Throughput is the goal — don't worry if a single GPU core is slower than a CPU core. The strength comes from having thousands of them working in parallel.
• SIMD-friendly algorithms — AI workloads naturally map to SIMD execution because they involve repetitive, predictable math on large datasets.
• Batching is critical — to fully utilize GPU throughput, you must batch multiple inputs together. A single input leaves most cores idle.

Understanding this design philosophy helps you make better decisions about when to use GPUs (parallel, data-heavy tasks) versus CPUs (sequential, branching, or latency-sensitive tasks) in your AI infrastructure.