37.1b Part IV–V: GPU Architecture and Networking — 90 questions with rationales¶

Welcome, new engineer! This section is designed to help you build a solid foundation in GPU architecture and networking — two critical pillars of modern AI infrastructure. You don't need prior experience with GPUs or data center networking; we'll break everything down into simple, digestible concepts. This practice bank contains 90 questions with rationales to reinforce your learning.

⚙️ Section 1: GPU Architecture Fundamentals (Questions 1–30)¶

🧠 What is a GPU?¶

• A GPU (Graphics Processing Unit) is a specialized processor designed for parallel computation.
• Unlike a CPU (which handles a few complex tasks quickly), a GPU handles thousands of simpler tasks simultaneously.
• This makes GPUs ideal for AI workloads like training neural networks.

🔑 Key GPU Architecture Concepts¶

• CUDA Cores: The basic processing units inside an NVIDIA GPU. More cores = more parallel processing power.
• Tensor Cores: Specialized cores for matrix math — essential for deep learning operations.
• Memory Bandwidth: How fast data moves between GPU memory and cores. Higher bandwidth = faster training.
• VRAM (Video RAM): The GPU's dedicated memory. Larger VRAM allows larger models and batch sizes.

📊 GPU Memory Hierarchy Comparison¶

🕵️ Sample Question with Rationale¶

Question: Why do AI workloads benefit more from GPU architecture than CPU architecture?

Rationale: GPUs contain thousands of smaller cores designed for parallel execution, while CPUs have fewer, more powerful cores optimized for sequential tasks. AI training involves massive matrix multiplications that can be split across many cores simultaneously.

🛠️ Section 2: GPU Memory & Data Flow (Questions 31–60)¶

🔄 How Data Moves Through a GPU¶

1. Host to Device: Data transfers from CPU RAM to GPU VRAM (via PCIe or NVLink).
2. Kernel Execution: GPU cores process data in parallel.
3. Device to Host: Results transfer back to CPU memory.

⚡ Critical Performance Factors¶

• PCIe Bandwidth: The speed of the connection between CPU and GPU. Gen4 and Gen5 offer higher throughput.
• NVLink: NVIDIA's high-speed interconnect for GPU-to-GPU communication (faster than PCIe).
• Memory Coalescing: Arranging memory access patterns so the GPU can read/write data efficiently.

📊 Data Transfer Bottlenecks¶

🕵️ Sample Question with Rationale¶

Question: What is the primary advantage of NVLink over PCIe for multi-GPU setups?

Rationale: NVLink provides significantly higher bandwidth and lower latency than PCIe, enabling faster direct GPU-to-GPU communication. This is critical for training large models that must be split across multiple GPUs.

📊 Visual Representation: GPU Networking Exam concept mapping¶

This diagram displays networking topics tested in the exam, focusing on InfiniBand speeds and RoCE transport layers.

🌐 Section 3: GPU Networking Fundamentals (Questions 61–90)¶

🖧 Why Networking Matters for AI¶

• Large AI models often exceed a single GPU's memory, requiring distributed training across multiple servers.
• Networking connects these servers, allowing them to share gradients, parameters, and data.
• Slow networking creates a bottleneck that wastes expensive GPU compute time.

🔗 Key Networking Technologies¶

• InfiniBand: High-speed, low-latency interconnect designed for HPC and AI clusters (commonly 200–400 Gbps).
• RoCE (RDMA over Converged Ethernet): Enables remote direct memory access over standard Ethernet networks.
• NVIDIA Quantum InfiniBand: NVIDIA's latest switch technology for AI fabrics.

📊 Network Topologies for AI Clusters¶

🕵️ Sample Question with Rationale¶

Question: Why is low latency critical in GPU networking for distributed training?

Rationale: During distributed training, GPUs must synchronize gradients after each training step. If network latency is high, GPUs sit idle waiting for data from other nodes, dramatically reducing overall throughput. Low latency ensures all GPUs stay busy.

📋 Comprehensive Practice Questions (90 Total)¶

🔹 Questions 1–10: GPU Core Architecture¶

1. What is the primary difference between CUDA cores and Tensor Cores?

2. Rationale: CUDA cores handle general-purpose parallel computing, while Tensor Cores are optimized specifically for matrix multiply-accumulate operations used in deep learning.

3. How does GPU memory bandwidth affect training performance?

4. Rationale: Higher memory bandwidth allows the GPU to feed data to cores faster, reducing idle time and improving throughput.

5. What happens when a model exceeds available VRAM?

6. Rationale: The GPU must use system RAM (via PCIe), which is much slower, causing severe performance degradation or out-of-memory errors.

7. Why do modern GPUs have multiple memory controllers?

8. Rationale: Multiple controllers increase total memory bandwidth by allowing parallel access to different memory banks.

9. What is the role of the GPU scheduler?

10. Rationale: It manages which warps (groups of threads) execute on the GPU, hiding memory latency by switching between warps.

11. How does tensor core precision (FP16 vs FP32) impact performance?

12. Rationale: Lower precision (FP16) allows faster computation and uses less memory, but may reduce model accuracy if not managed properly.

13. What is a GPU "warp"?

14. Rationale: A group of 32 threads that execute simultaneously on a CUDA core. All threads in a warp must execute the same instruction.

15. Why is GPU occupancy important?

16. Rationale: High occupancy means more warps are active, allowing the GPU to hide memory latency by switching between warps.

17. What is the purpose of L1 cache in a GPU?

18. Rationale: It stores frequently accessed data close to the cores, reducing latency compared to global memory.

19. How does GPU clock speed affect performance?

    • Rationale: Higher clock speeds increase the rate at which cores execute instructions, but also increase power consumption and heat.

🔹 Questions 11–20: Memory Hierarchy & Data Movement¶

1. What is the difference between unified memory and dedicated GPU memory?

   • Rationale: Unified memory allows CPU and GPU to share the same memory pool, simplifying programming but potentially reducing performance.

2. How does PCIe Gen4 compare to Gen3 for GPU data transfer?

   • Rationale: Gen4 offers double the bandwidth (16 GT/s vs 8 GT/s per lane), reducing data transfer bottlenecks.

3. What is memory coalescing and why does it matter?

   • Rationale: Coalescing means adjacent threads access adjacent memory addresses, maximizing memory bandwidth utilization.

4. How does data locality affect GPU performance?

   • Rationale: Keeping data close to the cores (in shared memory or cache) reduces expensive global memory accesses.

5. What is the role of the GPU driver in memory management?

   • Rationale: The driver handles allocation, deallocation, and transfer of memory between CPU and GPU.

6. Why is page-locked (pinned) memory used for GPU transfers?

   • Rationale: Pinned memory prevents the OS from swapping pages to disk, enabling faster DMA transfers between CPU and GPU.

7. How does multi-GPU memory pooling work?

   • Rationale: Multiple GPUs can share a unified memory address space, allowing models larger than any single GPU's VRAM.

8. What is the impact of memory fragmentation on GPU workloads?

   • Rationale: Fragmentation can prevent large contiguous memory allocations, causing out-of-memory errors even when total free memory is sufficient.

9. How does GPU memory bandwidth scale with more GPUs?

   • Rationale: Total bandwidth increases linearly with the number of GPUs, but inter-GPU communication overhead may reduce efficiency.

10. What is the purpose of GPU memory compression?

    • Rationale: Compression reduces the amount of data transferred between memory and cores, improving effective bandwidth.

🔹 Questions 21–30: GPU Compute Capabilities¶

1. What is compute capability in NVIDIA GPUs?

   • Rationale: A version number that indicates which features and instructions a GPU supports (e.g., tensor cores, ray tracing).

2. How does GPU compute capability affect software compatibility?

   • Rationale: Older GPUs may not support newer CUDA features, requiring fallback code paths or reduced performance.

3. What is the difference between FP32 and TF32 precision?

   • Rationale: TF32 uses 19-bit mantissa (vs 23-bit for FP32) but same exponent range, offering faster tensor core operations with minimal accuracy loss.

4. Why do AI models benefit from mixed precision training?

   • Rationale: Using FP16 for most operations and FP32 for critical parts speeds up training while maintaining accuracy.

5. What is the role of the GPU's instruction cache?

   • Rationale: It stores frequently executed kernel instructions, reducing instruction fetch latency.

6. How does GPU branch divergence affect performance?

   • Rationale: When threads in a warp take different code paths, all paths must be executed serially, reducing parallelism.

7. What is warp divergence and how can it be minimized?

   • Rationale: Occurs when threads in a warp follow different conditional branches. Minimized by restructuring code to avoid branching within warps.

8. How does GPU register pressure impact performance?

   • Rationale: Too many registers per thread reduces the number of active warps, decreasing occupancy and hiding memory latency.

9. What is the purpose of GPU shared memory?

   • Rationale: Fast, programmer-controlled memory that allows threads within a block to share data efficiently.

10. How does GPU cache hierarchy differ from CPU cache hierarchy?

    • Rationale: GPUs have larger L2 caches shared across all cores, while CPUs have larger per-core L1 caches.

🔹 Questions 31–40: Multi-GPU Architectures¶

1. What is NVLink and how does it differ from PCIe?

   • Rationale: NVLink is a high-bandwidth, low-latency interconnect for GPU-to-GPU communication, while PCIe is a general-purpose bus.

2. How does NVLink bandwidth compare to PCIe Gen5?

   • Rationale: NVLink can provide 900 GB/s+ bidirectional bandwidth per GPU, while PCIe Gen5 x16 offers ~64 GB/s.

3. What is the purpose of NVSwitch?

   • Rationale: A switch that connects multiple GPUs via NVLink, creating a fully connected fabric for all-to-all communication.

4. How does GPU topology affect distributed training performance?

   • Rationale: GPUs connected via NVLink can communicate faster than those connected via PCIe, reducing synchronization overhead.

5. What is the DGX platform?

   • Rationale: NVIDIA's integrated AI system combining multiple GPUs, NVSwitch, and high-speed networking in a single chassis.

6. How does GPU direct memory access (GPU-Direct) work?

   • Rationale: Allows GPUs to read/write data directly from/to network interfaces or storage, bypassing CPU memory.

7. What is the role of NCCL (NVIDIA Collective Communications Library)?

   • Rationale: Optimizes multi-GPU communication operations like all-reduce and broadcast for distributed training.

8. How does NCCL handle GPU-to-GPU communication?

   • Rationale: It automatically selects the fastest available path (NVLink, PCIe, or network) based on GPU topology.

9. What is the difference between synchronous and asynchronous GPU communication?

   • Rationale: Synchronous communication blocks until data is received, while asynchronous allows overlap with computation.

10. How does GPU memory bandwidth affect multi-GPU scaling?

    • Rationale: Limited memory bandwidth can become a bottleneck when multiple GPUs share the same memory controller.

🔹 Questions 41–50: Network Protocols for AI¶

1. What is InfiniBand and why is it used in AI clusters?

   • Rationale: A high-speed, low-latency interconnect that supports RDMA, ideal for distributed training requiring fast gradient synchronization.

2. How does RoCE compare to InfiniBand?

   • Rationale: RoCE runs over standard Ethernet, offering lower cost but potentially higher latency than InfiniBand.

3. What is RDMA and why is it important for GPU networking?

   • Rationale: Remote Direct Memory Access allows GPUs to read/write memory on remote nodes without CPU involvement, reducing latency.

4. How does TCP/IP differ from RDMA for GPU communication?

   • Rationale: TCP/IP involves CPU processing and multiple memory copies, while RDMA bypasses the CPU for lower latency.

5. What is the role of GPUDirect RDMA?

   • Rationale: Enables direct data transfer between GPU memory and network interface, avoiding CPU buffer copies.

6. How does network latency affect distributed training convergence?

   • Rationale: Higher latency increases the time for gradient synchronization, potentially slowing convergence if batch sizes are not adjusted.

7. What is the difference between all-reduce and all-gather operations?

   • Rationale: All-reduce sums gradients across all nodes and distributes the result, while all-gather collects data from all nodes without reduction.

8. How does ring all-reduce work?

   • Rationale: Data is passed in a ring topology, with each node reducing a portion and forwarding to the next, minimizing bandwidth usage.

9. What is the impact of network bandwidth on training throughput?

   • Rationale: Insufficient bandwidth creates a bottleneck where GPUs wait for data, reducing overall throughput.

10. How does network topology affect all-reduce performance?

    • Rationale: Topologies with higher bisection bandwidth (like fat trees) allow faster all-reduce operations.

🔹 Questions 51–60: Network Topologies & Design¶

1. What is bisection bandwidth in a network?

   • Rationale: The minimum bandwidth between any two halves of the network, critical for all-to-all communication patterns.

2. How does a fat-tree topology benefit AI training?

   • Rationale: Provides full bisection bandwidth, ensuring no single link becomes a bottleneck for distributed training.

3. What is the difference between leaf-spine and traditional three-tier networks?

   • Rationale: Leaf-spine provides uniform latency between any two nodes, while three-tier has variable latency.

4. How does oversubscription affect GPU cluster performance?

   • Rationale: Oversubscription means less bandwidth than needed, causing contention and slower training.

5. What is the role of network switches in GPU clusters?

   • Rationale: Switches forward data between nodes, with higher-port-count switches enabling larger clusters.

6. How does network latency scale with cluster size?

   • Rationale: Larger clusters may have more hops between nodes, increasing latency if not designed carefully.

7. What is the purpose of network congestion control in AI fabrics?

   • Rationale: Prevents packet loss and maintains low latency during bursty communication patterns.

8. How does QoS (Quality of Service) help in GPU networking?

   • Rationale: Prioritizes critical AI traffic (like gradient sync) over less time-sensitive data.

9. What is the difference between storage and compute networks?

   • Rationale: Storage networks handle data loading, while compute networks handle inter-GPU communication during training.

10. How does network redundancy affect AI cluster reliability?

    • Rationale: Redundant paths prevent single points of failure, but may increase complexity and cost.

🔹 Questions 61–70: NVIDIA Networking Technologies¶

1. What is NVIDIA Quantum InfiniBand?

   • Rationale: NVIDIA's 400 Gbps InfiniBand switch platform with in-network computing capabilities.

2. How does NVIDIA Spectrum Ethernet compare to Quantum InfiniBand?

   • Rationale: Spectrum is for Ethernet-based AI fabrics, while Quantum is for InfiniBand, each optimized for different deployment scenarios.

3. What is the role of BlueField DPUs in AI networking?

   • Rationale: Data Processing Units offload networking and storage tasks from CPUs, freeing resources for AI workloads.

4. How does NVIDIA ConnectX NIC support GPU networking?

   • Rationale: ConnectX SmartNICs provide RDMA and GPUDirect support for efficient GPU-to-GPU communication.

5. What is the purpose of NVIDIA Magnum IO?

   • Rationale: A suite of software tools optimizing data movement between GPUs, storage, and networks.

6. How does NVIDIA GPUDirect Storage work?

   • Rationale: Enables direct data transfer between GPUs and storage devices, bypassing CPU memory.

7. What is the role of NVIDIA SHARP (Scalable Hierarchical Aggregation and Reduction Protocol)?

   • Rationale: Offloads all-reduce operations to network switches, reducing GPU overhead.

8. How does NVIDIA NetQ help with network operations?

   • Rationale: Provides real-time visibility and troubleshooting for AI fabrics.

9. What is the difference between NVIDIA Air and physical network testing?

   • Rationale: NVIDIA Air is a simulation platform for testing network designs without physical hardware.

10. How does NVIDIA DOCA (Data Center Infrastructure-on-a-Chip Architecture) support AI networking?

    • Rationale: Provides APIs for programming BlueField DPUs to accelerate AI workloads.

🔹 Questions 71–80: Performance Optimization¶

1. How does gradient compression reduce network bandwidth needs?

   • Rationale: Compressing gradients before transmission reduces data volume, but may affect model accuracy.

2. What is the impact of batch size on network utilization?

   • Rationale: Larger batch sizes produce larger gradients, increasing network bandwidth requirements.

3. How does gradient accumulation affect communication frequency?

   • Rationale: Accumulating gradients over multiple steps reduces communication frequency but increases memory usage.

4. What is the role of asynchronous communication in training?

   • Rationale: Allows computation to continue while communication happens in the background, hiding latency.

5. How does model parallelism differ from data parallelism in networking needs?

   • Rationale: Model parallelism requires more frequent, smaller communications, while data parallelism uses less frequent, larger communications.

6. What is pipeline parallelism and its networking implications?

   • Rationale: Spl