34.1b Azure: NDv5 (H100), NDmv4 (A100) VM series¶

🌐 Context Introduction¶

When you are training large AI models or running complex inference workloads in the cloud, you need virtual machines (VMs) that are purpose-built for high-performance computing (HPC) and GPU-accelerated tasks. Microsoft Azure offers two flagship GPU VM families for these workloads: the NDv5 series (powered by NVIDIA H100 GPUs) and the NDmv4 series (powered by NVIDIA A100 GPUs). These VMs are designed to handle the most demanding AI training, deep learning, and scientific simulation tasks. For a new engineer, think of these as the "heavy lifters" of Azure's AI infrastructure — they provide the raw compute power, memory bandwidth, and networking speed needed to train models like GPT, BERT, or large vision transformers.

⚙️ What Are NDv5 and NDmv4 VM Series?¶

• NDv5 series — The latest generation, featuring NVIDIA H100 Tensor Core GPUs. These are built for next-generation AI workloads, including large language models (LLMs) and generative AI. They offer significant improvements in performance over the previous generation.
• NDmv4 series — The previous generation, featuring NVIDIA A100 Tensor Core GPUs. These are still very powerful and widely used for training and inference of medium-to-large AI models. They are a cost-effective option for many production workloads.

Both series are part of Azure's ND (N-series for Deep Learning) family, optimized for distributed training with high-speed interconnects.

📊 Key Differences Between NDv5 and NDmv4¶

🛠️ When to Use Each Series¶

• Use NDv5 (H100) when:
• You are training very large models (e.g., 100B+ parameter LLMs).
• You need the fastest possible training time for generative AI.
• You require the highest memory bandwidth and inter-GPU communication speed.

• Your workload benefits from the new Transformer Engine and FP8 precision support in H100.

• Use NDmv4 (A100) when:

• Your model fits within the capabilities of A100 GPUs.
• You are looking for a more cost-effective solution for production training or inference.
• You need a proven, stable platform with broad regional availability.
• Your workload does not require the extreme performance of H100.

📊 Visual Representation: Azure ND-Series GPU instances¶

This diagram displays Azure GPU instances, highlighting NDv5 (H100) and NDv4 (A100) topologies with InfiniBand interconnects.

🕵️ Key Technical Specifications (Simplified)¶

NDv5 (H100) — Example Instance: Standard_ND96isr_H100_v5¶

• GPUs: 8 x NVIDIA H100 (80 GB each)
• Total GPU Memory: 640 GB
• vCPUs: 96 (AMD EPYC)
• System Memory: 1900 GB RAM
• Interconnect: 8 x NVLink 4.0 bridges per GPU pair, plus 8 x Quantum-2 InfiniBand (400 Gb/s each)
• Storage: Supports NVMe SSDs (local) and Azure Managed Disks

NDmv4 (A100) — Example Instance: Standard_ND96asr_v4¶

• GPUs: 8 x NVIDIA A100 (80 GB each)
• Total GPU Memory: 640 GB
• vCPUs: 96 (AMD EPYC)
• System Memory: 900 GB RAM
• Interconnect: 8 x NVLink 3.0 bridges per GPU pair, plus 8 x HDR InfiniBand (200 Gb/s each)
• Storage: Supports NVMe SSDs (local) and Azure Managed Disks

🔧 How to Get Started (No Commands, Just Steps)¶

1. Choose your region — Check Azure region availability for NDv5 or NDmv4 VMs. Some regions may have limited capacity.
2. Select the right VM size — For most AI training, you will use the largest size (e.g., ND96isr_H100_v5 or ND96asr_v4) to get 8 GPUs. Smaller sizes exist for lower-scale workloads.
3. Configure networking — These VMs require InfiniBand for high-speed GPU-to-GPU communication. Ensure your virtual network is set up with accelerated networking and InfiniBand support.
4. Attach storage — Use Azure Managed Disks for OS and data, and optionally attach local NVMe SSDs for high-speed scratch space.
5. Install software stack — You will need NVIDIA drivers, CUDA toolkit, and a deep learning framework (e.g., PyTorch, TensorFlow, JAX). Azure provides pre-configured images (e.g., Data Science Virtual Machine) that include these.
6. Launch your training job — Use a job scheduler (e.g., SLURM, Azure Batch, or direct SSH) to start your distributed training script.

💡 Pro Tips for New Engineers¶

• Always use InfiniBand for multi-GPU or multi-node training. Ethernet is too slow for efficient scaling.
• Monitor GPU utilization — Use tools like nvidia-smi (run from the terminal) to check if your GPUs are fully utilized. Low utilization means your code may have bottlenecks.
• Start with NDmv4 if you are new to Azure GPU VMs — it is more forgiving on cost and has better availability.
• Use Azure Spot VMs for non-critical or interruptible training jobs to save up to 90% on compute costs.
• Leverage Azure's pre-built HPC images — They come with optimized drivers and libraries already installed.

📚 Summary¶

Both the NDv5 and NDmv4 series are excellent choices for AI infrastructure on Azure. As a new engineer, start by understanding your workload's GPU memory and communication requirements, then choose the series that best fits your budget and performance needs.