27.1b Operator architecture: the operator pattern and Custom Resource Definitions (CRDs)¶

🧭 Context Introduction¶

When you're managing AI workloads on Kubernetes, you quickly realize that standard Kubernetes resources (like Pods, Deployments, and Services) aren't enough to handle specialized hardware like NVIDIA GPUs. You need a way to tell Kubernetes: "Hey, I need a GPU, and I need the right drivers, tools, and configurations installed automatically."

This is where Operators and Custom Resource Definitions (CRDs) come in. They extend Kubernetes to understand and manage complex, stateful applications — like the NVIDIA GPU software stack — as if they were native Kubernetes objects.

Think of an Operator as a smart, automated administrator that watches over a specific application and takes action when things change. The CRD is the blueprint that defines what that application looks like in Kubernetes.

⚙️ What is the Operator Pattern?¶

The Operator pattern is a method of packaging, deploying, and managing a Kubernetes application. It uses custom controllers that watch for changes to custom resources and automatically perform tasks to keep the system in the desired state.

Key Concepts:¶

• Controller: A loop that continuously watches the state of resources and reconciles the current state with the desired state.
• Custom Resource (CR): An instance of a CRD — like a specific configuration for your GPU setup.
• Reconciliation Loop: The process of checking the current state, comparing it to the desired state, and taking corrective actions.

How it Works in Simple Terms:¶

1. You define what you want (e.g., "I want GPU drivers version 535 installed on all nodes").
2. You create a Custom Resource with those specifications.
3. The Operator's controller sees the new resource.
4. The controller takes action (installs drivers, configures tools, etc.).
5. The controller continuously monitors and fixes any drift from the desired state.

📊 What are Custom Resource Definitions (CRDs)?¶

A Custom Resource Definition (CRD) is a way to extend the Kubernetes API. It defines a new type of resource that Kubernetes doesn't natively understand.

Analogy:¶

• Standard Kubernetes resources are like pre-built furniture (chairs, tables).
• CRDs are like a custom furniture blueprint — you define exactly what you want, and the Operator builds it for you.

Common CRD Examples in AI Infrastructure:¶

🛠️ The Operator Architecture — How It All Fits Together¶

The NVIDIA GPU Operator follows a layered architecture:

Layer 1: CRDs (The Blueprints)¶

• Define what resources the Operator can manage.
• Stored in the Kubernetes API server.
• Examples: ClusterPolicy, NVIDIADriver, NVIDIAToolkit.

Layer 2: Controllers (The Brains)¶

• Watch for changes to CRDs.
• Execute reconciliation logic.
• Interact with Kubernetes API to create/update/delete standard resources (DaemonSets, ConfigMaps, etc.).

Layer 3: Operands (The Workers)¶

• The actual software components that get deployed.
• Examples: GPU driver DaemonSet, NVIDIA Container Toolkit, MIG Manager.

Flow Diagram (Text-Based):¶

You create a ClusterPolicy CR
        ↓
Controller detects the new CR
        ↓
Controller reads the desired state (e.g., driver version, toolkit settings)
        ↓
Controller creates standard Kubernetes resources (DaemonSets, ConfigMaps)
        ↓
Kubernetes schedules and runs the operands on nodes
        ↓
Controller monitors and ensures the state stays correct

📊 Visual Representation: GPU Operator Controller Components¶

This diagram details the core pods deployed by the GPU Operator to manage local hardware nodes.

🕵️ Why This Matters for AI Infrastructure¶

For engineers managing GPU fleets, the Operator pattern solves several critical problems:

• Automation: No more manually installing GPU drivers on every node.
• Consistency: Every node gets the exact same configuration.
• Self-Healing: If a node reboots or a driver crashes, the Operator re-deploys the correct state.
• Version Management: Easily upgrade or rollback GPU software across the entire cluster.
• Lifecycle Management: The Operator handles upgrades, scaling, and cleanup automatically.

📋 Comparison: Traditional vs. Operator-Based Management¶

🎯 Real-World Example: NVIDIA GPU Operator in Action¶

When you deploy the NVIDIA GPU Operator, here's what happens behind the scenes:

1. You apply a ClusterPolicy CR that specifies:
2. Driver version: 535.154.05
3. Toolkit enabled: true

4. MIG mode: none

5. The Operator's controller detects this new CR and:

6. Creates a DaemonSet to install the driver on all GPU nodes.
7. Creates a ConfigMap with toolkit configuration.

8. Deploys the NVIDIA device plugin for Kubernetes.

9. The controller continuously watches:

10. If a new GPU node joins the cluster, the driver is automatically installed.
11. If a driver pod crashes, it's automatically restarted.
12. If you update the CR to a new driver version, the Operator performs a rolling update.

✅ Key Takeaways for New Engineers¶

• Operators are like automated site reliability engineers (SREs) for specific applications — they watch, react, and fix.
• CRDs are the language you use to tell the Operator what you want — they define the "what," not the "how."
• The NVIDIA GPU Operator uses this pattern to manage the entire GPU software stack — from drivers to container runtimes to monitoring tools.
• You don't need to write Operators to use them — you just need to understand how to create and manage the CRs.
• The Operator pattern is the standard way to manage complex, stateful applications in Kubernetes — it's used for databases, message queues, and of course, AI infrastructure.

🔍 Next Steps for Learning¶

• Explore the NVIDIA GPU Operator documentation to see all available CRDs.
• Practice creating a simple ClusterPolicy CR in a test cluster.
• Observe how the Operator reacts when you modify the CR.
• Learn about the Helm chart that simplifies deploying the Operator itself.

Remember: The Operator pattern is your friend. It takes the repetitive, error-prone work of managing GPU infrastructure and turns it into a declarative, automated process. As you gain experience, you'll appreciate how much time and headache it saves when operating AI workloads at scale.