26.2b etcd: the distributed key-value store — the authoritative cluster state

📦 Cluster Orchestration 📖 Kubernetes Fundamentals

🧠 Context Introduction

In a Kubernetes cluster, every decision the control plane makes—like scheduling a GPU workload, scaling a deployment, or detecting a node failure—depends on knowing the current state of the cluster. Where is this "state" stored? That's where etcd comes in.

etcd is a distributed, consistent key-value store that acts as the single source of truth for your entire Kubernetes cluster. Think of it as the cluster's memory: if etcd goes down or becomes corrupted, the cluster loses its ability to function. For engineers managing AI infrastructure at scale, understanding etcd is critical because it directly impacts cluster reliability, recovery, and performance.

⚙️ What is etcd?

  • Distributed: etcd runs across multiple nodes (typically 3, 5, or 7) to provide high availability.
  • Key-Value Store: Data is stored as simple key-value pairs (e.g., key: /registry/pods/default/my-gpu-pod, value: the pod's full specification).
  • Consistent: Uses the Raft consensus algorithm to ensure all nodes agree on the same data, even if some nodes fail.
  • Authoritative: Kubernetes writes all cluster state changes (pod creation, node registration, secret updates) to etcd. It reads from etcd to make scheduling and recovery decisions.

📊 How Kubernetes Uses etcd

Component What It Stores in etcd Why It Matters for AI Ops
API Server All API objects (pods, services, configmaps, secrets) Every kubectl apply or kubectl get reads/writes to etcd
Scheduler Node availability, pod binding decisions Decides which GPU node gets the next training job
Controller Manager Desired vs. actual state (replicas, node health) Ensures your GPU pods are always running as intended
etcd itself Cluster membership, leader election, Raft logs Keeps the control plane alive during node failures

🛠️ Key Concepts for New Engineers

🧩 Raft Consensus Algorithm

  • etcd uses Raft to elect a leader node. All writes go through the leader.
  • If the leader fails, a new leader is elected automatically.
  • Quorum is required: for a cluster of 3 nodes, at least 2 must agree on any write. This means you can lose 1 node and still operate.

📦 Data Model

  • Data is stored in a hierarchical key space (like a filesystem).
  • Example keys:
  • /registry/pods/default/my-training-job
  • /registry/nodes/gpu-node-01
  • /registry/secrets/ai-model-weights
  • All data is versioned—every change creates a new revision, enabling watch-based updates.

🔄 Watches

  • Kubernetes components watch etcd for changes. When a pod is created, the API server writes to etcd, and the scheduler sees the change immediately.
  • This is how the cluster reacts in real-time without polling.

📊 Visual Representation: etcd Raft Consensus Database

This diagram displays etcd: storing the source-of-truth state across a distributed raft-replicated node cluster.

flowchart LR APIServer["kube-apiserver Write"] --> Leader["etcd Leader Node"] Leader -->|Raft Replication| Follower1["etcd Follower 1"] Leader -->|Raft Replication| Follower2["etcd Follower 2"] classDef cpu fill:#eafaf1,stroke:#76b900,stroke-width:2px,rx:6px,ry:6px; classDef memory fill:#f0f7ff,stroke:#3498db,stroke-width:1.5px,rx:4px,ry:4px; classDef system fill:#f1f5f9,stroke:#64748b,stroke-width:1.5px; class Leader cpu; class Follower1,Follower2 memory; class APIServer system;

🕵️ Why etcd Matters for AI Infrastructure

  • GPU Workload Reliability: If etcd is slow or unavailable, the scheduler cannot assign GPU pods to nodes. Your training jobs stall.
  • Disaster Recovery: Backing up etcd is the only way to restore a cluster after a catastrophic failure. Without a valid etcd snapshot, you lose all cluster state.
  • Performance Tuning: Large clusters (1000+ nodes) can put heavy write load on etcd. Engineers must monitor etcd latency and disk I/O to prevent bottlenecks.
  • Security: etcd stores secrets (like API keys for model registries) in plaintext by default. Engineers must enable encryption at rest.

📈 Best Practices for Engineers

  • Always run an odd number of etcd members (3, 5, or 7) to maintain quorum.
  • Use dedicated SSD storage for etcd—it is extremely sensitive to disk latency.
  • Monitor etcd metrics:
  • Leader changes (should be rare)
  • Database size (keep under 8 GB for optimal performance)
  • fsync duration (should be under 10ms)
  • Regularly back up etcd using etcdctl snapshot save. Test your restore process.
  • Never directly edit etcd—always use kubectl or the Kubernetes API. Direct edits can corrupt the cluster.

🧪 Simple Analogy

Imagine a library (your Kubernetes cluster): - The books (pods, services, nodes) are the resources. - The librarian (API server) manages the books. - The card catalog (etcd) stores the exact location and status of every book. - If the card catalog is lost or damaged, no one knows where anything is—the library stops working.

As an engineer, your job is to keep the card catalog healthy, backed up, and fast.

✅ Summary

  • etcd is the distributed key-value store that holds the authoritative state of your Kubernetes cluster.
  • It uses the Raft consensus algorithm to ensure consistency and high availability.
  • Every Kubernetes component reads from and writes to etcd—it is the cluster's single source of truth.
  • For AI infrastructure, etcd health directly impacts GPU workload scheduling, cluster recovery, and overall reliability.
  • Engineers must monitor, back up, and tune etcd to keep large-scale GPU fleets running smoothly.