🗺️ Roadmap: The Operating System Layer — Linux for AI Infrastructure Operators¶

Below is the syllabus timeline of lessons in this module. Track your study progress here.

📈 Progress¶

---

📁 3 Linux System Fundamentals — The OS From the Ground Up¶

• 3.1a The role of the OS: hardware abstraction, resource scheduling, security
• 3.1b Monolithic vs. microkernel architectures
• 3.1c System calls: how user programs request kernel services
• 3.1d The /proc and /sys virtual filesystems: live kernel data in files
• 3.2a Stage 1: BIOS/UEFI POST and hardware initialization
• 3.2b Stage 2: GRUB bootloader — menu, kernel selection, and initrd/initramfs
• 3.2c Stage 3: Kernel loading, module initialization, and mounting root filesystem
• 3.2d Stage 4: systemd PID 1 — the init system that bootstraps all services
• 3.2e Debugging boot failures: reading systemd logs and kernel ring buffer (dmesg)
• 3.3a Ubuntu Server LTS: the dominant choice for AI labs — release cadence and support lifecycle
• 3.3b Red Hat Enterprise Linux (RHEL) and CentOS Stream for enterprise environments
• 3.3c SUSE Linux Enterprise: use cases in HPC and financial sectors
• 3.3d Minimal vs. Desktop installs: why AI servers use headless minimal installations
• 3.4a Units: service, socket, target, mount, timer — the systemd object model
• 3.4b systemctl commands: start, stop, enable, disable, status, mask
• 3.4c Dependency ordering: After=, Requires=, Wants= in unit files
• 3.4d journalctl: filtering by time, unit, priority, and following live logs
• 3.4e Creating custom unit files for AI training services and watchdog processes
• 3.4f systemd-analyze blame: profiling boot time and identifying slow services

📁 4 Core Linux Administration for AI Operators¶

• 4.1a Filesystem Hierarchy Standard (FHS): the purpose of every top-level directory
• 4.1b /etc — system configuration files; /var — variable runtime data; /opt — third-party software
• 4.1c /dev — device files: block devices (sda, nvme0n1) vs character devices (tty, random)
• 4.1d /proc — process and kernel information; /sys — hardware and driver sysfs interface
• 4.1e Absolute vs. relative paths, and navigating with cd, ls, find, locate
• 4.2a /etc/passwd and /etc/shadow: anatomy of user account files
• 4.2b useradd, usermod, userdel, passwd: user lifecycle management
• 4.2c groupadd, gpasswd: managing groups for shared GPU access
• 4.2d The sudoers file and visudo: safely granting administrative privileges
• 4.2e LDAP and Active Directory integration for enterprise user management
• 4.3a Read, Write, Execute bits for User, Group, Other — decoding ls -la output
• 4.3b Octal notation: 755, 644, 600 — what every combination means
• 4.3c chmod, chown, chgrp: changing permissions and ownership
• 4.3d Special bits: setuid, setgid, and the sticky bit
• 4.3e ACLs (Access Control Lists): getfacl and setfacl for granular control
• 4.3f umask: default permission masking for new files and directories
• 4.4a APT (Debian/Ubuntu): apt update, apt install, apt-cache, sources.list
• 4.4b YUM/DNF (RHEL/CentOS): package groups, modules, and repositories
• 4.4c Adding NVIDIA package repositories for drivers and CUDA
• 4.4d Pinning package versions to prevent accidental upgrades of GPU drivers
• 4.4e Snap packages, AppImage, and when to use tarball manual installs
• 4.5a Vim essentials: modes, navigation, insert, save, quit — the minimum to survive
• 4.5b grep, awk, sed: searching, filtering, and transforming text streams
• 4.5c cut, sort, uniq, wc, tr: building command pipelines with pipes and redirection
• 4.5d Parsing GPU logs and DCGM output with awk and grep
• 4.5e Heredocs and tee: redirecting output while still displaying it
• 4.6a Variables, quoting rules, and command substitution ($())
• 4.6b Conditionals: if/elif/else, test expressions, and compound conditions
• 4.6c Loops: for, while, until — iterating over GPU lists and node arrays
• 4.6d Functions, return codes, and exit status best practices
• 4.6e Writing health-check scripts for GPU cluster nodes

📁 5 Storage & Resource Management for AI Workloads¶

• 5.1a top and htop: load average, CPU%, MEM%, process states explained
• 5.1b vmstat: virtual memory, swap, and I/O statistics per second
• 5.1c iostat: block device I/O throughput — identifying storage bottlenecks
• 5.1d lsof and fuser: finding open files and processes using GPU device files
• 5.1e free, /proc/meminfo: dissecting memory usage — buffers, cache, available
• 5.1f sar (System Activity Reporter): historical resource trending
• 5.2a Block device naming: /dev/sda, /dev/nvme0n1 and partition naming conventions
• 5.2b Partitioning with fdisk and gdisk: MBR vs GPT partition tables
• 5.2c Creating filesystems: mkfs.ext4, mkfs.xfs — choosing for AI workloads
• 5.2d Mounting filesystems: mount command, /etc/fstab, and persistent mounts
• 5.2e NFS client mounting: automounting shared training datasets
• 5.3a LVM concepts: Physical Volumes (PV), Volume Groups (VG), Logical Volumes (LV)
• 5.3b Creating and extending volumes across multiple NVMe drives
• 5.3c Snapshots: point-in-time copies for backup before model updates
• 5.3d Thin provisioning for checkpoint storage optimization
• 5.4a RAID 0, 1, 5, 6, 10: performance and redundancy trade-offs
• 5.4b Linux software RAID with mdadm
• 5.4c Hardware RAID controllers: HBA passthrough for NVMe performance

📁 6 Linux Networking & Security Hardening¶

• 6.1a Network interface naming: ens3, eth0, ib0 — predictable naming and exceptions
• 6.1b Configuring static IPs with netplan (Ubuntu 18.04+) and YAML syntax
• 6.1c NetworkManager: nmcli and nmtui for interactive configuration
• 6.1d Bonding and teaming: aggregating multiple interfaces for redundancy and bandwidth
• 6.1e Jumbo frames (MTU 9000): required for high-performance AI network traffic
• 6.2a ip addr, ip link, ip route: the modern replacements for ifconfig and route
• 6.2b ping and ping6: ICMP-based connectivity testing
• 6.2c traceroute and mtr: path tracing for latency diagnosis
• 6.2d ss (socket statistics): replacing netstat for port and connection analysis
• 6.2e tcpdump: packet capture and filtering for deep network debugging
• 6.2f iperf3: measuring raw network throughput between cluster nodes
• 6.3a How SSH works: key exchange, symmetric encryption, and MACs
• 6.3b Generating RSA and Ed25519 key pairs with ssh-keygen
• 6.3c ssh-copy-id and authorized_keys: passwordless authentication
• 6.3d /etc/ssh/sshd_config: disabling password auth, changing port, AllowUsers
• 6.3e SSH tunneling, jump hosts, and ~/.ssh/config for multi-node clusters
• 6.3f Managing cluster-wide SSH with pssh and Ansible for parallelism
• 6.4a iptables fundamentals: chains, tables, and rules
• 6.4b ufw (Uncomplicated Firewall): the Ubuntu-friendly interface
• 6.4c fail2ban: automatic IP blocking after failed SSH attempts
• 6.4d Security-Enhanced Linux (SELinux) and AppArmor: mandatory access control
• 6.4e audit daemon (auditd): tracking system calls for security compliance