Nodes in the lost AZ are unreachable. The kube-controller-manager detects this and marks those nodes as Unknown, and the cloud provider controller deletes the instances. Pods that were running on those nodes are marked as Terminating and eventually Unknown. Kubernetes recreates pods on remaining nodes if the ReplicaSet or Deployment controller is healthy. Pods with PodDisruptionBudgets that were violated during the outage are recreated once the PDB allows. StatefulSets with volumes may not reschedule immediately since their PersistentVolumes in the lost AZ are unavailable — the StatefulSet controller waits for the volumes or for the pod to be force-deleted. Recovery steps: verify the cluster is healthy after the AZ returns, check PersistentVolume claims and manually delete any pods stuck in Terminating state (kubectl delete pod --grace-period=0 --force). Verify auto-scaling was triggered and pods are back to desired count.

With 3-node etcd, you can tolerate 1 member failure without cluster unavailability. The cluster continues serving requests with 2 nodes. The failed member's log snapshot falls behind. Recovery depends on whether the data is recoverable: if the node comes back quickly, etcd automatically rejoins and catches up from the leader. If the node is permanently lost, you must remove it from the etcd cluster using etcdctl member remove, then add a new member using etcdctl member add. For managed Kubernetes (GKE, EKS), the control plane handles this automatically — the managed etcd redundancy is one of the main benefits of managed clusters. For self-managed: always run etcd on dedicated nodes with proper monitoring on etcd cluster health metrics.

For managed clusters, use the platform's rolling upgrade feature — GKE, EKS, and AKS upgrade the control plane nodes without cluster downtime by using zero-downtime configurations. For self-managed: upgrade etcd first (since API server depends on it), then kube-apiserver, then controller-manager and scheduler. Use the kubectl cordon and drain approach on control plane nodes — treat them like worker nodes for upgrade purposes. Maintain etcd backups before any control plane changes using etcdctl snapshot save. The golden rule: never upgrade multiple control plane components simultaneously, and always verify each component is healthy before proceeding to the next. For single-node development clusters, downtime is unavoidable — schedule upgrades during maintenance windows.

Expected answer points:

• HPA flapping is rapid scale-up and scale-down cycles that waste resources and cause instability
• It happens when the stabilization window is too short or scaling thresholds are too tight
• A pod that scales to 20, triggers scale-down, bounces back, is a classic symptom
• Set stabilizationWindowSeconds on scale-down to give the system time to settle (5 minutes is typical)
• Use scale-up stabilization for stateful services to avoid connection churn during brief load fluctuations

Voluntary disruptions are actions initiated by administrators or the Kubernetes system: node drain operations for upgrades or repairs, kubectl drain commands, HPA scale-down events, and cluster autoscaler node removal. Involuntary disruptions are unplanned events outside cluster control: node hardware failures, cloud provider availability zone outages, network partitions, and kernel panics.

PDB only protects against voluntary disruptions. This is why PDB design must account for actual risk: an involuntary AZ outage that takes down nodes is not protected by a PDB, but the pods will be recreated by ReplicaSets once the cluster recovers. The distinction matters because PDBs cannot prevent all failure modes — they only ensure minimum availability during deliberate cluster operations. For true multi-AZ protection, combine PDBs with podAntiAffinity and topologySpreadConstraints so that involuntary disruptions also leave minimum replicas available.

topologySpreadConstraints provide more granular control than podAntiAffinity alone. With podAntiAffinity, you can enforce that pods spread across zones, but you cannot control the exact balance — the scheduler might place 2 pods in one AZ and 1 in another if that is the only option. topologySpreadConstraints let you define a maxSkew value that enforces the maximum allowed difference in pod count between topology domains.

Example: with maxSkew=1 and topologyKey=topology.kubernetes.io/zone, the scheduler ensures that no zone has more than one extra pod compared to the smallest zone. If you have 3 replicas and 3 zones, you get exactly 1 pod per zone. If a zone fails, the remaining 2 zones each get 1 pod and the failed zone has 0 — this is balanced. Additionally, topologySpreadConstraints work with the scheduler's even-packing logic to achieve better resource utilization while maintaining balance, which anti-affinity cannot do. For StatefulSets where you need N replicas across K zones, this is essential.

The interaction can create a thrashing scenario if not designed correctly. During a node drain, the ReplicaSet controller notices desired replicas versus available replicas and might signal to HPA that capacity is reduced. Without a PDB, HPA sees the reduced replica count and scales up to compensate. The newly created pods get scheduled, then the node drain continues and evicts them again, creating a cycle of pod creation and eviction.

With a PDB in place, the eviction is rate-limited — the PDB prevents more than N pods being unavailable at once. This actually gives HPA time to react without thrashing. However, if the PDB minAvailable is set too high (equal to the replica count), the drain blocks entirely. Best practice: set HPA minReplicas high enough that PDBs can allow a drain without triggering HPA to think the service is under-replicated. Pairing HPA with a properly configured PDB and using percentage-based minAvailable (e.g., 50%) prevents this interaction from causing problems.

minAvailable specifies the minimum number of pods that must remain available — if you set 3, at least 3 pods must be up at all times. maxUnavailable specifies the maximum number of pods that can be unavailable simultaneously — if you set 1, at most 1 pod can be down.

minAvailable is expressed as an absolute number or percentage of total replicas. Use minAvailable when you know the minimum your service needs to handle current load, and you want the PDB to protect that floor. This is useful for stateful services where N replicas are needed for quorum.

maxUnavailable is easier to reason about when you know your replica count and want to express "at most X can be down." A PDB with maxUnavailable: 1 on a 3-replica deployment means 2 are always up (67% availability). Use maxUnavailable when you want to ensure a specific number of replicas remain active during disruptions and you are comfortable with a fixed count rather than a percentage.

For services where replica count might change dynamically (HPA-enabled), maxUnavailable as a percentage is safer because it automatically scales with replica count. For stateful services with quorum requirements, minAvailable absolute values are clearer.

A DaemonSet ensures that exactly one pod runs on each node (or on nodes matching a selector). For HA, this is valuable for log collectors, metrics agents, and networking plugins that must run everywhere. If a node fails, the DaemonSet controller does not recreate the pod on that node — the pod is gone with the node. On new nodes added to the cluster, DaemonSet pods are automatically scheduled. This means DaemonSets do not provide resilience against node failures the way ReplicaSets do.

DaemonSet HA limitations: there is no built-in replica count or scaling — a DaemonSet pod exists once per node, not N times regardless of node count. For a 5-node cluster, you get 5 pods. If 2 nodes fail, you have 3 pods. If your workload needs exactly 3 replicas for HA regardless of node count, a Deployment with 3 replicas is better. DaemonSets are best for cluster-level services (monitoring, logging, networking) rather than application HA.

An ingress controller (nginx, Traefik, cloud LB) should be deployed as multiple replicas distributed across AZs with podAntiAffinity using topology.kubernetes.io/zone. The cloud provider's load balancer sits in front of the ingress replicas and distributes traffic. For AWS ALB/NLB, enable cross-zone load balancing so the load balancer can route to any ingress pod in any AZ.

Key design points: use a Deployment (not DaemonSet) with minReplicas >= 3 and podAntiAffinity to enforce zone spreading. Configure the cloud LB health check to detect ingress pod failures and remove failed pods from rotation. Set appropriate readinessProbe so the LB only sends traffic to ready ingress pods. For nginx ingress, use the -controller elections to ensure only one pod handles configuration updates to prevent conflicts. Consider using a PodDisruptionBudget on the ingress replicas so that upgrades do not cause brief unavailability.

Kubernetes API server latency directly affects etcd write latency because every API server write goes through etcd. If API server request latency increases (e.g., from 10ms to 500ms), every kubectl apply, pod create, and deployment update slows proportionally. For a cluster handling 1000 writes per second, this compounds quickly.

etcd write latency is critical because etcd uses the Raft consensus protocol — writes require quorum from a majority of etcd nodes. Slow writes cause cascading effects: the API server times out waiting for etcd, kubelets report node heartbeats late, the scheduler becomes sluggish, and the controllers (ReplicaSet, Deployment) slow down. In the worst case, repeated API server timeouts cause watch streams to close, forcing controllers to re-list resources and potentially miss reconciliation events.

Best practices: keep etcd on dedicated nodes with SSD storage, monitor etcd_disk_wal_fsync_duration_seconds and etcd_server_leader_changes_total, and ensure etcd instances are in the same region as the API server. For managed Kubernetes, cloud providers handle this optimization.

Node pools allow you to segment nodes by capability (compute optimized, memory optimized, GPU) while maintaining HA. For production, use multiple node pools: a general-purpose pool for standard workloads, a system pool for cluster critical components (DNS, monitoring), and an application pool sized for your workloads.

HA considerations for node pools: distribute nodes across AZs within each pool — a single large node pool with 10 nodes should have nodes in 3 AZs, not all 10 in one AZ. Use Cluster Autoscaler with node pool awareness so that scale-up events add nodes in the right AZ. Pod placement using topologySpreadConstraints and node affinity ensures your workloads use the right pool while maintaining zone balance. Do not mix system pods with application pods on the same nodes if you need guaranteed resource availability during node pressure.

For workloads with specific requirements (e.g., GPU for ML), a dedicated node pool with taints and tolerations isolates those workloads while allowing the autoscaler to scale that pool independently.

Readiness probes determine whether a pod receives traffic through a Service endpoint. PDBs determine whether a pod can be evicted during voluntary disruptions. These are orthogonal but both critical for availability. A pod that fails its readiness probe is removed from Service endpoints, so it receives no traffic even if it is running. A pod that passes its readiness probe but is protected by a PDB cannot be evicted.

During a disruption, if a pod's readiness probe fails before eviction, the Service already removed it from endpoints — the eviction causes no traffic drop. If the readiness probe is too aggressive and marks pods as not ready during normal load fluctuations, you lose capacity unnecessarily. PDBs protect against eviction of pods that are otherwise healthy. Set readiness probes to detect genuine unavailability (process crash, health endpoint returning 500), not load-related latency spikes, so that pods remain available during normal traffic variation.

Managed Kubernetes (GKE, EKS, AKS) runs the control plane as a managed service with built-in HA. The etcd cluster is replicated across multiple nodes managed by the cloud provider, with automatic failure detection and recovery. API server instances are behind load balancers with health checks, and the platform handles rolling updates of control plane components without cluster downtime. You do not see or manage these nodes — you trust the SLA.

Self-managed HA control plane requires you to provision multiple etcd nodes (odd number for quorum), multiple API server nodes behind a load balancer, and multiple controller-manager and scheduler instances with leader election. The etcd cluster needs explicit quorum management — if you lose the majority, the cluster goes read-only or down. You are responsible for upgrades, backups, and failure recovery. The trade-off is control versus operational burden. For most production workloads, managed Kubernetes with a multi-control-plane offering (e.g., GKE standard tier) provides sufficient HA without the operational complexity.

etcd is the state store for the entire cluster — every object (pods, services, configmaps, secrets) is stored there. Write performance directly controls API server responsiveness. The critical metrics to monitor: etcd_disk_wal_fsync_duration_seconds (write-ahead log sync time — should be under 10ms), etcd_server_leader_changes_total (frequent leader changes indicate network or load issues), etcd_server_proposals_failed_total (failed Raft proposals indicate quorum problems), and apiserver_request_latencies (API latency spikes often trace back to etcd).

When etcd write latency increases above 50ms, API server request latency increases proportionally. kubelet health checks start timing out, causing pods to be marked Unknown. Controller reconciliation loops slow down, causing delayed deployment rollouts and HPA reactions. At extreme latency (>500ms), the cluster can become effectively unresponsive to kubectl commands. Defensive measures: use etcd with SSD on dedicated nodes, enable etcd metrics in Prometheus, alert on apiserver_storage_latency_seconds and etcd_server_leader_changes_total.

StatefulSet failure modes during AZ outage depend heavily on storage binding. StatefulSets with PersistentVolumeClaims bound to volumes in a single AZ will lose those volumes if the AZ goes down — pods cannot reschedule because their storage is unavailable. The StatefulSet controller waits for the volume to become available again (which may take until the AZ recovers) or requires manual force-deletion of the stuck pods.

Mitigation strategies: use regional storage solutions (e.g., AWS EFS, GCE PD Regional Persistent Disks) that replicate data across AZs rather than zonal volumes. For databases that need low-latency storage, configure zone-pinned storage with a PDB that prevents all replicas from being in the same AZ. For critical StatefulSets, deploy at least one replica in each AZ and use podAntiAffinity to prevent co-location. Accept that during an AZ outage, your StatefulSet will have reduced capacity — design your application to handle degraded replica count with reduced throughput.

Zero-downtime control plane upgrades require careful sequencing. First, back up etcd using etcdctl snapshot save. Then upgrade etcd on all control plane nodes (do not upgrade multiple nodes simultaneously — upgrade one, verify health, then proceed). etcd must be upgraded before the API server because the API server depends on etcd. After etcd is healthy, upgrade the API server on one node at a time, verifying that kubectl continues to work after each node's upgrade.

After API server, upgrade controller-manager and scheduler. These use leader election, so the non-leader can be upgraded first with no service impact. The leader upgrade requires failing over leadership — this is handled automatically by the component but should be monitored. Finally, upgrade kubelet and kube-proxy on worker nodes using cordon, drain, upgrade, uncordon sequence. Use kubectl drain with PodDisruptionBudgets to ensure applications remain available throughout the node upgrade process. Never upgrade more than one control plane node at a time, and always verify cluster health after each component upgrade before proceeding.

Pod affinity rules tell the scheduler to place pods relative to other pods (same node, same zone, or different zone). Anti-affinity tells the scheduler to avoid placing pods near other pods. During scheduling, the scheduler evaluates these rules alongside resource availability and other constraints.

requiredDuringSchedulingIgnoredDuringExecution enforces hard constraints — if a pod cannot satisfy affinity rules (e.g., no node in the preferred zone has capacity), scheduling fails. preferredDuringSchedulingIgnoredDuringExecution is a soft preference — the scheduler tries to satisfy it but will place the pod elsewhere if it cannot. For zone spreading, use requiredDuringSchedulingIgnoredDuringExecution with topologyKey=topology.kubernetes.io/zone for strict distribution, or preferred for soft balancing. The scheduler treats each topologyKey value as a separate domain and attempts to spread pods evenly across domains while respecting affinity constraints.

During a rolling update, the Deployment controller creates new pods before terminating old ones (rolling update strategy). If you try to drain a node during this process, the PDB constraint combines with the rolling update. The Deployment has 5 pods running, then creates 1 new pod (6 total). When you drain a node with PDB minAvailable: 4, the drain attempts to evict pods but the PDB blocks evictions if it would bring available pods below 4.

With 6 pods and PDB minAvailable: 4, 2 pods can be evicted. The drain proceeds, removes pods from the draining node, the ReplicaSet controller recreates them on other nodes. After the rolling update completes, you are back to 5 pods. If the rolling update is also in progress, you could have 7 pods temporarily (5 old + 1 new being created + 1 being rolled out). The drain still respects PDB and only evicts up to 2 pods. This demonstrates why it is important to ensure rolling update strategy and PDB are coordinated — a PDB set too high can block both rolling updates and node drains simultaneously.

Cluster-level anti-affinity with topologyKey=topology.kubernetes.io/zone spreads pods across availability zones. This protects against entire AZ failures but increases network latency — a pod in us-east-1a talking to a pod in us-east-1b has higher latency than two pods in the same AZ. For stateless web applications where requests are short-lived and load-balanced, this latency is usually acceptable noise.

Node-level anti-affinity (topologyKey=kubernetes.io/hostname) spreads pods across individual nodes within a single AZ. This protects against node-level failures (hardware failure, kernel panic) but not AZ failures. Network latency between nodes in the same AZ is minimal. For latency-sensitive applications where pods frequently call each other, node-level spreading may be preferable for lower network latency, accepting the risk of AZ-level failures.

For production internet-facing applications, the best approach combines both: use topologySpreadConstraints with maxSkew=1 and topologyKey=topology.kubernetes.io/zone to spread across zones, with a soft preferred preference for kubernetes.io/hostname to also spread within a zone when possible. This gives AZ-level failure protection with some node-level spreading automatically.