Draining a Node for Maintenance in Kubernetes

Introduction

Performing maintenance on a Kubernetes node requires safely evicting running pods without causing service disruptions. Kubernetes provides the kubectl drain command to gracefully remove a node from the cluster while ensuring workload continuity.

Why Drain a Node?

• Applying OS or Kubernetes updates
• Performing hardware upgrades or troubleshooting
• Scaling down nodes in a cluster

Draining a Node Safely

Use the following command to safely drain a node:

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

Explanation of Flags:

• --ignore-daemonsets: Ensures that daemonset-managed pods (like logging or monitoring agents) are not evicted.
• --delete-emptydir-data: Ensures that data stored in emptyDir volumes is deleted.
• --force: Can be used to force eviction when pods have no associated controllers (use with caution).

Best Practices for Node Draining

1. Check Running Pods Before Drainingkubectl get pods --all-namespaces --field-selector spec.nodeName=<node-name>This helps identify which workloads will be affected.
2. Ensure Pod Disruption Budgets (PDBs) are ConfiguredapiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: my-app-pdb spec: minAvailable: 1 selector: matchLabels: app: my-appThis prevents more than the allowed number of replicas from being evicted simultaneously.
3. Cordon the Node Before Drainingkubectl cordon <node-name>This prevents new pods from being scheduled on the node before draining.
4. Monitor Pod Evictionskubectl get pods -o wide --watchEnsures all evictions occur without errors before proceeding.
5. Uncordon the Node After Maintenancekubectl uncordon <node-name>This allows the node to accept new pods again.

Alternative Solutions

Using Node Taints

Instead of draining, apply a taint to prevent scheduling while allowing existing pods to run:

kubectl taint nodes <node-name> key=value:NoSchedule

Using Cluster Autoscaler

For managed Kubernetes services like EKS, GKE, and AKS, cluster autoscaler can handle node removal and rebalancing automatically.

Conclusion

Using kubectl drain is a safe way to prepare a Kubernetes node for maintenance without disrupting applications. Following best practices ensures minimal downtime and workload continuity.

---