Here are some important things you need to know about pod preemption:

How is Pod QoS related to Pod Priority & Preemption?

In the Pod QoS lesson, we learned that when a node faces resource pressure (like CPU or memory shortage), the kubelet removes pods based on their QoS class (Guaranteed, Burstable, BestEffort). This eviction process doesn't consider pod priority or the scheduling queue.

However, Pod priority based preemption happens only when high-priority pods are waiting in the scheduling queue. In this case, the scheduler removes lower-priority pods without considering their QoS class.

Simply put:

• QoS-based eviction: Handled by kubelet during resource shortages (no scheduling queue involved).
• Priority-based preemption: Handled by the scheduler when higher-priority pods wait in the scheduling queue (QoS is ignored).

What is the effect of PodDisruptionBudget on Preemption?

PodDisruptionBudget (PDB) limits how many pods Kubernetes can evict (remove) at the same time during planned actions like upgrades or maintenance.

For example, you have 5 pods running, and set a PDB saying at least 3 pods must stay running, Kubernetes normally respects this rule and won’t evict more than 1 pod at a time.

When the scheduler considers pods for preemption, it respects PDB constraints where possible. It tries not to violate a PDB by ensuring the minimum number of pods specified in the PDB remains running.

However, if a very high-priority pod needs to be scheduled, Kubernetes will still remove lower-priority pods, even if it violates the PDB rules.

How to Handle Evictions Gracefully?

When a pod is being removed (preemption or eviction), we must make sure the application shuts down smoothly.

How can we do this?

Here are some ways:

1. Add graceful shutdown handlers to application code: These catch the SIGTERM signal and do things like:
   1. Close database connections
   2. Finish ongoing requests or tasks
   3. Save data in memory or clear caches
2. Set a proper shutdown time: In the Pod settings, set a terminationGracePeriodSeconds. The default is 30 seconds, but you can change it based on what your app needs.
3. Use preStop hooks (Container Lifecycle Hooks): This lets you run custom cleanup steps before the pod shuts down.
4. Add readiness probes: These help make sure traffic doesn't go to a pod that’s shutting down or not ready.
5. Test your setup: Try eviction and shutdown cases in a test environment to see how your app behaves.