Repository navigation
Node lifecycle controller does not markPodsNotReady when the node Ready state changes from false to unknown #112733
Description
What happened?
When kubelet loses connect, the node goes into the unknown state. The node lifecycle controller marks the pod as not ready by the markPodsNotReady function because the health check status of the pod can not be obtained through kubelet. This feature is available only when node's Ready state transitions from true to unknown.
However, if the node is already in the fail state (such as a containerd failure), markPodsNotReady will not take effect if the node loses its connection at this time.
kubernetes/pkg/controller/nodelifecycle/node_lifecycle_controller.go
Lines 883 to 888 in cac5388
In this case, the pod may accidentally remain ready, which may cause some network traffic to be accidentally forwarded to this node.
What did you expect to happen?
As long as the node loses its connection beyond grace time, MarkPodsNotReady should always work
How can we reproduce it (as minimally and precisely as possible)?
- Stop containerd and wait for the node
Readystate to false - Stop kubelet or shutdown the node and wait the node
Readystate to unknown - The pods which not be evicted on this node would be always ready
Anything else we need to know?
In the node lifecycle controller logic,MarkPodsNotReady is just triggered when a node goes from true state to an unknown state. The correct way is to trigger when the node becomes unknown state regardless of whether the node state was previously true
Kubernetes version
Details
$ kubectl version
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.15", GitCommit:"1d79bc3bcccfba7466c44cc2055d6e7442e140ea", GitTreeState:"clean", BuildDate:"2022-09-22T06:03:36Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}Cloud provider
Details
OS version
Details
# On Linux:
$ cat /etc/os-release
$ uname -a
5.4.119-1-tlinux4-0008 #1 SMP Fri Nov 26 11:17:45 CST 2021 x86_64 x86_64 x86_64 GNU/Linux
# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output hereInstall tools
Details
Container runtime (CRI) and version (if applicable)
Details
Related plugins (CNI, CSI, ...) and versions (if applicable)
Details
Metadata
Metadata
Labels
Type
Projects
- Status
Activity
[-]Node lifecycle controller does not `markPodsNotReady` when the node `Ready` state changes from `fail` to `unknown`[/-][+]Node lifecycle controller does not `markPodsNotReady` when the node `Ready` state changes from `false` to `unknown`[/+]answer1991 commentedon Sep 26, 2022
/sig node
bobbypage commentedon Sep 26, 2022
Thanks for the report and the simple repro steps. This sounds like a generalized duplicate of #109998
/triage accepted
akankshakumari393 commentedon Sep 28, 2022
/assign
akankshakumari393 commentedon Sep 28, 2022
Is this a
good-first-issue. I would like to work on it?SergeyKanzhelev commentedon Sep 28, 2022
/triage accepted
SergeyKanzhelev commentedon Sep 28, 2022
Yes, I believe the change required is very localized.
35 remaining items
ch0wdreN commentedon Dec 6, 2025
/assign
Fix node lifecycle controller to mark pods not ready on False->Unknow…
Merge branch 'dev/fix-mark-pod-not-ready' into 'tke/v1.32.2' (merge r…
ngpayyappilly commentedon May 18, 2026
/assign
node lifecycle controller: mark pods not ready on false to unknown tr…
mohitdebian commentedon Aug 3, 2026
Hello! I opened PR (#141089) addressing the node lifecycle controller behavior when node readiness transitions from
FalsetoUnknown, ensuring pods are marked NotReady appropriately.aryawadhwa commentedon Aug 10, 2026
Hi all! I've been investigating this issue locally and found that the node lifecycle controller indeed misses the
False->Unknowntransition for executingMarkPodsNotReady. I've successfully implemented a fix by updating the transition checks around line 756 innode_lifecycle_controller.goto also queue retries forcurrentReadyCondition.Status == v1.ConditionUnknown && observedReadyCondition.Status == v1.ConditionFalse, and falling through to executeMarkPodsNotReady. I've also added unit tests for this specific state transition which are passing. Let me know if I can raise a PR for this, or if #141089 has already addressed it!mohitdebian commentedon Aug 10, 2026
Hi @aryawadhwa, thanks for looking into this! PR #141089 is already open for this and includes the state transition unit tests. I just pushed a minor refactor that also ensures the retry queues correctly on API server failures during the transition. Feel free to drop by the PR and take a look!
neoLsH commentedon Aug 24, 2026
/assign
mohitdebian commentedon Aug 25, 2026
hey folks, thanks for looking into this! just a heads up that there's already an open PR (#141089) with the fix and tests for this that's currently waiting for review. feel free to drop by that PR and take a look!
kpdagrt22 commentedon Oct 6, 2026
/assign