Repository navigation
kubelet parameter(eviction-max-pod-grace-period ), not work as expected like officical comment. #118172
Description
What happened?
Hey brothers , feedback a confusing point about kubelet soft eviction:
Just like kubelet --help show:
--eviction-max-pod-grace-period int32 Maximum allowed grace period (in seconds) to use when terminating pods in response to a soft eviction threshold being met. If negative, defer to pod specified value.
I guess if this parameter is set to negative ,such as -1, soft eviction would use pod specified value(TerminationGracePeriodSeconds),
But When I try to evict pod by creating node pressure(such as memory pressure), I found it's always -1 send to CRI runtime.
Then pod container stopped immediately with sigkill, exit 137.
In short， this parameter：
（1）set as active number：work as expected；
（2）no set， kubelet read its default number：0， work as expected；
（3）set as negative number: not work as kubelet help show.
Thanks for your response. May it be a bug ?
What did you expect to happen?
--eviction-max-pod-grace-period set to negative, then kubelet syncPod logic send pod‘s TerminationGracePeriodSeconds as gracePeriod to CRI runtime to stop container.
How can we reproduce it (as minimally and precisely as possible)?
(1) set this parameter to a negative number, such as -1;
(2) config kubelet soft eviction ;
(3) running a pod asking a lot memory , to trigger node memory pressure;
(4) check CRI runtime log, such as containerd ;
(5) then you will find timeout parameter -1 sent to pod container , then container exit with code 137(sigkill);
Anything else we need to know?
I running these tests on Tencent Cloud TKE.
And I thought it seem not related to Cloud Service Provider.
Kubernetes version
Details
Client Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.3-tke.34", GitCommit:"61cc96d2f7e9277e89e29b4b04d045f27d6e75df", GitTreeState:"clean", BuildDate:"2023-03-08T07:40:04Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.3-tke.32", GitCommit:"8ca5807ae095365d68754a2ba20b10bbe5a8998c", GitTreeState:"clean", BuildDate:"2022-10-08T04:00:33Z", GoVersion:"go1.12.14", Compiler:"gc", Platform:"linux/amd64"}
Cloud provider
Details
Tencent Cloud TKEOS version
Details
AME="Ubuntu" VERSION="18.04.4 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.4 LTS" VERSION_ID="18.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=bionic UBUNTU_CODENAME=bionicInstall tools
Details
Container runtime (CRI) and version (if applicable)
Details
containerd --version
containerd github.com/containerd/containerd v1.4.3-tke.2 a11500cbfa0b4d2fc9b905e03c35f349ef5b1a9f
Related plugins (CNI, CSI, ...) and versions (if applicable)
Details
Metadata
Metadata
Assignees
Labels
Type
Projects
- Status
Activity
tzneal commentedon May 22, 2023
/sig node
SergeyKanzhelev commentedon May 24, 2023
/triage accepted
The question is whether we fix code and make the backward incompatible change or fix a doc. I'd vote for a doc change which is less work. But blocks an interesting scenarios.
@rainbowBPF2 were you looking for the described behavior? Does your scenario requires this?
/priority important-longterm
rainbowBPF2 commentedon May 25, 2023
@SergeyKanzhelev
Thanks for brothers‘ attention on this point.
Because we have a more reasonable choice to terminate pods in case of soft-eviction, to realize Pod-Graceful-Shutdown
In short，no required but deep expected, 😄。
fgksgf commentedon May 25, 2023
/assign
k8s-triage-robot commentedon May 24, 2024
This issue has not been updated in over 1 year, and should be re-triaged.
You can:
/triage accepted(org members only)/closeFor more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/
/remove-triage accepted
54 remaining items
kubelet: fix eviction-max-pod-grace-period=-1 sending SIGKILL
frank-bee commentedon Mar 11, 2026
A fix for this issue has been submitted in PR #137624.
The bug was in
pkg/kubelet/eviction/eviction_manager.go: the soft eviction path usedmin(MaxPodGracePeriodSeconds, podGracePeriod), which whenMaxPodGracePeriodSeconds=-1always evaluated to-1. The CRI runtime then received-1and immediately SIGKILLed the container (exit code 137), ignoringterminationGracePeriodSecondsentirely.The fix checks for a negative value first and defers to the pod's own
TerminationGracePeriodSeconds(or the default 30s if unset), matching the documented flag behavior.SergeyKanzhelev commentedon Mar 24, 2026
Can you please read the top of the thread and comment on how to address backward incompatibility of the code fix. This issue needs to build the case: why code fix is better, what is the worst case scenario of breaking existing behavior for customers, and how can the backward incompatibility be mitigated
nimdrak commentedon Mar 25, 2026
Hi @SergeyKanzhelev,
Could you please take a look at this PR when you have a moment?
It clarifies the Kubelet help text and docs for
--eviction-max-pod-grace-periodto match the actual behavior. You’ve already given it a LGTM, but it still needs a formal /approve to move forward from another.I’ve tried reaching out on Slack and mentioning others, but haven't had any luck yet.
I’d appreciate your help in getting this merged!
If it is merged, we can close this issue.
kunalworldwide commentedon May 26, 2026
@SergeyKanzhelev re the hold on #137624 — trying to answer the backward-compat question.
The operators most likely affected are anyone using
-1as a "just SIGKILL it immediately" shortcut during soft eviction. It's not documented that way, but I've seen it used like that in practice — memory pressure situations where people don't want pods hanging around. After this fix, those pods would hang for up toterminationGracePeriodSecondsbefore terminating, which could be a real surprise under pressure.Two ways to handle that:
EvictionNegativeGracePeriodFix, off by default for one release) so people have time to adapt0if you want SIGKILL behavior"Personally the flag is documented to mean "defer to pod value" so the current behavior is already a bug, not a feature. A release note feels proportionate. But if the concern is that enough clusters are relying on the current broken behavior, a feature gate is the safer path.
Does that cover what you were looking for, or is there a specific scenario I'm missing?
neoLsH commentedon Aug 24, 2026
/assign
Hariharanpugazh commentedon Sep 1, 2026
/assign
reetig commentedon Sep 28, 2026
/assign
kubernetes-prow commentedon Sep 28, 2026
@reetig: GitHub didn't allow me to assign the following users: reetig.
Note that only kubernetes members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide
Details
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
Hariharanpugazh commentedon Sep 29, 2026
Opened #142504 to fix this. When
--eviction-max-pod-grace-periodis negative, soft eviction now defers to the pod'sterminationGracePeriodSeconds(or the 30s default), instead of sending the negative value to the CRI and triggering an immediate SIGKILL.On the backward-compat question raised earlier: the only behavior that changes is for clusters relying on a negative value as an "immediate kill" shortcut, which contradicts the documented flag contract. I've added a release note; happy to add a feature gate for one release if that's the preferred rollout. Regression tests included (they fail on the current code and pass with the fix).
kpdagrt22 commentedon Oct 6, 2026
/assign
kubernetes-prow commentedon Oct 6, 2026
@kpdagrt22: GitHub didn't allow me to assign the following users: kpdagrt22.
Note that only kubernetes members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide
Details
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
kpdagrt22 commentedon Oct 6, 2026
I have reproduced this locally (min(-1, podGrace) passes -1 to CRI) and opened #142714 with a fix plus unit test. The issue is at the 10-assignee cap so /assign could not apply; requesting a review when possible.