Repository navigation
[Bug] Logs report "failed to acquire semaphore" during deletion #7818
Description
What were you trying to accomplish?
Delete the cluster (it seem to work fine).
What happened?
Logs report this message during deletion:
[ℹ] deleting EKS cluster "redacted"
[ℹ] will drain 0 unmanaged nodegroup(s) in cluster "redacted"
[ℹ] starting parallel draining, max in-flight of 1
[✖] failed to acquire semaphore while waiting for all routines to finish: %!w(*errors.errorString=&{context canceled})
Deletion still finished without errors, so it does not look like this affect anything. But the log does look like there's a problem.
The behavior is reproduced on all attempts.
How to reproduce it?
Create cluster with 1 managed nodegroup and no unmanaged nodegroups, then delete it.
Versions
eksctl version: 0.180.0
EKS version 1.30
The message was not present on eksctl version 0.176.0 with EKS version 1.29 (there are no changes in cluster config besides EKS version).
Reacted by Chris B, Todd Thomas and Ilya Egorov
Activity
cPu1 commentedon Jun 6, 2024
@artem-nefedov, this is a bug in the logging and concurrency handling but it should not affect normal operation of the command. That part of the codebase is a bit dated and could use some refactoring. We'll look into this soon.
lgb861213 commentedon Jul 6, 2024
We also encountered the same error log information when deleting EKS version 1.30, and our eksctl version is 0.183.
the error message that is following:
2024-07-06 16:56:08 [ℹ] deleting EKS cluster "test"
2024-07-06 16:56:11 [ℹ] will drain 0 unmanaged nodegroup(s) in cluster "aloda-test"
2024-07-06 16:56:11 [ℹ] starting parallel draining, max in-flight of 1
2024-07-06 16:56:11 [✖] failed to acquire semaphore while waiting for all routines to finish: %!w(*errors.errorString=&{context canceled})
2024-07-06 16:56:14 [ℹ] deleted 0 Fargate profile(s)
2024-07-06 16:56:16 [✔] kubeconfig has been updated
2024-07-06 16:56:16 [ℹ] cleaning up AWS load balancers created by Kubernetes objects of Kind Service or Ingress
2024-07-06 16:56:23 [ℹ]
AmitBenAmi commentedon Jul 10, 2024
Seeing the same issue with version
0.185.0acarey-haus commentedon Jul 18, 2024
Seeing this issue with eksctl version
0.187.0when deleting a nodegroup. The deletion succeeded.AndrewFarley commentedon Oct 11, 2024
Guys, this is my main selling point and favorite part of EKSCTL. Without this working, there's far less reason to use this tool. What this ends up doing below is AGGRESSIVELY removing all the nodes when the ASG is removed without draining gracefully. This is fairly critical folks, please fix ASAP. If you can't fix ASAP, then it should at least catch this condition and not just skip over draining without an
--force-removeor something argument. This is an outage-causing bug.Versions
EKSCTL Versions occurred for me: 0.183.0, so then I upgraded to 0.191.0, happened on both
EKS 1.29
Debug output / proof