Repository navigation
[Bug] Degraded managedNodeGroups when using a pathed instanceRoleARN #7846
Description
What were you trying to accomplish?
We launch EKS clusters using instanceRoleARN to attach managed policies (AmazonEKSWorkerNodePolicy, AmazonEKS_CNI_Policy, AmazonEC2ContainerRegistryReadOnly) to our node group instances.
We provided a path on these roles of "/eks/" for organizational purposes. We'd like to be able to manage these node groups, but the pathing seems to cause a degradation in node group health.
What happened?
The cluster creates as expected but after about an hour or so the node group shows up as degraded
It's a little tough to tell with the redactions, but the ARN shown in the "Affected resources" column lacks the /eks/ path prefix.
Removing the path parameter from the role seems to avoid the issue.
How to reproduce it?
We use a eksctl config template like this:
managedNodeGroups:
- name: stable-{{ .CLUSTER_NAME_WITH_HYPHENS }}
instanceType: r5.8xlarge
desiredCapacity: 2
minSize: 2
maxSize: 2
privateNetworking: true
volumeSize: 40
volumeType: gp3
volumeEncrypted: true
labels:
stable: "true"
tags:
<<: *tags
iam:
instanceRoleARN: {{ .STABLE_NODES_ROLE_ARN }}
Where the instance role ARN is "arn:aws:iam::ACCOUNT:role/eks/ROLE_NAME"
Logs
Output from eksctl during creation is normal.
Anything else we need to know?
What OS are you using? macos
Are you using a downloaded binary or did you compile eksctl? downloaded via asdf
What type of AWS credentials are you using (i.e. default/named profile, MFA)? SSO
Versions
❯ eksctl info
eksctl version: 0.183.0
kubectl version: v1.30.2
OS: darwin
Activity
github-actions commentedon Jun 24, 2024
Hello matschaffer-roblox 👋 Thank you for opening an issue in
eksctlproject. The team will review the issue and aim to respond within 1-5 business days. Meanwhile, please read about the Contribution and Code of Conduct guidelines here. You can find out more information abouteksctlon our websitematschaffer-roblox commentedon Jun 27, 2024
Removing the
/eks/path from the role seems to be a viable workaround (arn:aws:iam::ACCOUNT:role/ROLE_NAME)AWS support provided some steps for their reproduction of the issue:
Step 1 => I created a trust policy with the below mentioned content:
Step 2 => I created a role with path using the below mentioned command:
Step 3 => I created an EKS cluster and nodegroup with the below mentioned config file "eksctl create cluster -f test.yaml" :
Step 4 => The nodegroup that craeted shows IAM role as arn:aws:iam::55555555555:role/test-node-role" on the EKS console. The access entry that is created automatically has the complete path "/eks/" included but it is stripped from the node group. The CreateNodegroup API call and Cloudformation stack show below mentioned configuration for node role passed:
So, eksctl seems to be stripping the path from the node role which is eventually leading to health issues on the node with the error "access entry not found in cluster".
github-actions commentedon Jul 28, 2024
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.
matschaffer-roblox commentedon Jul 28, 2024
Bump for stalebot
github-actions commentedon Aug 28, 2024
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.
matschaffer-roblox commentedon Aug 28, 2024
Bump for stalebot