Repository navigation
Silent CI failures due to enforced action restrictions with no prior notice #574
Description
Activity
@clambertus thats the trivy incident. I see github actions like
pypa/cibuildwheelandastral-sh/setup-uvsuddenly failing.Is it due to this? I didnt understand what "verified creator" means
ASF Infra and ASF Security agreed to disable all previously allowed "verified creator" actions while the incident is being investigated
Reacted by Drewanother example, apache/iceberg#15742 (comment)
Known issue with GitHub Actions - lack of jobs causes "green" PR by definition. But adding "context" in protected_branch definition (required status check for the PR to be mergable) solves the issue.
Reacted by Kevin LiuI thought a bit - I think we should add some incentive for people who use pinning properly and prevent those sillent errors to happen. That woudl be a carrot instead of stick:
- Currently when we do Dependabot PR reviews - we need to do that manually anywayth
- I created a small CLI tool that makes it super easy Add action build verification script #561
- Similar tool could probably easily use gh APIs tot go through apache repositories using actions and see if the action is used in an of the repositories and make a notification after PRs are merged with new versions, notifying all the repos that use those - even creating issues in their repos that they should upgrade those particular actions
- Similarly a week or few days before scheduler removal, we could open the project's that still have those versions used with notification that the WF will stop working (inclding which workflows) - another issue could be created.
Then it would be enitrely up to the PMC to react to those.
With Claude /AI - it can be entirely Clauded in a rather short time (hours) and maintenance of it will be near 0 (you can always ask Claude to fix any errors you experience in minutes). I would do it - but I am not sure if such contributions are really useful - the one for actions review I did in 2 hours and reviewed/closed > 10 prs - but it might take much longer for @raboof @dfoulks1 and others who review the actions to accept it a nd start using it
This is what we learn from current AI driven development - that new tools like that can be created very quickly by those who started using them - and quickly, but it still takes stakeholders a lot of time to accept that such tooling can now be created in almost no time.
My friend @kaxil wrote about it in this post - https://kaxil.substack.com/p/i-havent-written-a-line-of-code-in - and yes- what I see is exactly that - people need to get used to the fact that you can create such tools 10x faster now.
So I do not want to go ahead of those decisions even more - But - at some point in time I am happy to contribute it here if there is an appetite for it and willingness to accept such generated code.
Reacted by Kaxil Naik and Kevin LiuKnown issue with GitHub Actions - lack of jobs causes "green" PR by definition. But adding "context" in protected_branch definition (required status check for the PR to be mergable) solves the issue.
Thanks for the hint here @potiuk . I took a look at this solution. It involves very carefully specifying the exact name of each CI that should be "green" (which honestly should be all of them!). There are a lot of caveats and gotchas according to Github and ASF.
I found another potential solution/workaround. I added a runtime check in CI, comparing all the github actions against the
approved_patterns.ymlfile. Heres the code: apache/iceberg@f6c8e58
This gives the failure signal we're looking for in the CI (and PR) (https://github.com/apache/iceberg/actions/runs/23525855725/job/68479029725?pr=15749). The "Checks" tab still shows a small black status for the silent failure, but this check will loudly tell everyone what the problem is.Might be a good idea to turn this into an ASF github action that all projects can pull. WDYT?
EDIT: heres a first draft of what the github action would look like, #587
thanks @potiuk could you give it a review when you get a chance?
Known issue with GitHub Actions - lack of jobs causes "green" PR by definition.
Indeed. While it's indeed well-known, do you know if this is actually tracked somewhere?
But adding "context" in protected_branch definition (required status check for the PR to be mergable) solves the issue.
Right, but many projects will not do this (either because of ignorance or because of a legitimate desire to have some jobs that are ran but not block the merge. Also that doesn't help for workflows that are triggered some other way, such as scheduled ones.
I like #587, but that's only for projects that are already aware. I wonder if we should also have a way to 'proactively' notify projects. One way might be to include this check in https://github.com/apache/infrastructure-gha-workflow-scanner - though we'd want to make sure it doesn't get too noisy. Also perhaps we could create some kind of report somewhere?
Indeed. While it's indeed well-known, do you know if this is actually tracked somewhere?
I tracked it for about 2 years in a personal ticket, but It seems to be an architectural issue and will not be fixed and I closed it.
I am not sure if anything else can be done on GH side to fix it. So this issue is basically tracking it on our side and maybe we will find some solutions to that.Right, but many projects will not do this (either because of ignorance or because of a legitimate desire to have some jobs that are ran but not block the merge. Also that doesn't help for workflows that are triggered some other way, such as scheduled ones.
Correct - that might not be a viable path for many. Unfortunately - I do not have any other idea how to deal with it other than halping to develop tools (or contribute to them) that might be ok for infra to use long term. As volunteers we migh help to create solutions - but regularly elmaintaining them when we have vacations, sudden spikes of other work is quite infeasible. I've learned that proposing things without earlier buy-in from infra and then trying to push it to them is a bad idea (I anticipated htat in my comment above as well. But if there is an infra buy-in and agreement on the kind of notification schema that we should do, I am happy to help to add such features.
I like #587, but that's only for projects that are already aware. I wonder if we should also have a way to 'proactively' notify projects. One way might be to include this check in https://github.com/apache/infrastructure-gha-workflow-scanner - though we'd want to make sure it doesn't get too noisy. Also perhaps we could create some kind of report somewhere?
That's basically what I proposed in #574 (comment) - (if it can be integrated in the scanner, then that would be great).
But I believe the action proposed by @kevinjqliu is way better solution.
The https://github.com/apache/infrastructure-gha-workflow-scanner might be used to check for PRESENCE of this action - not for action hashes. This would be way, way, way better - because you would just notify about lack of this action - and then once added, the action would automatically notify the maintainers about any issues with their action hashes. I think it's way more noisy, way less invasive and also it opens up the nice capability that we could check for MORE things - once we make everyone in ASF using GH actions to add this action.
I think this is a brilliant way of solving the noise problem. Eventially the https://github.com/apache/infrastructure-gha-workflow-scanner could ONLY check for presence of the action and be even very annoying about it - making it essentially a requirement for all ASF projects. And then the action called from every repo could do all kinds of check we want - for example zizmor check for security etc. This is I think fantastic way for infra (if they would want to be active in this area) force whatever checks they do and they will run not out-of-bound - but whenever PRs or "main" builds happen in the ASF repos - making them actually owners to solve their failing builds, rather than responding to email notifications.
Of course that also bears certain responsibility on whoever is the owner of such action - to follow up and resolve any problems it might cause - answer questions of the PMCs and so on, but this is a very elegant problem to solve INFRA "basic checks" issue that @killerbees mentioned in slack today. Such action might very cleary define what the INFRA enforces on every repo and make it part of INFRA very concrete, automatically enforcable set of expectations they have.
This is resolved by adding the
apache/infrastructure-actions/allowlist-checkaction project's CI.
For example, in the apache/iceberg repo: https://github.com/apache/iceberg/blob/dde712ec9ed6c9d28183ee4615d50f97b246af5d/.github/workflows/asf-allowlist-check.ymlI tihnk the final step here is to add this to asf infra documentation somewhere as best practice recommendation
- added a commit that references this issue
on Aug 4, 2026
Continuation of #550 (closed; unable to reopen)
This issue focuses specifically on verified actions silently failing in GitHub workflows following the recently enforced policies.
Problem
Several GitHub workflows are now failing silently as a result of the new action restrictions. For example: https://github.com/apache/iceberg-python/actions/runs/23416549033
These failures are marked with "Status: Startup failure" and the following error:
While it's great to see efforts to improve ASF's security posture, it would be really helpful to have some advance notice before changes like this are enforced, so projects aren't caught off guard.
Impact
Solution
EDIT: Added a new github action (
apache/infrastructure-actions/allowlist-check@main) to address this issue, #587