Repository navigation
Full unit test suite isn't running on CI #1439
Description
If you take a look at the unit tests running in CI, it appears that they aren't all run.
See this run as an example:
Firstly, is this intentional?
If it's not, perhaps we could update the make test command to provide the all tag that seems to be missing?
The other issue is that when it run all the tests locally, a few of them fail. I haven't dived into this yet to see if it's a legitimate failure or an issue with my environment.
Anyone else seeing this?
Community Note
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Terraform (and Azure DevOps Provider) Version
N/A
Affected Resource(s)
Unit tests are not being run - an example is: data_projects_test.go
Terraform Configuration Files
N/A
Debug Output
See CI test run here:
Panic Output
N/A
Expected Behavior
All tests apart from acceptance should run.
Actual Behavior
A subset of all tests are run.
Steps to Reproduce
- Clone and create a development environment
- Run
make test
Or run CI on the latest main.
Important Factoids
N/A
References
N/A
Activity
magodo commentedon Sep 30, 2025
Andrew Cardy (@andrwca) Thanks for raising this question! The test cases are split to unit test and acceptance test (e2e test). In the CI, we only cover the unit tests. The acceptance tests are only meant to run separately, on demand. Unfortunately, as you may have noticed, the quality of the acctests are not well maintained (actually for a while), therefore you won't expect all the tests are bug free.. The bottom line is that, once you changed a resource, you'll be responsible to pass the acctests of the impacted resource(s).
andrwca commentedon Sep 30, 2025
magodo (@magodo) thanks for the extra detail.
I realise my original report wasn't too clear, sorry!.
Essentially what I'm seeing is that
make testis running a subset of the unit tests. This is aside from acceptance tests.To test this, I changed an assert in one of the project unit tests* to fail, ran
make test, but all the tests still succeeded. Upon reviewing the output, the test didn't run. I manually ran the tests again using the commandgo test -v -tags="all" ./...and I could see that the project test was run.Can you reproduce this?
* /workspaces/terraform-provider-azuredevops/azuredevops/internal/service/core/data_projects_test.go
magodo commentedon Sep 30, 2025
Hi Andrew Cardy (@andrwca), thanks for the additional information! I overlooked these test cases when I remove those build tags in #1436. It takes time to fix all these broken UTs, and then it should be good to remove the build tags from these files as well.
andrwca commentedon Oct 2, 2025
magodo (@magodo), ah - I didn't see that PR - that explains explain a lot. :)
saryn17 commentedon Sep 16, 2026
I'd like to take this one, unless Andrew Cardy (@andrwca) is already on it.
I have the two phases from your comment working locally. For the record on the original
question:
make testcompiles 35 of the 536 unit tests in the repo. After repairing thebroken ones and dropping the tags, 534 pass (two turned out to be exact duplicates).
Two things I'd rather flag here than have you find in the diff:
azuredevops_build_definitionhas been sending"triggers": nullinstead of"triggers": []since Move the PR CI from ADO to github action #1396, and the round-trip UT that asserts the old payload is oneof the tests that never runs. I put the old behaviour back, but I can't verify against a
real org, so say the word if you'd rather keep
nulland change the fixture instead.320s of
time.SleepandStateChangeConf.Delay. I turned those intervals into packagevariables that
TestMainzeroes, production values untouched. If you don't want thatseam in the resource code I'll find another way.
PR shortly. Could you assign the issue to me?
andrwca commentedon Sep 20, 2026
Sounds good, I'm not working on it.