Repository navigation
registry-rest-api-tests CI workflow sometimes fails due to missing HTTP readiness check #5977
Description
Activity
Hello,
I’ve gone through this issue and I’m confident I can address the CI instability in the
registry-rest-api-testsworkflow.My background includes backend development and CI reliability work, particularly around HTTP service readiness, integration test timing, and race-condition failures. I can investigate the root cause and submit a focused PR to make the workflow deterministic.
Let me know if you’d like me to take this on.
Best regards,
Hi @mstfzhoseini-sketch, thanks for the interest! Just a heads up — there's already a PR addressing this issue: #5978.
Even after adding checks and timeout 180s, I see sometimes registry server won't get ready
Waiting for HTTP endpoint to become ready (timeout=180s): http://feast.kind.test/api/v1/projects HTTP endpoint returned 502, retrying in 5s... HTTP endpoint returned 502, retrying in 5s... HTTP endpoint returned 502, retrying in 5s... HTTP endpoint returned 502, retrying in 5s... HTTP endpoint returned 502, retrying in 5s... HTTP endpoint returned 502, retrying in 5s... HTTP endpoint returned 502, retrying in 5s... HTTP endpoint returned 502, retrying in 5s... HTTP endpoint returned 502, retrying in 5s... HTTP endpoint returned 502, retrying in 5s... HTTP endpoint returned 502, retrying in 5s... HTTP endpoint returned 502, retrying in 5s... Deleting namespace: test-ns-feast-rest ERRORhttps://github.com/feast-dev/feast/actions/workflows/registry-rest-api-tests.yml?query=is%3Afailure
@mstfzhoseini-sketch See if you can resolve this
Problem
The registry-rest-api-tests workflow (pr-rest-API-tests) is structurally flaky. When it fails, all 28 tests fail with the same error:
assert 502 == 200This happens across multiple branches (master, fix_complex_type, dependabot/*, etc.) and is not caused by any specific code change.
Suspected Root Cause
The test fixture checks two conditions before starting tests:
However, the HTTP endpoint behind the ingress may not be ready to serve traffic even when Pod/CR are Ready. Tests start immediately, hitting the ingress before the backend is fully initialized, resulting in 502 Bad Gateway responses.
Possible Solution
add an HTTP readiness poll to the test fixture in
sdk/python/tests/integration/registration/rest_api/test_registry_rest_api.py.After confirming Pod/CR readiness, poll the actual HTTP endpoint until it returns 200 before starting tests: