Visitar URL original
ci(python): run the integration tests on pytest-xdist workers by rusackas · Pull Request #45756 · apache/superset · GitHub
Skip to content

ci(python): run the integration tests on pytest-xdist workers - #45756

Draft
rusackas wants to merge 4 commits into
masterfrom
ci/integration-tests-xdist
Draft

rusackas wants to merge 4 commits into
masterfrom
ci/integration-tests-xdist

Conversation

@rusackas

Copy link
Copy Markdown
Member

SUMMARY

The Python integration suite runs serially: scripts/python_tests.sh calls plain pytest, which takes 20 to 28 minutes per database leg (MySQL is the slowest) on a 4 vCPU runner that is mostly idle. Unlike sharding across more runners, which only moves the same CPU work around and adds a setup cost per shard, running the suite on pytest-xdist workers inside the one job should cut the runner minutes themselves, not just the wall clock.

This is a draft spike to find out what the suite needs to run in parallel:

  • PYTEST_WORKERS (the three database jobs set it to 4) makes scripts/python_tests.sh run pytest -n <N> --dist loadfile, so each test file stays on one worker and module-scoped fixtures keep working.
  • Each xdist worker gets its own metadata database, examples database and Redis database range. The test config derives them from PYTEST_XDIST_WORKER (superset becomes superset_gw0, superset.db becomes superset-gw0.db, Redis dbs 2/3/4 shift by three per worker), so nothing in the test code changes.
  • The script migrates and seeds one database per worker (in parallel) before pytest starts; setup-postgres and setup-mysql in bashlib.sh create the per-worker databases, and celery-worker starts one Celery worker per xdist worker on that worker's broker database.
  • A plain pytest run, or PYTEST_WORKERS=1, keeps today's behaviour exactly.

What I expect this run to surface: tests that depend on state another test file left behind, and MySQL tests that touch the shared sqllab_test_db and admin_database databases (on Postgres those are schemas inside each per-worker database, on MySQL they are still shared). I'll fix or isolate whatever shows up and post the runner-minute numbers against the serial runs before taking this out of draft.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

N/A

TESTING INSTRUCTIONS

CI: the three database legs of the Python-Integration workflow run with four workers. Compare their job durations with the serial runs on master.

ADDITIONAL INFORMATION

  • Has associated issue:
  • Required feature flags:
  • Changes UI
  • Includes DB Migration (follow approval process in SIP-59)
    • Migration is atomic, supports rollback & is backwards-compatible
    • Confirm DB migration upgrade and downgrade tested
    • Runtime estimates and downtime expectations provided
  • Introduces new feature or API
  • Removes existing feature or API

🤖 Generated with Claude Code

@github-actions github-actions Bot added github_actions Pull requests that update GitHub Actions code risk:ci-script PR modifies scripts that execute in CI (supply chain risk) preset-io labels Oct 27, 2026
@codecov

codecov Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.76%. Comparing base (2649e3b) to head (3e228a4).

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #45756      +/-   ##
==========================================
- Coverage   85.24%   82.76%   -2.49%     
==========================================
  Files        3086     3086              
  Lines      211717   211769      +52     
  Branches    49920    49934      +14     
==========================================
- Hits       180475   175263    -5212     
- Misses      27834    32905    +5071     
- Partials     3408     3601     +193     
Flag Coverage Δ
hive 34.65% <ø> (+0.60%) ⬆️
javascript 81.62% <ø> (+0.02%) ⬆️
mysql ?
postgres ?
presto 35.69% <ø> (+<0.01%) ⬆️
python 83.67% <ø> (-4.46%) ⬇️
sqlite ?
unit 83.27% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@netlify

netlify Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for superset-docs-preview ready!

Name Link
🔨 Latest commit 3e228a4
🔍 Latest deploy log https://app.netlify.com/projects/superset-docs-preview/deploys/6af6b7b2aac6f5df22c9f71a
😎 Deploy Preview https://deploy-preview-45756--superset-docs-preview.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

rusackas and others added 4 commits November 12, 2026 23:09
The integration suite runs serially in scripts/python_tests.sh, 20 to 28
minutes per database leg on a 4-vCPU runner that is mostly idle.

With PYTEST_WORKERS > 1 (the three database jobs set 4) the script runs
pytest -n with --dist loadfile. Every xdist worker gets its own metadata
database, examples database and Redis database range, derived in the test
config from PYTEST_XDIST_WORKER; the script migrates and seeds one database
per worker before pytest starts, bashlib's setup-postgres/setup-mysql
create the per-worker databases, and celery-worker starts one Celery worker
per xdist worker on that worker's broker. A plain pytest run or
PYTEST_WORKERS=1 keeps today's behaviour.

Co-Authored-By: Evan Rusackas <evan@preset.io>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…names table)

Two leftovers of the serial run that pytest-xdist workers do not inherit:

- PUBLIC_ROLE_LIKE: `superset init` gave Public the Gamma permissions, but
  the first public_role_like_* fixture teardown cleared them and most of the
  suite asserts anonymous requests are rejected on that cleared role. The
  test config now starts from a cleared Public role; tests that need Gamma
  use the public_role_like_gamma fixtures.
- Table id 1: the first load_birth_names_dashboard_with_slices use created
  the birth_names SqlaTable and left the row behind, which the many
  insert_chart(..., 1) callers and the "[examples].[birth_names](id:1)"
  assertions rely on. An autouse session fixture now does that once up
  front.

test_get_catalog_names tolerates the per-worker superset_gw<N> databases.

Co-Authored-By: Evan Rusackas <evan@preset.io>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… alone

Under pytest-xdist each worker runs only part of the suite, so state that
one file used to leave behind for a later one is no longer guaranteed to
be there. Fix the tests that depended on it:

- sql_lab/test_execute_sql_statements: superset.sql_lab.results_backend is
  a werkzeug LocalProxy, which defines __await__, so patch() built an
  AsyncMock whose set() returned a truthy coroutine. In a serial run
  sql_lab/api_tests.py had already swapped the proxy for a plain Mock, which
  hid this. Pin new_callable=MagicMock.
- core_tests permalink tests: the anonymous-access assertions need the
  Public role to reach the view, which the serial run got from the first
  public_role fixture leaking Gamma permissions. Use
  public_role_like_gamma_scoped explicitly.
- fixtures/users.create_role_with_permissions: add_role() returns an
  existing role untouched and the fixtures never delete their roles, so the
  "dar" role created by databases/api_tests.py with only
  all_database_access shadowed the datasource_access one security_tests
  expected. Make the role carry the requested permissions.
- reports/api_tests native-filter tests: they query the first Dashboard
  without creating one; load the birth_names dashboard.
- saved_queries related-databases test: compare by database id instead of
  relying on physical row order, which PostgreSQL does not preserve across
  updates.

Co-Authored-By: Evan Rusackas <evan@preset.io>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…migrated

The semantic-cache coordination job runs integration tests against a
bare SQLite file, so the session autouse fixture failed at setup with
'no such table: dbs'.

Co-Authored-By: Evan Rusackas <evan@preset.io>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@rusackas
rusackas force-pushed the ci/integration-tests-xdist branch from e7b3433 to 3e228a4 Compare November 13, 2026 07:09

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

github_actions Pull requests that update GitHub Actions code preset-io review:draft risk:ci-script PR modifies scripts that execute in CI (supply chain risk) size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants