Repository navigation
intermittent ResourceTracker called reentrantly for resource cleanup in tests #131788
Description
Bug report
Bug description:
I occasionally see
UserWarning: ResourceTracker called reentrantly for resource cleanup, which is unsupported. The semaphore object '/mp-gtkzf9rd' might leak.warned in the test logs, I suspect test_force_shutdown_workers_dead_workers is the culprit.
It looks like the ResourceTracker was built to warn on re-entrant execution but I think it can be made to work instead.
CPython versions tested on:
CPython main branch
Operating systems tested on:
No response
Activity
gh-131788: make resource_tracker re-entrant safe (GH-131787)
pythongh-131788: make resource_tracker re-entrant safe (pythonGH-131787)
pythongh-131788: make resource_tracker re-entrant safe (pythonGH-131787)
[3.13] gh-131788: make resource_tracker re-entrant safe (GH-131787) (#…
pythongh-131788: make resource_tracker re-entrant safe (pythonGH-131787)
gpshead commentedon Aug 20, 2025
heads up that backports seem to have happened and the PR shipped in 3.13.7 and at least one person says it is causing unspecified as of yet issues cc: @ngoldbaum - #137738 (comment)
my own comment on the PR many moons ago when approving did question if this was safe enough to backport, but the automation was still in place so those understandably happened regardless (it has not landed in 3.14 yet due to rc phase). #131787 (comment)
ngoldbaum commentedon Aug 20, 2025
It looks like the backport of the fix for this issue is causing problems in the
lokyproject, which is used byjoblib(in turn used by lots of stuff, includingscikit-learn).joblib/loky#459
joblib/joblib#1738
geofft commentedon Aug 21, 2025
I think we have another couple of reports of this regression over at astral-sh/python-build-standalone#774. Specifically note that all the reports are that this is failing on Windows with an ImportError for
_posixsubprocess, which is imported bymultiprocessing.util.spawnv_passfds(), the call to which moved around in this PR.Did something on this PR cause code to run on Windows that was previously getting skipped (or ImportError to be swallowed)?
zanieb commentedon Aug 21, 2025
@geofft fyi I opened #138031 for that specific error since @ngoldbaum's report also includes
ValueError: Cannot register X for automatic cleanup: unknown resource type Y.Revert "[3.13] pythongh-131788: make resource_tracker re-entrant safe (…
ngoldbaum commentedon Oct 16, 2025
It looks like this got reverted in 3.13 - is there any appetite to also revert it for 3.14? It looks like the mypy tests also hit this under 3.14t.
EDIT: never mind, I'm seeing the errors that were fixed by this PR. So what I'd need is for this PR to be applied to the 3.14 branch, which would probably then break uv.
ngoldbaum commentedon Oct 16, 2025
here's a failing mypy CI run using 3.14.0t: https://github.com/ngoldbaum/mypy/actions/runs/18566460316/job/52928759605?pr=1
zanieb commentedon Oct 16, 2025
I posted a revert at #138222 but it never merged, instead loky fixed the issue on their end and that was considered sufficient for closing #138031.
#137737 was put on hold due to the loky regression, but I think we can merge that now — the outstanding questions have been resolved (cc @gpshead).
This won't break uv, to my knowledge. We just got reports of the loky issue when people upgraded to the latest version of 3.13.
ngoldbaum commentedon Oct 16, 2025
Thanks for explaining, so I guess the fix is to merge #137737.
ngoldbaum commentedon Oct 16, 2025
ping @hugovk as 3.14 release manager
gpshead commentedon Oct 17, 2025
I synced the 3.14 PR and kicked off extra tests. re-ping me on that tomorrow it hasn't been merged already.
[3.14] gh-131788: make resource_tracker re-entrant safe (GH-131787) (#…