Repository navigation
Deadlock when calling PyGILState_Ensure() from a fresh C thread #96071
Description
Activity
Do you have a self-contained example that demonstrates the problem? I quickly checked using PyObjC on macOS (which basically does this when starting threads using the Cocoa interface), and that code did not deadlock.
Do you have a self-contained example that demonstrates the problem? I quickly checked using PyObjC on macOS (which basically does this when starting threads using the Cocoa interface), and that code did not deadlock.
Ok, I investigated a little deeper and admittedly I have a bit more going on in the environment where I found this. Tried duplicating until I found the difference:
tracemalloc. Whentracemallocis imported and started the function flow is as shown in the header of this issue. When it is not, then afterobmalloc:PyMem_RawCalloc()it goes toobmalloc:_PyMem_DebugRawCalloc()instead of_tracemalloc:_PyMem_Raw.calloc()which avoids the recursive call toPyGILState_Ensure()and works fine. I will see about getting some poc module code here tomorrow or after.I can reproduce using PyObjC by using
python3.11 -X tracemalloc script.pyto start the PyObjC using script I mentioned earlier.@pablogsal is this a release blocker?
@pablogsal is this a release blocker?
Indeed it is. Thanks for pinging me.
@ronaldoussoren Can you share the reproducer?
Or alternatively, could you bisect the issue?
Or alternatively, could you bisect the issue?
In previous versions the memory for the new
PyThreadStatewas allocated before theHEAD_LOCK()inpystate:new_threadstate(), if you revert to that behavior the bug is fixed.The reproducer needs PyObjC (from v8.5-branch in https://github.com/ronaldoussoren/pyobjc because the latest release does not work with Python 3.11 yet).
The reproducer script:
from Cocoa import NSObject, NSThread import time class Runner(NSObject): def doit_(self, arg): print("hello world") print(NSThread.currentThread().isMainThread()) runner = Runner.alloc().init() thread = NSThread.alloc().initWithTarget_selector_object_(runner, b"doit:", None) thread.start() print(thread) time.sleep(1) while thread.isExecuting(): time.sleep(1)This script hangs when using "python3.11 -X tracemalloc script.py
and works fine when using an older version of Python or leaving out-X tracemalloc``.It is probably easier to reproduce this using a small C extension that uses
pthread_createto launch a thread that callPyGILState_Ensure.I can look into this during the weekend at the earliest, but also need to get a release of PyObjC out to fix Python 3.11 support (which is needed due to documented API changes, nothing to worry about there).
I will prepare a patch tomorrow. I have enough information to produce a fix.
16 remaining items
Closing as I have fixed both the issue and added tests to ensure no further regression. Thanks!
- Repository owner moved this from In Progress to Done in Release and Deferred blockers 🚫
on Aug 24, 2022 Thanks, @kumaraditya303!
Reacted by Kumar Aditya- added a commit that references this issue
on Jul 25, 2023 - added a commit that references this issue
on Dec 2, 2024
Bug report
This used to work with py 3.8, 3.9 and 3.10 but now fails. The failure is due to a deadlock condition due to the move of an alloc from before a HEAD_LOCK() to after causing a recursive call to PyGILState_Ensure() which eventually deadlocks on the HEAD_LOCK().
The following is a trace of calls from the initial call to PyGILState_Ensure() to the eventual deadlock:
Steps to reproduce: In a Python C extension module, create a C thread then try to call PyGILState_Ensure() in that thread with no previous existing Python threadstate.
Your environment
Python 3.11.0rc1
Linux tom-VirtualBox 5.15.0-46-generic #49~20.04.1-Ubuntu SMP Thu Aug 4 19:15:44 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux