os.fork() child can hang inside PyOS_AfterFork_Child() on the warnings lock (3.14 regression) #158973

Bug report

Since 3.14, the warnings module protects its state with a lock in the interpreter, added in #128386 for #128384. PyOS_AfterFork_Child() does not reset this lock. If another thread holds it when a thread calls os.fork(), the child inherits it locked, and the first warning in the child blocks forever. The child does not need to run any code for this to happen. PyOS_AfterFork_Child() clears the states of the other threads. Objects held only in those states, such as threading.local() data, are freed there, and a finalizer can issue a warning before os.fork() returns.

In the reproducer, one thread keeps an unclosed socket in threading.local() and issues warnings in a loop. The child only calls os._exit(0), so a child that hangs is stuck inside os.fork():

import os, socket, sys, threading, time, warnings

# Every warning is shown, so warn() runs showwarning(), which is Python code,
# while it holds the warnings lock.
warnings.simplefilter("always")
warnings.showwarning = lambda *args, **kwargs: None

local = threading.local()

def warn_forever():
    local.sock = socket.socket()  # never closed: freeing it issues a ResourceWarning
    while True:
        warnings.warn("from another thread")

threading.Thread(target=warn_forever, daemon=True).start()
time.sleep(0.5)

hung = 0
for _ in range(20):
    pid = os.fork()
    if pid == 0:
        os._exit(0)  # the child runs no other code
    time.sleep(0.5)
    if os.waitpid(pid, os.WNOHANG) == (0, 0):
        hung += 1
        os.kill(pid, 9)
        os.waitpid(pid, 0)

print(f"{sys.version.split()[0]}: {hung} of 20 children hung", flush=True)
os._exit(0)  # skip interpreter shutdown, which can hang too (see below)

Results on Linux x86_64 with glibc 2.43:

gdb on a hung child (3.14.4, Ubuntu build without debug symbols; frames without a name are cut) shows a finalizer that runs in PyThreadState_Clear() and waits for the lock in PyErr_ResourceWarning():

#10 0x00000000005cbfc6 in _PyRecursiveMutex_Lock ()
...
#14 0x000000000047da7c in PyErr_ResourceWarning ()
...
#16 0x00000000005cd2d8 in PyObject_CallFinalizerFromDealloc ()
...
#23 0x00000000005d9bf6 in PyObject_ClearWeakRefs ()
...
#26 0x00000000006d0f22 in PyThreadState_Clear ()
#27 0x00000000004850c2 in PyOS_AfterFork_Child ()

The lock can be held while other threads run. do_warn() holds it across warn_explicit(), and warn_explicit() runs the Python function showwarning() that writes the message:

catch_warnings.__enter__() and __exit__(), filterwarnings() and simplefilter() also run Python code while they hold the lock. Every C warning call takes the lock before it checks the filters, so the ResourceWarning in the child blocks although the default filters ignore it.

The lock is not in LOCKS_INIT, the list of runtime locks that the child resets. On main, PyOS_AfterFork_Child() also resets _Py_jit_debug_mutex. It then clears the dead thread states, and the comment above that call notes that this may call destructors:

Holding the lock across the fork avoids the hang. With these two lines added to the reproducer, 0 of 20 children hang on 3.14.4, 3.14.8 and 3.15.0rc3:

import _warnings
os.register_at_fork(before=_warnings._acquire_lock, after_in_parent=_warnings._release_lock, after_in_child=_warnings._release_lock)

A fix could take the lock in PyOS_BeforeFork() and release it after the fork, as is done for the import lock, or reset it in PyOS_AfterFork_Child(), as is done for _Py_jit_debug_mutex.

This hangs JupyterLab terminals. terminado starts them with ptyprocess on the server's event loop thread, and ptyprocess waits for exec() with a blocking read after pty.fork(). When the child hangs inside os.forkpty(), the whole Jupyter server stops responding. IPython runs ! commands through pexpect, and pexpect uses the same code.

I know this is grey territory. The os.fork() documentation says that "it has never been safe to mix threading with os.fork() on POSIX platforms", and earlier reports of hangs in a fork child were closed on that basis (#111635, #112788). Here the child runs no code of its own, and it waits for a lock that CPython added in 3.14. Two earlier issues about a lock or state that the child inherits were fixed:

• deadlocked child process after forking on pystate.c's HEAD_LOCK #112275: the child inherited HEAD_LOCK locked and deadlocked on it in PyOS_AfterFork_Child(). The fix was backported to 3.8.
• Multiprocessing with "fork" fails with Python 3.14 but works with Python 3.13 #139894: a 3.14 regression in the asyncio state that the child inherits. It was fixed in 3.14 by adding reset_asyncio_state() to PyOS_AfterFork_Child().

The same lock can also hang interpreter shutdown, with no fork. At shutdown, the daemon thread is frozen while it holds the lock, and the ResourceWarning for its socket then blocks the main thread. The reproducer without the fork loop and the final os._exit(0) hangs at exit in 10 of 10 runs on 3.14.4, 8 of 10 on 3.14.8 and 9 of 10 on 3.15.0rc3. It hangs in 0 of 10 runs on 3.13.16, and on 3.14.4 without the socket. I can open a separate issue for it.

I used Opus 5.5 for assistance with investigation and the writeup.

CPython versions tested on:

3.12, 3.13, 3.14, 3.15

Operating systems tested on:

Linux

Linked PRs

• gh-158973: Reset the warnings lock after fork in the child #158974