Repository navigation
Python/pystate.c:2218: _PyThreadState_PopFrame: Assertion `tstate->datastack_top >= base' failed. #93252
Description
Activity
Spawning and killing lots of threads does suggest a race condition.
Do you have the values of
datastack_topanddatastack_limit, not what they point to?
Could you also get the contents of the stack chunk when the assert fails?
p *tstate->datastack_chunk
I don't need the data in it, just theprevious,sizeandtopfields.The question I want to answer is this: Has one of the internal pointers exceeded its bounds, or is it pointing to the wrong piece of memory?
Thanks
What's strange is that this always seems to happen at around the same time. I'm not entirely sure it's always the same number of repetitions, but it certainly always seems to be 50 ± 5 so far (spawning 3 threads each, from what I can see in gdb). Might well always be 47 times, it's somewhat hard to count since the process sending the commands runs independently.
#6 0x0000555555800261 in _PyThreadState_PopFrame (tstate=tstate@entry=0x555555b2deb8 <_PyRuntime+166136>, frame=frame@entry=0x7ffff7821f70) at Python/pystate.c:2218 2218 assert(tstate->datastack_top >= base); (gdb) p tstate->datastack_top $1 = (PyObject **) 0x7fffe4c06090 (gdb) p tstate->datastack_limit $2 = (PyObject **) 0x7fffe4c0a000 (gdb) p *tstate->datastack_chunk $3 = {previous = 0x7ffff781e000, size = 16384, top = 0, data = {<function at remote 0x7fffe4f58e10>}}Also,
info threadsif that's of any value:Details
(gdb) info threads Id Target Id Frame * 1 Thread 0x7ffff7c85740 (LWP 3097731) "python3" __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44 2 Thread 0x7fffe45ff640 (LWP 3097779) "QXcbEventQueue" 0x00007ffff7d8dfaf in __GI___poll (fds=0x7fffe45feca8, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29 3 Thread 0x7fffe1a0b640 (LWP 3097780) "python3:cs0" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x555556718910) at futex-internal.c:57 4 Thread 0x7fffe120a640 (LWP 3097781) "python3:disk$0" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x555556711058) at futex-internal.c:57 5 Thread 0x7fffe0a09640 (LWP 3097782) "python3:sh0" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5555567219ac) at futex-internal.c:57 6 Thread 0x7fffd3a20640 (LWP 3097783) "python3:sh1" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5555567219a8) at futex-internal.c:57 7 Thread 0x7fffd321f640 (LWP 3097784) "python3:sh2" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5555567219ac) at futex-internal.c:57 8 Thread 0x7fffd2a1e640 (LWP 3097785) "python3:sh3" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5555567219a8) at futex-internal.c:57 9 Thread 0x7fffd221d640 (LWP 3097786) "python3:sh4" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5555567219a8) at futex-internal.c:57 10 Thread 0x7fffd1a1c640 (LWP 3097787) "python3:sh5" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5555567219a8) at futex-internal.c:57 11 Thread 0x7fffd121b640 (LWP 3097788) "python3:sh6" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5555567219a8) at futex-internal.c:57 12 Thread 0x7fffd0a1a640 (LWP 3097789) "python3:sh7" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5555567219a8) at futex-internal.c:57 13 Thread 0x7fffabfff640 (LWP 3097790) "python3:sh8" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5555567219a8) at futex-internal.c:57 14 Thread 0x7fffab7fe640 (LWP 3097791) "python3:sh9" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5555567219a8) at futex-internal.c:57 15 Thread 0x7fffaaffd640 (LWP 3097792) "python3:sh10" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5555567219a8) at futex-internal.c:57 16 Thread 0x7fffaa7fc640 (LWP 3097793) "python3:sh11" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5555567219ac) at futex-internal.c:57 17 Thread 0x7fffa9ffb640 (LWP 3097794) "python3:shlo0" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x555556721f18) at futex-internal.c:57 18 Thread 0x7fffa97fa640 (LWP 3097795) "python3:shlo1" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x555556721f18) at futex-internal.c:57 19 Thread 0x7fffa8ff9640 (LWP 3097796) "python3:shlo2" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x555556721f18) at futex-internal.c:57 20 Thread 0x7fff8bfff640 (LWP 3097797) "python3:shlo3" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x555556721f18) at futex-internal.c:57 21 Thread 0x7fff8b7fe640 (LWP 3097798) "python3:shlo4" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x555556721f18) at futex-internal.c:57 22 Thread 0x7fff8affd640 (LWP 3097799) "python3:gdrv0" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5555567e6220) at futex-internal.c:57 23 Thread 0x7fff7bfff640 (LWP 3097800) "Thread (pooled)" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fff7bffed00, op=137, expected=0, futex_word=0x555556891080) at futex-internal.c:57 24 Thread 0x7fff7b7fe640 (LWP 3097801) "Thread (pooled)" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fff7b7fdd00, op=137, expected=0, futex_word=0x555556891494) at futex-internal.c:57 25 Thread 0x7fff7affd640 (LWP 3097802) "Thread (pooled)" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fff7affcd00, op=137, expected=0, futex_word=0x555556890e54) at futex-internal.c:57 26 Thread 0x7fff7a7fc640 (LWP 3097803) "Thread (pooled)" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fff7a7fbd00, op=137, expected=0, futex_word=0x55555689f3b4) at futex-internal.c:57 27 Thread 0x7fff79ffb640 (LWP 3097804) "Thread (pooled)" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fff79ffad00, op=137, expected=0, futex_word=0x5555568a0414) at futex-internal.c:57 28 Thread 0x7fff797fa640 (LWP 3097805) "Thread (pooled)" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fff797f9d00, op=137, expected=0, futex_word=0x5555568960b4) at futex-internal.c:57 29 Thread 0x7fff78ff9640 (LWP 3097806) "Thread (pooled)" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fff78ff8d00, op=137, expected=0, futex_word=0x5555568964c4) at futex-internal.c:57 30 Thread 0x7fff5ffff640 (LWP 3097807) "Thread (pooled)" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fff5fffed00, op=137, expected=0, futex_word=0x555556896980) at futex-internal.c:57 31 Thread 0x7fff5f7fe640 (LWP 3097808) "Thread (pooled)" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fff5f7fdd00, op=137, expected=0, futex_word=0x555556896e60) at futex-internal.c:57 32 Thread 0x7fff5effd640 (LWP 3097809) "Thread (pooled)" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fff5effcd00, op=137, expected=0, futex_word=0x555556897340) at futex-internal.c:57 33 Thread 0x7fff5e7fc640 (LWP 3097810) "Thread (pooled)" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fff5e7fbd00, op=137, expected=0, futex_word=0x555556897824) at futex-internal.c:57 34 Thread 0x7fff5dffb640 (LWP 3097811) "Thread (pooled)" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fff5dffad00, op=137, expected=0, futex_word=0x55555689be80) at futex-internal.c:57 35 Thread 0x7fff5d7fa640 (LWP 3097812) "Thread (pooled)" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fff5d7f9d00, op=137, expected=0, futex_word=0x55555689c364) at futex-internal.c:57 36 Thread 0x7fff5cff9640 (LWP 3097813) "Thread (pooled)" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fff5cff8d00, op=137, expected=0, futex_word=0x55555689c840) at futex-internal.c:57 37 Thread 0x7fff3bfff640 (LWP 3097814) "Thread (pooled)" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fff3bffed00, op=137, expected=0, futex_word=0x55555689cd24) at futex-internal.c:57 38 Thread 0x7fff3b7fe640 (LWP 3097815) "Thread (pooled)" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fff3b7fdd00, op=137, expected=0, futex_word=0x55555689d204) at futex-internal.c:57 40 Thread 0x7fff3affd640 (LWP 3097817) "sandbox_ipc_thr" 0x00007ffff7d8dfaf in __GI___poll (fds=0x7fff3affcc40, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29 41 Thread 0x7fff39df7640 (LWP 3097823) "python3" 0x00007ffff7d64f9f in __GI___wait4 (pid=3097820, stat_loc=0x7fff39df6d9c, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30 42 Thread 0x7fff395f6640 (LWP 3097824) "ThreadPoolServi" 0x00007ffff7d99f3e in epoll_wait (epfd=46, events=0x7fff1c0024c0, maxevents=32, timeout=39322) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30 43 Thread 0x7fff38df5640 (LWP 3097825) "ThreadPoolForeg" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fff38df4b20, op=137, expected=0, futex_word=0x7fff38df4cb8) at futex-internal.c:57 44 Thread 0x7fff1bfff640 (LWP 3097826) "Chrome_IOThread" 0x00007ffff7d99f3e in epoll_wait (epfd=52, events=0x7fff14002480, maxevents=32, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30 45 Thread 0x7fff1b0f5640 (LWP 3097827) "ThreadPoolForeg" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fff1b0f4b20, op=137, expected=0, futex_word=0x7fff1b0f4cb8) at futex-internal.c:57 46 Thread 0x7fff1a8f4640 (LWP 3097828) "ThreadPoolForeg" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fff1a8f3b20, op=137, expected=0, futex_word=0x7fff1a8f3cb8) at futex-internal.c:57 47 Thread 0x7fff1a0f3640 (LWP 3097829) "ThreadPoolForeg" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fff1a0f2b20, op=137, expected=0, futex_word=0x7fff1a0f2cb8) at futex-internal.c:57 48 Thread 0x7fff198f2640 (LWP 3097830) "inotify_reader" 0x00007ffff7d907ec in __GI___select (nfds=58, readfds=0x7fff198f1d70, writefds=0x0, exceptfds=0x0, timeout=0x0) at ../sysdeps/unix/sysv/linux/select.c:69 49 Thread 0x7fff190f1640 (LWP 3097831) "ThreadPoolForeg" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fff190f0b20, op=137, expected=0, futex_word=0x7fff190f0cb8) at futex-internal.c:57 50 Thread 0x7fff188f0640 (LWP 3097832) "CompositorTileW" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55555697aa98) at futex-internal.c:57 51 Thread 0x7fff13fff640 (LWP 3097833) "Chrome_InProcGp" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fff13ffea20, op=137, expected=0, futex_word=0x7fff13ffebb8) at futex-internal.c:57 52 Thread 0x7fff137fe640 (LWP 3097834) "Chrome_ChildIOT" 0x00007ffff7d99f3e in epoll_wait (epfd=62, events=0x7ffef00024c0, maxevents=32, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30 53 Thread 0x7fff12ffd640 (LWP 3097835) "VideoCaptureThr" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7fff12ffcb98) at futex-internal.c:57 56 Thread 0x7fff127fc640 (LWP 3097838) "VizCompositorTh" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fff127fba20, op=137, expected=0, futex_word=0x7fff127fbbb8) at futex-internal.c:57 57 Thread 0x7fff11ffb640 (LWP 3097839) "python3:gdrv0" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffef8051680) at futex-internal.c:57 58 Thread 0x7fff117fa640 (LWP 3097840) "Qt bearer threa" 0x00007ffff7d8dfaf in __GI___poll (fds=0x7ffedc0052d0, nfds=1, timeout=9991) at ../sysdeps/unix/sysv/linux/poll.c:29 59 Thread 0x7fff10ff9640 (LWP 3097841) "QDBusConnection" 0x00007ffff7d8dfaf in __GI___poll (fds=0x7ffee004c000, nfds=4, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29 60 Thread 0x7ffedbbff640 (LWP 3097846) "ThreadPoolSingl" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffedbbfec98) at futex-internal.c:57 61 Thread 0x7ffedb3fe640 (LWP 3097848) "CacheThread_Blo" 0x00007ffff7d99f3e in epoll_wait (epfd=82, events=0x7ffecc0024c0, maxevents=32, timeout=30000) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30 63 Thread 0x7ffeda3fc640 (LWP 3097863) "python3:gdrv0" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55555772edd0) at futex-internal.c:57 64 Thread 0x7ffed9bfb640 (LWP 3097867) "MemoryInfra" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffed9bfab98) at futex-internal.c:57 65 Thread 0x7ffed93fa640 (LWP 3097873) "python3:gdrv0" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x555557911dc0) at futex-internal.c:57 66 Thread 0x7ffed8bf9640 (LWP 3097874) "python3:gdrv0" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffef8160394) at futex-internal.c:57 68 Thread 0x7ffebb7fe640 (LWP 3097876) "ThreadPoolSingl" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffebb7fdc98) at futex-internal.c:57 361 Thread 0x7ffedabfd640 (LWP 3099598) "python3:gdrv0" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5555575d9240) at futex-internal.c:57Thanks for taking a look! Let me know if you need more information. Appreciate you taking a look, despite the lack of a simpler reproducer.
Sorry, I didn't make myself clear. Could I get all the addresses, so I can see how they are inconsistent.
Could you include the above andbaseandtstate->datastack_chunk(the address, as well as the contents).
Also could you also include*frame? if the debugger doesn't haveframe, thenp *(_PyInterpreterFrame *)basedatastack_topshould point into the chunk, anddatastack_limitshould point to the top of the chunk.
The following should be truedatastack_chunk < base < datastack_top < datastack_limit, but isn't 😞
Having the contents offramemight help as we can then see which parts of the stack it is consistent with.Thanks for helping.
Sure:
Name Type Value tstate->datastack_chunk(_PyStackChunk *)0x7fffe4c06000base(PyObject **)0x7ffff7821f10tstate->datastack_top(PyObject **)0x7fffe4c06098tstate->datastack_limit(PyObject **)0x7fffe4c0a000>>> datastack_chunk = 0x7fffe4c06000 >>> base = 0x7ffff7821f10 >>> datastack_top = 0x7fffe4c06098 >>> datastack_limit = 0x7fffe4c0a000 >>> >>> datastack_chunk < base True >>> base < datastack_top False >>> datastack_top < datastack_limit True
(gdb) p *tstate->datastack_chunk $12 = {previous = 0x7ffff781e000, size = 16384, top = 0, data = {<function at remote 0x7fffe4f58520>}}(gdb) p *frame $8 = {f_func = 0x7fffe59ab960, f_globals = {'__name__': 'qutebrowser.browser.browsertab', '__doc__': 'Base class for a wrapper over QWebView/QWebEngineView.', '__package__': 'qutebrowser.browser', '__loader__': <SourceFileLoader(name='qutebrowser.browser.browsertab', path='/home/florian/proj/qutebrowser/git/qutebrowser/browser/browsertab.py') at remote 0x7fffe61e16e0>, '__spec__': <ModuleSpec(name='qutebrowser.browser.browsertab', loader=<...>, origin='/home/florian/proj/qutebrowser/git/qutebrowser/browser/browsertab.py', loader_state=None, submodule_search_locations=None, _uninitialized_submodules=[], _set_fileattr=True, _cached='/home/florian/proj/qutebrowser/git/qutebrowser/browser/__pycache__/browsertab.cpython-311.pyc', _initializing=False) at remote 0x7fffe61e1730>, '__file__': '/home/florian/proj/qutebrowser/git/qutebrowser/browser/browsertab.py', '__cached__': '/home/florian/proj/qutebrowser/git/qutebrowser/browser/__pycache__/browsertab.cpython-311.pyc', '__builtins__': {'__name__': 'builtins', '__doc__': "Built-in functions, exceptions, and ...(truncated), f_builtins = {'__name__': 'builtins', '__doc__': "Built-in functions, exceptions, and other objects.\n\nNoteworthy: None is the `nil' object; Ellipsis represents `...' in slices.", '__package__': '', '__loader__': <type at remote 0x555555bbaad0>, '__spec__': <ModuleSpec(name='builtins', loader=<type at remote 0x555555bbaad0>, origin='built-in', loader_state=None, submodule_search_locations=None, _uninitialized_submodules=[], _set_fileattr=False, _cached=None) at remote 0x7ffff78e5d20>, '__build_class__': <built-in method __build_class__ of module object at remote 0x7ffff78e31d0>, '__import__': <built-in method __import__ of module object at remote 0x7ffff78e31d0>, 'abs': <built-in method abs of module object at remote 0x7ffff78e31d0>, 'all': <built-in method all of module object at remote 0x7ffff78e31d0>, 'any': <built-in method any of module object at remote 0x7ffff78e31d0>, 'ascii': <built-in method ascii of module object at remote 0x7ffff78e31d0>, 'bin': <built-in method bin of module object at remote 0x7ffff78e31d0>, ...(truncated), f_locals = 0x0, f_code = 0x7fffe654a040, frame_obj = 0x0, previous = 0x7ffff7821e88, prev_instr = 0x7fffe654a21e, stacktop = 2, is_entry = false, owner = 0 '\000', localsplus = { <WebEngineTab(is_private=False, win_id=0, tab_id=52, registry=<ObjectRegistry(data={'tab': <...>}, _partial_objs={'tab': <functools.partial at remote 0x7fffa1f5c8a0>}, command_only=[]) at remote 0x7fffe508b670>, data=<TabData(keep_icon=False, viewing_source=False, inspector=None, open_target=<ClickTarget(_value_=1, _name_='normal', __objclass__=<EnumType(_generate_next_value_=<function at remote 0x7ffff7676570>, __module__='qutebrowser.utils.usertypes', __doc__='How to open a clicked link.', _new_member_=<built-in method __new__ of type object at remote 0x555555a1a800>, _use_args_=False, _member_names_=['normal', 'tab', 'tab_bg', 'window', 'hover'], _member_map_={'normal': <...>, 'tab': <ClickTarget(_value_=2, _name_='tab', __objclass__=<...>, _sort_order_=1) at remote 0x7fffe6d48d80>, 'tab_bg': <ClickTarget(_value_=3, _name_='tab_bg', __objclass__=<...>, _sort_order_=2) at remote 0x7fffe6d48e20>, 'window': <ClickTarget(_value_=4, _name_='window', __objclass__=<...>, _sort_order_=3) at remote 0x7fffe6d48ec0>,...(truncated)}}(gdb) pp *frame [Thread 0x7fff1a0f3640 (LWP 3108203) exited] [Thread 0x7fff1a8f4640 (LWP 3108202) exited] [Thread 0x7fff5effd640 (LWP 3108183) exited] [Thread 0x7fff5e7fc640 (LWP 3108184) exited] [Thread 0x7fff5cff9640 (LWP 3108187) exited] [Thread 0x7fff7bfff640 (LWP 3108174) exited] [Thread 0x7fff79ffb640 (LWP 3108178) exited] [Thread 0x7fff7b7fe640 (LWP 3108175) exited] [Thread 0x7fff7affd640 (LWP 3108176) exited] [Thread 0x7fff78ff9640 (LWP 3108180) exited] [Thread 0x7fff3bfff640 (LWP 3108188) exited] [Thread 0x7fff7a7fc640 (LWP 3108177) exited] [Thread 0x7fff5f7fe640 (LWP 3108182) exited] [Thread 0x7fff5d7fa640 (LWP 3108186) exited] [Thread 0x7fff5ffff640 (LWP 3108181) exited] [Thread 0x7fff3b7fe640 (LWP 3108189) exited] [Thread 0x7fff5dffb640 (LWP 3108185) exited] [Thread 0x7fff797fa640 (LWP 3108179) exited] *frame = [ f_func = autoderefcount="1",<PyFunctionObject> = {"{...}"} f_globals = autoderefcount="1",<PyObject> = {"{...}"} f_builtins = autoderefcount="1",<PyObject> = {"{...}"} f_locals = <PyObject*> = {"0x0"} f_code = autoderefcount="1",<PyCodeObject> = {"{...}"} frame_obj = <PyFrameObject*> = {"0x0"} previous = autoderefcount="1",<struct _PyInterpreterFrame> = {"{...}"} prev_instr = autoderefcount="1",<_Py_CODEUNIT> = {"83"} stacktop = <int> = {"2"} is_entry = <_Bool> = {"0"} owner = <char> = {"0"} localsplus = <PyObject*[1]> = {"@0x7ffff7821f58"} ],<_PyInterpreterFrame> = {"{...}"}It looks like the frame and thread's frame stack are both consistent, but that the frame belongs to another threadstate.
Could you check whether the tstate's frame is the same as the frame.
tstate->cframe->current_frame == frameshould be true.
Alsotstateshould be the correct thread. If youupuntil you're in_PyEval_EvalFrameDefaultdoeststate->cframe == &cframe(same run as before, keeping it open as long as I don't need to reboot)
Could you check whether the tstate's frame is the same as the frame.
tstate->cframe->current_frame == frameshould be true.It is:
(gdb) p frame $21 = (_PyInterpreterFrame *) 0x7ffff7821f10 (gdb) p tstate->cframe->current_frame $22 = (struct _PyInterpreterFrame *) 0x7ffff7821f10Also
tstateshould be the correct thread. If youupuntil you're in_PyEval_EvalFrameDefaultdoeststate->cframe == &cframeYep!
(gdb) p tstate->cframe $24 = (_PyCFrame *) 0x7fffffffb7d0 (gdb) p &cframe $25 = (_PyCFrame *) 0x7fffffffb7d0@markshannon You (or someone who knows their way around this code more than I do) don't happen to be at PyConIT by any chance (or perhaps Europython, though that's still a bit farther away)? Would be happy to go for a debugging session there if that's an option, might be easier, given that the issue seems rather tricky to track down.
I will be at EuroPython. Hopefully we will have fixed this before then
Reacted by Freya BruhinUnfortunately I had to cancel my Europython attendance thanks to COVID, so I won't be available for any in-person debugging there 😢.
If you'd be interested in a video call or something to debug this together on my machine, I'd be in for that, though. Or I can just continue to provide information here if you tell me what to look at - I have no idea what the involved code even does exactly, so I'm afraid I probably won't be able to find out anything interesting alone.
Have you tested the current main branch? Have you tried tuning the gc thresholds to see if it makes it easier to trigger?
The mian branch fails in the same way:
python3: Python/pystate.c:2202: _PyThreadState_PopFrame: Assertion `tstate->datastack_top >= base' failed.I don't think it's related to the gc in any way. I tried a
gc.disable()and it still crashes in the same way.@pablogsal should this perhaps be a deferred/release blocker? I realize I'm currently the only reporter seeing this, but it also happens with a Python 3.11 release build and is a crash happening during normal user usage - thus making qutebrowser unusable for daily usage on Python 3.11.
36 remaining items
Thanks @brandtbucher, you rock 🤘
Reacted by Eric Snow, Freya Bruhin and yashI suppose there aren't many reason for the locals allocation to fail other than funky destructors, but is a bit worrisome that our entire test suite never touched this path
One thing that's still unclear to me: How does the locals allocation fail in my original reproducer using
QObject::destroyfrom here? #93252 (comment) (I don't necessarily expect you to answer this, since it might be something PyQt does - but I don't have an answer either...).FWIW I'm currently trying out the fix proposed by @kumaraditya303 above on top of b3, and it looks like the qutebrowser testsuite looks much better with it! ✨
(I also saw a weird SIGIOT when trying to run pytest with the latest 3.11 branch, but that's probably something for a new issue, if none exists yet...)
One thing that's still unclear to me: How does the locals allocation fail in my original reproducer using QObject::destroy from here?
Likely there is some C call that's failing and is raising unraisable exceptions or is cleaning the error indicator. These calls consume stacks that are never pop-ed but they are cleaned.
If the original error doesn't happen again, it has to be something like this (fail to allocate the local stack) because otherwise that would indicate that we are failing to pop in other places.
@pablogsal Out of curiosity, I tried to build with
CXX=clang CC=clang(on Linux), but I could still reproduce after that. Not sure why you can't... I don't have a recent macOS available to test with I'm afraid.I will give #94693 a try somewhen over the next couple of days, I need to take a break after hours of reproducing bugs 😅.
Finally, the pytest issue I have encountered while looking into this one:
The bug has been fixed in main and 3.11 branch. Thank you @brandtbucher and @kumaraditya303
Reacted by Freya Bruhin and yashReacted by Freya Bruhin and Alex WaygoodI just tested the 3.11 fix, and can confirm everything seems to work properly now, including the qutebrowser testsuite (minus a few issues which are probably not CPython bugs).
Thanks a lot, @markshannon @pablogsal @kumaraditya303 @brandtbucher @tiran! ❤️
Reacted by Kumar Aditya, Alex Waygood, Pablo Galindo Salgado, Rahul Jha, yash and Eric Snow
Crash report
When running my application (qutebrowser) in a way it spawns/kills a lot of threads, after around 140 (?) threads spawned/killed, I get (depending on which commit I'm testing) either of:
Referring to:
cpython/Python/pystate.c
Line 2218 in 8d32a5c
I've spent hours on trying to find a more minimal reproducer, but unfortunately, the best I can offer without some help about what I could try next is this:
git clone https://github.com/qutebrowser/qutebrowser.gitcd qutebrowserpython3.11 -m venv .venv.venv/bin/pip install -r requirements.txt -r misc/requirements/requirements-pyqt.txt.venv/bin/python3 -m qutebrowser --temp-basedir -s tabs.last_close blank -s qt.chromium.sandboxing disable-seccomp-bpfdwhich will close the current tabuwhich will reopen itcrasher.pyand make it executable.venv/bin/python3 -m qutebrowser --temp-basedir -s tabs.last_close blank -s qt.chromium.sandboxing disable-seccomp-bpf ":later 1000 spawn -u $PWD/crasher.py"After some time (due to having to skip most commits because of #92112), I was able to bisect this to:
ae0a2b7 ("bpo-44590: Lazily allocate frame objects (GH-27077)", @markshannon)
Error messages
Stacktrace:
Details
Debugging:
Details
Happy to try more or report some debugging information, but I'm afraid I'm stuck at this point.
Your environment