Repository navigation
Global String Objects are Interned Only in the First Interpreter #106931
Description
Activity
I prefer option 1 and would like to try make it work after I am in better health.
Reacted by Eric SnowI likewise prefer the first option, though with some other data structure than a dict.
Solutions I've considered:
- use a global
_Py_hashtablefor statically allocated strings and keep the per-interpreter dict for strings allocated by each interpreter - keep all per-interpreter dicts in sync, making a copy of the string in each one (or no copy if statically allocated)
- use a global
_Py_hashtablefor all strings; store only main-interpreter-owned strings (i.e. copy if needed)
- use a global
This is done.
FYI, I didn't backport the fix to 3.12, given how late in the release cycle we are and the reservations of the release manager.
I reopen the issue.
Using tracemalloc, test_sys now fails with a fatal error since commit b72947a:
$ git checkout b72947a8d26915156323ccfd04d273199ecb870c $ make $ ./python -X tracemalloc -m test test_sys -m test_subinterp_intern_dynamically_allocated (...) Debug memory block at address p=0x7f2ae1b16f70: API '' 15987177244304867328 bytes originally requested The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfd): at p-7: 0x00 *** OUCH at p-6: 0x00 *** OUCH at p-5: 0x00 *** OUCH at p-4: 0x00 *** OUCH at p-3: 0x00 *** OUCH at p-2: 0x00 *** OUCH at p-1: 0x00 *** OUCH Because memory is corrupted at the start, the count of bytes requested may be bogus, and checking the trailing pad bytes may segfault. The 8 pad bytes at tail=0xddde5c2ae1b16f70 are Fatal Python error: Segmentation fault Current thread 0x00007f2aef939740 (most recent call first): <no Python frame>gdb traceback:
Thread 1 "python" received signal SIGSEGV, Segmentation fault. 0x0000000000561014 in _PyObject_DebugDumpAddress (p=0x7fffe9ebaf70) at Objects/obmalloc.c:2409 2409 if (tail[i] != PYMEM_FORBIDDENBYTE) { Missing separate debuginfos, use: dnf debuginfo-install bzip2-libs-1.0.8-13.fc38.x86_64 glibc-2.37-4.fc38.x86_64 libffi-3.4.4-2.fc38.x86_64 openssl-libs-3.0.9-2.fc38.x86_64 xz-libs-5.4.1-1.fc38.x86_64 zlib-1.2.13-3.fc38.x86_64 (gdb) where #0 0x0000000000561014 in _PyObject_DebugDumpAddress (p=0x7fffe9ebaf70) at Objects/obmalloc.c:2409 #1 0x0000000000560cfd in _PyMem_DebugCheckAddress (func=0x7e8850 <__func__.8> "_PyMem_DebugRawFree", api=114 'r', p=0x7fffe9ebaf70) at Objects/obmalloc.c:2327 #2 0x000000000056082c in _PyMem_DebugRawFree (ctx=0xb15920 <_PyRuntime+512>, p=0x7fffe9ebaf70) at Objects/obmalloc.c:2160 #3 0x000000000055e202 in PyMem_RawFree (ptr=0x7fffe9ebaf70) at Objects/obmalloc.c:686 #4 0x00000000005603c0 in _PyObject_Free (ctx=0x0, p=0x7fffe9ebaf70) at Objects/obmalloc.c:1854 #5 0x0000000000560872 in _PyMem_DebugRawFree (ctx=0xb15980 <_PyRuntime+608>, p=0x7fffe9ebaf80) at Objects/obmalloc.c:2164 #6 0x0000000000560c63 in _PyMem_DebugFree (ctx=0xb15980 <_PyRuntime+608>, ptr=0x7fffe9ebaf80) at Objects/obmalloc.c:2297 #7 0x000000000055e58a in PyObject_Free (ptr=0x7fffe9ebaf80) at Objects/obmalloc.c:831 #8 0x00000000005b03e0 in unicode_dealloc (unicode=<unknown at remote 0x7fffe9ebaf80>) at Objects/unicodeobject.c:1613 #9 0x000000000055cfdf in _Py_Dealloc (op=<unknown at remote 0x7fffe9ebaf80>) at Objects/object.c:2615 #10 0x00000000006fe895 in Py_DECREF (filename=0x8e8cce "Python/tracemalloc.c", lineno=775, op=<unknown at remote 0x7fffe9ebaf80>) at ./Include/object.h:699 #11 0x00000000006ffeb0 in tracemalloc_clear_filename (value=0x7fffe9ebaf80) at Python/tracemalloc.c:775 #12 0x00000000006a369e in _Py_hashtable_destroy_entry (ht=0xbdbf10, entry=0x18d0890) at Python/hashtable.c:383 #13 0x00000000006a3724 in _Py_hashtable_clear (ht=0xbdbf10) at Python/hashtable.c:399 #14 0x00000000006fff4e in tracemalloc_clear_traces () at Python/tracemalloc.c:795 #15 0x0000000000700373 in _PyTraceMalloc_Stop () at Python/tracemalloc.c:971 #16 0x0000000000700106 in tracemalloc_deinit () at Python/tracemalloc.c:870 #17 0x0000000000700cf1 in _PyTraceMalloc_Fini () at Python/tracemalloc.c:1344 #18 0x00000000006d49f6 in Py_FinalizeEx () at Python/pylifecycle.c:1895 #19 0x00000000007115d0 in Py_RunMain () at Modules/main.c:690 #20 0x000000000071165f in pymain_main (args=0x7fffffffde10) at Modules/main.c:718 #21 0x00000000007116d9 in Py_BytesMain (argc=8, argv=0x7fffffffdf78) at Modules/main.c:742 #22 0x000000000041b836 in main (argc=8, argv=0x7fffffffdf78) at ./Programs/python.c:15 (gdb) frame 11 #11 0x00000000006ffeb0 in tracemalloc_clear_filename (value=0x7fffe9ebaf80) at Python/tracemalloc.c:775 775 Py_DECREF(filename); (gdb) p /x *filename $4 = { _ob_next = <unknown at remote 0xdddddddddddddddd>, _ob_prev = <unknown at remote 0xdddddddddddddddd>, { ob_refcnt = 0xdddddddddddddddd, ob_refcnt_split = {0xdddddddd, 0xdddddddd} }, ob_type = 0xdddddddddddddddd }
When a string object is interned via
_PyUnicode_InternInPlace(), its "state.interned" field is set. Afterward, subsequent calls to_PyUnicode_InternInPlace()will skip that string. The problem is that some strings may be used in multiple interpreters, which each have their own interned dict. The string is shared between the interpreters, along with its "state.interned" field. That means the string will only be interned in the first interpreter where_PyUnicode_InternInPlace()is called (ignoring races in the function, e.g. gh-106930).We need to fix it so one of the following is true:
Linked PRs