Visitar URL original
gh-136872: Build pymalloc in sanitizer builds, but disable it by defa… · python/cpython@2791c35 · GitHub
Skip to content

Commit 2791c35

Browse files
vasiliykvasiliyk
authored
gh-136872: Build pymalloc in sanitizer builds, but disable it by default (GH-157934)
--with-*address-sanitizer and --with-memory-sanitizer unconditionally disabled pymalloc (even when --with-pymalloc was given explicitly). Enable it, so that it's available with `PYTHONMALLOC=pymalloc`. Disable it by default on the affected sanitizer builds.
1 parent acfa776 commit 2791c35

9 files changed

Lines changed: 48 additions & 28 deletions

File tree

‎Doc/c-api/memory.rst‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -437,6 +437,8 @@ Release build ``"pymalloc"`` ``malloc``
437437
Debug build ``"pymalloc_debug"`` ``malloc`` + debug ``pymalloc`` + debug ``pymalloc`` + debug
438438
Release build, without pymalloc ``"malloc"`` ``malloc`` ``malloc`` ``malloc``
439439
Debug build, without pymalloc ``"malloc_debug"`` ``malloc`` + debug ``malloc`` + debug ``malloc`` + debug
440+
Release build, with ASan or MSan ``"malloc"`` ``malloc`` ``malloc`` ``malloc``
441+
Debug build, with ASan or MSan ``"malloc_debug"`` ``malloc`` + debug ``malloc`` + debug ``malloc`` + debug
440442
Free-threaded build ``"mimalloc"`` ``mimalloc`` ``mimalloc`` ``mimalloc``
441443
Free-threaded debug build ``"mimalloc_debug"`` ``mimalloc`` + debug ``mimalloc`` + debug ``mimalloc`` + debug
442444
=================================== ======================= ==================== ====================== ======================
@@ -451,6 +453,10 @@ Legend:
451453
* "+ debug": with :ref:`debug hooks on the Python memory allocators
452454
<pymem-debug-hooks>`.
453455
* "Debug build": :ref:`Python build in debug mode <debug-build>`.
456+
* "with ASan or MSan": sanitizer build as configured using the
457+
:option:`--with-address-sanitizer`,
458+
:option:`--with-hwaddress-sanitizer`, and/or
459+
:option:`--with-memory-sanitizer` option.
454460
455461
.. _customize-memory-allocators:
456462
@@ -705,9 +711,11 @@ This allocator is disabled if Python is configured with the
705711
:option:`--without-pymalloc` option. It can also be disabled at runtime using
706712
the :envvar:`PYTHONMALLOC` environment variable (ex: ``PYTHONMALLOC=malloc``).
707713
708-
Typically, it makes sense to disable the pymalloc allocator when building
709-
Python with AddressSanitizer (:option:`--with-address-sanitizer`) which helps
710-
uncover low level bugs within the C code.
714+
The pymalloc allocator is disabled by default when Python is built with
715+
a sanitizer which does not track pymalloc allocations
716+
(:option:`--with-address-sanitizer`, :option:`--with-hwaddress-sanitizer`,
717+
:option:`--with-memory-sanitizer`).
718+
Use :envvar:`PYTHONMALLOC=pymalloc <PYTHONMALLOC>` to enable pymalloc.
711719
712720
Customize pymalloc Arena Allocator
713721
----------------------------------

‎Doc/using/configure.rst‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1016,9 +1016,9 @@ Debug options
10161016
.. option:: --with-address-sanitizer
10171017

10181018
Enable AddressSanitizer memory error detector, ``asan`` (default is no).
1019-
To improve ASan detection capabilities you may also want to combine this
1020-
with :option:`--without-pymalloc` to disable the specialized small-object
1021-
allocator whose allocations are not tracked by ASan.
1019+
1020+
When built with ``asan``, Python uses ``malloc`` instead of :ref:`pymalloc <pymalloc>` by default.
1021+
Set :envvar:`PYTHONMALLOC=pymalloc <PYTHONMALLOC>` to use pymalloc.
10221022

10231023
.. versionadded:: 3.6
10241024

@@ -1033,6 +1033,9 @@ Debug options
10331033
<https://clang.llvm.org/docs/HardwareAssistedAddressSanitizerDesign.html>`_
10341034
for more information.
10351035

1036+
When built with ``hwasan``, Python uses ``malloc`` instead of :ref:`pymalloc <pymalloc>` by default.
1037+
Set :envvar:`PYTHONMALLOC=pymalloc <PYTHONMALLOC>` to use pymalloc.
1038+
10361039
.. versionadded:: next
10371040

10381041
.. option:: --with-memory-sanitizer
@@ -1043,6 +1046,9 @@ Debug options
10431046
not built with MSan, so either build all dependencies with MSan or disable
10441047
the extension modules that use them in :file:`Modules/Setup.local`.
10451048

1049+
When built with ``msan``, Python uses ``malloc`` instead of :ref:`pymalloc <pymalloc>` by default.
1050+
Set :envvar:`PYTHONMALLOC=pymalloc <PYTHONMALLOC>` to use pymalloc.
1051+
10461052
.. versionadded:: 3.6
10471053

10481054
.. option:: --with-undefined-behavior-sanitizer

‎Include/internal/pycore_pymem_init.h‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,10 @@ extern void _PyMem_MiRawFree(void *, void *);
3636
extern void* _PyMem_MiRawRealloc(void *, void *, size_t);
3737
# undef PYRAW_ALLOC
3838
# define PYRAW_ALLOC {NULL, _PyMem_MiRawMalloc, _PyMem_MiRawCalloc, _PyMem_MiRawRealloc, _PyMem_MiRawFree}
39+
#elif defined(_Py_ADDRESS_SANITIZER) || defined(_Py_MEMORY_SANITIZER)
40+
// Keep in sync with the default allocators in Objects/obmalloc.c.
41+
# define PYOBJ_ALLOC PYRAW_ALLOC
42+
# define PYMEM_ALLOC PYOBJ_ALLOC
3943
#elif defined(WITH_PYMALLOC)
4044
extern void* _PyObject_Malloc(void *, size_t);
4145
extern void* _PyObject_Calloc(void *, size_t, size_t);

‎Lib/test/test_cmd_line.py‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -845,10 +845,13 @@ def test_xdev(self):
845845
code = "import _testinternalcapi; print(_testinternalcapi.pymem_getallocatorsname())"
846846
with support.SuppressCrashReport():
847847
out = self.run_xdev("-c", code, check_exitcode=False)
848-
if support.with_pymalloc():
849-
alloc_name = "pymalloc_debug"
850-
elif support.Py_GIL_DISABLED:
848+
if support.Py_GIL_DISABLED:
851849
alloc_name = "mimalloc_debug"
850+
elif support.check_sanitizer(address=True, memory=True):
851+
# ASan and MSan builds default to malloc, even with pymalloc.
852+
alloc_name = "malloc_debug"
853+
elif support.with_pymalloc():
854+
alloc_name = "pymalloc_debug"
852855
else:
853856
alloc_name = "malloc_debug"
854857
self.assertEqual(out, alloc_name)
@@ -927,10 +930,15 @@ def test_pythonmalloc(self):
927930
# Test the PYTHONMALLOC environment variable
928931
malloc = not support.Py_GIL_DISABLED
929932
pymalloc = support.with_pymalloc()
933+
sanitizer = support.check_sanitizer(address=True, memory=True)
930934
mimalloc = support.with_mimalloc()
931935
if support.Py_GIL_DISABLED:
932936
default_name = 'mimalloc_debug' if support.Py_DEBUG else 'mimalloc'
933937
default_name_debug = 'mimalloc_debug'
938+
elif sanitizer:
939+
# ASan and MSan builds default to malloc, even with pymalloc.
940+
default_name = 'malloc_debug' if support.Py_DEBUG else 'malloc'
941+
default_name_debug = 'malloc_debug'
934942
elif pymalloc:
935943
default_name = 'pymalloc_debug' if support.Py_DEBUG else 'pymalloc'
936944
default_name_debug = 'pymalloc_debug'

‎Lib/test/test_sys.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1194,8 +1194,10 @@ def test_debugmallocstats(self):
11941194
# The sysconfig vars are not available on Windows.
11951195
if sys.platform != "win32":
11961196
with_pymalloc = sysconfig.get_config_var("WITH_PYMALLOC")
1197+
with_sanitizer = support.check_sanitizer(address=True, memory=True)
11971198
self.assertIn(b"free PyDictObjects", err)
1198-
if with_pymalloc:
1199+
# ASan and MSan builds default to malloc, even with pymalloc.
1200+
if with_pymalloc and not with_sanitizer:
11991201
self.assertIn(b'Small block threshold', err)
12001202

12011203
# The function has no parameter

‎Misc/NEWS.d/next/Build/2026-09-21-19-20-42.gh-issue-136872.a2tSc_.rst‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
Python built with :option:`--with-address-sanitizer` or
2+
:option:`--with-memory-sanitizer` now includes pymalloc, but uses ``malloc``
3+
as the default memory allocator. Set
4+
:envvar:`PYTHONMALLOC=pymalloc <PYTHONMALLOC>` to use pymalloc.

‎Objects/obmalloc.c‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -433,6 +433,12 @@ void* _PyObject_Realloc(void *ctx, void *ptr, size_t size);
433433
# define PYRAW_ALLOC MIMALLOC_RAWALLOC
434434
# define PYMEM_ALLOC MIMALLOC_ALLOC
435435
# define PYOBJ_ALLOC MIMALLOC_OBJALLOC
436+
#elif defined(_Py_ADDRESS_SANITIZER) || defined(_Py_MEMORY_SANITIZER)
437+
// ASan and MSan do not track pymalloc blocks, so use malloc by default.
438+
// pymalloc can still be selected at runtime.
439+
# define PYRAW_ALLOC MALLOC_ALLOC
440+
# define PYMEM_ALLOC MALLOC_ALLOC
441+
# define PYOBJ_ALLOC MALLOC_ALLOC
436442
#elif defined(WITH_PYMALLOC)
437443
# define PYRAW_ALLOC MALLOC_ALLOC
438444
# define PYMEM_ALLOC PYMALLOC_ALLOC

‎configure‎

Lines changed: 0 additions & 9 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎configure.ac‎

Lines changed: 0 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -3526,8 +3526,6 @@ AC_ARG_WITH([address_sanitizer],
35263526
AC_MSG_RESULT([$withval])
35273527
BASECFLAGS="-fsanitize=address -fno-omit-frame-pointer $BASECFLAGS"
35283528
LDFLAGS="-fsanitize=address $LDFLAGS"
3529-
# ASan works by controlling memory allocation, our own malloc interferes.
3530-
with_pymalloc="no"
35313529
],
35323530
[AC_MSG_RESULT([no])])
35333531

@@ -3551,11 +3549,6 @@ AX_CHECK_COMPILE_FLAG([$hwasan_flags],[
35513549
BASECFLAGS="$hwasan_flags -fno-omit-frame-pointer $BASECFLAGS"
35523550
LDFLAGS="$hwasan_flags $LDFLAGS"
35533551
],[AC_MSG_ERROR([The selected compiler doesn't support hardware address sanitizer])])
3554-
# HWASan works by controlling memory allocation, our own malloc interferes,
3555-
# so disable it by default, but allow --with-pymalloc to override.
3556-
if test -z "$with_pymalloc"; then
3557-
with_pymalloc="no"
3558-
fi
35593552
],
35603553
[AC_MSG_RESULT([no])])
35613554

@@ -3572,8 +3565,6 @@ AX_CHECK_COMPILE_FLAG([-fsanitize=memory],[
35723565
BASECFLAGS="-fsanitize=memory -fsanitize-memory-track-origins=2 -fno-omit-frame-pointer $BASECFLAGS"
35733566
LDFLAGS="-fsanitize=memory -fsanitize-memory-track-origins=2 $LDFLAGS"
35743567
],[AC_MSG_ERROR([The selected compiler doesn't support memory sanitizer])])
3575-
# MSan works by controlling memory allocation, our own malloc interferes.
3576-
with_pymalloc="no"
35773568
],
35783569
[AC_MSG_RESULT([no])])
35793570

0 commit comments

Comments
 (0)