Add documentation

encukou · Aug 15, 2024 · Dec 31, 2023 · Dec 31, 2023 · Jan 15, 2024 · Jan 15, 2024
commit ed09466ad0084ead701b334422da6558537b80ff
@@ -386,11 +386,15 @@ Initializing and finalizing the interpreter
    Undo all initializations made by :c:func:`Py_Initialize` and subsequent use of
    Python/C API functions, and destroy all sub-interpreters (see
    :c:func:`Py_NewInterpreter` below) that were created and not yet destroyed since
-   the last call to :c:func:`Py_Initialize`.  Ideally, this frees all memory
-   allocated by the Python interpreter.  This is a no-op when called for a second
+   the last call to :c:func:`Py_Initialize`. This is a no-op when called for a second
    time (without calling :c:func:`Py_Initialize` again first).  Normally the
    return value is ``0``.  If there were errors during finalization
    (flushing buffered data), ``-1`` is returned.
+
+   Note that Python will do a best effort at freeing all memory allocated by the Python
+   interpreter.  Therefore, any C-Extension should make sure to correctly clean up all
+   of the preveiously allocated PyObjects before using them in subsequent calls to
+   `Py_Initialize`.  Otherwise it introduces vulnerabilities and incorrect behavior.
 
    This function is provided for a number of reasons.  An embedding application
    might want to restart Python without having to restart the application itself.
@@ -406,10 +410,11 @@ Initializing and finalizing the interpreter
    loaded extension modules loaded by Python are not unloaded.  Small amounts of
    memory allocated by the Python interpreter may not be freed (if you find a leak,
    please report it).  Memory tied up in circular references between objects is not
-   freed.  Some memory allocated by extension modules may not be freed.  Some
-   extensions may not work properly if their initialization routine is called more
-   than once; this can happen if an application calls :c:func:`Py_Initialize` and
-   :c:func:`Py_FinalizeEx` more than once.
+   freed.  Interned strings will all be deallocated regarldess of their reference count.  
+   Some memory allocated by extension modules may not be freed.  Some extensions may not
+   work properly if their initialization routine is called more than once; this can
+   happen if an application calls :c:func:`Py_Initialize` and :c:func:`Py_FinalizeEx`
+   more than once.
 
    .. audit-event:: cpython._PySys_ClearAuditHooks "" c.Py_FinalizeEx
 

diff --git a/Doc/whatsnew/3.13.rst b/Doc/whatsnew/3.13.rst
@@ -1363,6 +1363,17 @@ New Features
 * Add :c:func:`Py_HashPointer` function to hash a pointer.
   (Contributed by Victor Stinner in :gh:`111545`.)
 
+* Modified :c:func:`_PyUnicode_ClearInterned` function to always delete all
+  interned strings during a call to :c:func:`Py_Finalize`. This makes all
+  is backwards incompatible to any C-Extension that holds onto an interned
+  string after a call to c:func:`Py_Finalize` and is then reused after a 
+  call to c:func:`Py_Initialize`. Any issues arising from this behavior will
+  normally result in crashes during the exectuion of the subsequent call to
+  c:func:`Py_Initatilize` from accessing uninitialized memory. To fix, use
+  an address sanitizer (i.e. ASAN) to identify any use-after-free coming from
+  an interned string and deallocate it during module shutdown.
+  (Contribued by Eddie Elizondo in :gh:`113601`.)
+
 
 Porting to Python 3.13
 ----------------------

diff --git a/Misc/NEWS.d/next/Core and Builtins/2024-01-15-18-11-48.gh-issue-113190.OwQX64.rst b/Misc/NEWS.d/next/Core and Builtins/2024-01-15-18-11-48.gh-issue-113190.OwQX64.rst
@@ -0,0 +1,20 @@
+This updates the interned string deallocation function
+`_PyUnicode_ClearInterned` to delete all interned strings during runtime
+finalization when calling `Py_Finalize`, regardless of their reference
+count.
+
+Worth noting that if an extension accidentally holds onto a interned string,
+event after calling Py_Finalize, it will result in use-after-free error,
+leaving the user to a potential vulnerabilities. That said, the history of
+how these interned strings are handled have been changing during throughout
+different versions. For example:
+
+* In Python 3.9 and older, interned strings were never deleted. Only special
+build for Valgrind and Purity cleared them at exit
+* In Python 3.10 and 3.11, interned strings are always deleted at exit
+* In Python 3.12, interned strings are deleted only if Python is built in
+debug mode: they are not deleted in release mode
+* In Python 3.13, interned strings will now be akll deleted
+
+Given how we've changed guarantees throughout different versions, we do not
+expect that users actively rely on this behavior for their extensions.