Visitar URL original
C API: Cython 3.0 uses private functions removed in Python 3.13 (numpy 1.25.1 fails to build) · Issue #107076 · python/cpython · GitHub
Skip to content

C API: Cython 3.0 uses private functions removed in Python 3.13 (numpy 1.25.1 fails to build) #107076

Closed as duplicate
Closed as duplicate
C API: Cython 3.0 uses private functions removed in Python 3.13 (numpy 1.25.1 fails to build)#107076
Labels
topic-C-APItype-bugAn unexpected behavior, bug, or error

Description

@vstinner
vstinner
opened on Jul 22, 2023

numpy 1.25.1 fails to build on Python 3.13: Cython uses private C API removed in Python 3.13. Example of build errors.

I open an issue since Python 3.13 does not provide obvious replacement for these removed functions.

(1) Cython uses removed _PyInterpreterState_GetConfig():

  static void __Pyx_init_assertions_enabled(void) {
    __pyx_assertions_enabled_flag = ! _PyInterpreterState_GetConfig(__Pyx_PyThreadState_Current->interp)->optimization_level;
  }

(2) Cython uses removed _PyVectorcall_Function() in __Pyx_PyObject_FastCallDict():

static CYTHON_INLINE PyObject* __Pyx_PyObject_FastCallDict(PyObject *func, PyObject **args, size_t _nargs, PyObject *kwargs) {
    ..
    #if CYTHON_VECTORCALL
    vectorcallfunc f = _PyVectorcall_Function(func);
    if (f) {
        return f(func, args, (size_t)nargs, kwargs);
    }
    ...
}

(3) Cython uses removed _PyUnicode_FastCopyCharacters() in __Pyx_PyUnicode_ConcatInPlaceImpl()

static CYTHON_INLINE PyObject *__Pyx_PyUnicode_ConcatInPlaceImpl(PyObject **p_left, PyObject *right
        #if CYTHON_REFNANNY
        , void* __pyx_refnanny
        #endif
    ) {
        ...
        // copy 'right' into the newly allocated area of 'left'
        _PyUnicode_FastCopyCharacters(*p_left, left_len, right, 0, right_len);
        return *p_left;
        ...
  }

But also in __Pyx_PyUnicode_Join():

static PyObject* __Pyx_PyUnicode_Join(PyObject* value_tuple, Py_ssize_t value_count, Py_ssize_t result_ulength,
                                      Py_UCS4 max_char) {
            ...
            #if CYTHON_COMPILING_IN_CPYTHON && PY_VERSION_HEX >= 0x030300F0 || defined(_PyUnicode_FastCopyCharacters)
            _PyUnicode_FastCopyCharacters(result_uval, char_pos, uval, 0, ulength);
            #else
            ...
}

Activity

  1. vstinner
    added
    type-bugAn unexpected behavior, bug, or error
    topic-C-API
    on Jul 22, 2023

  2. vstinner commented on Jul 22, 2023

    @vstinner
    vstinner
    on Jul 22, 2023
    MemberAuthor

    (1) Cython uses removed _PyInterpreterState_GetConfig()

    In May 2022, I deprecated legacy global configuration variables like Py_VerboseFlag: issue #93103.

    In November 2022, @scoder (one of Cython maintainers) created issue #99872 "Allow efficient read access to PyConfig options".

  3. vstinner commented on Jul 22, 2023

    @vstinner
    vstinner
    on Jul 22, 2023
    MemberAuthor

    (2) Cython uses removed _PyVectorcall_Function() in __Pyx_PyObject_FastCallDict()

    The public PyVectorcall_Function() function is available since Python 3.8: https://docs.python.org/dev/c-api/call.html#c.PyVectorcall_Function

  4. vstinner commented on Jul 22, 2023

    @vstinner
    vstinner
    on Jul 22, 2023
    MemberAuthor

    (3) Cython uses removed _PyUnicode_FastCopyCharacters() in __Pyx_PyUnicode_ConcatInPlaceImpl()

    The public PyUnicode_CopyCharacters() can be used instead, it's available since Python 3.3: https://docs.python.org/dev/c-api/unicode.html#c.PyUnicode_CopyCharacters

    Is a fast unsafe replacement needed here? Where unsafe means: without error checking (use assertions instead).

  5. scoder commented on Jul 23, 2023

    @scoder
    scoder
    on Jul 23, 2023
    Contributor

    PyUnicode_CopyCharacters
    Is a fast unsafe replacement needed here? Where unsafe means: without error checking (use assertions instead).

    Well. It would be more fair in comparison, although that's barely a reason. For the case that characters are copied between different unicode string types a lot, especially for loads of short strings, this difference would probably be visible, i.e. Py3.13 would appear slower than Py3.12 here. But proving that would require a proper, realistic benchmark. I'll use the public function in Cython on Py3.13 for now and wait for an actual user to complain.

  6. vstinner commented on Jul 23, 2023

    @vstinner
    vstinner
    on Jul 23, 2023
    MemberAuthor

    It's unclear to me why Python exposes an API to modify immutable strings 😁 I'm considering adding a public API for _PyUnicodeWriter which may cover _PyUnicode_FastCopyCharacters() usecase.

    I'm also open to expose _PyUnicode_FastCopyCharacters() in public with an appropriate documentation.

  7. da-woods commented on Jul 23, 2023

    @da-woods
    da-woods
    on Jul 23, 2023
    Contributor

    (2) Cython uses removed _PyVectorcall_Function() in __Pyx_PyObject_FastCallDict()

    The public PyVectorcall_Function() function is available since Python 3.8: https://docs.python.org/dev/c-api/call.html#c.PyVectorcall_Function

    This isn't true - it's documented badly. #107140

    I mention it because the misleading documentation caught us out when trying to replace it (and you out in the comment above)

  8. da-woods commented on Jul 23, 2023

    @da-woods
    da-woods
    on Jul 23, 2023
    Contributor

    (Deleted previous comment because it wasn't quite right).

    The __Pyx_PyUnicode_ConcatInPlaceImpl is used to optimize repeated addition to the same string. It's an anti-pattern that'd be better replaced with .join but it's an optimization CPython does internally too, and it does make a difference. I wouldn't worry too much about a performance degradation there.

    The use case in .join is probably more significant given that's the recommended way to write string concatenations.

  9. vstinner commented on Jul 23, 2023

    @vstinner
    vstinner
    on Jul 23, 2023
    MemberAuthor

    I'm not asking Cython to use slower code path on purpose. I created this issue to discuss which private functions are used by Cython and see what can be done to provide better APIs for Cython.

  10. vstinner commented on Jul 24, 2023

    @vstinner
    vstinner
    on Jul 24, 2023
    MemberAuthor

    By the way, the main branch of Cython uses the following private _PyDict functions:

    • _PyDict_GetItem_KnownHash()
    • _PyDict_NewPresized()
    • _PyDict_Pop()
    • _PyDict_SetItem_KnownHash()

    Maybe we should consider making them public, or Cython should avoid them (which can have a negative effect on performance). See PR #107145.

    I kept these functions for now, to not make break Cython more :-)

  11. scoder commented on Jul 24, 2023

    @scoder
    scoder
    on Jul 24, 2023
    Contributor

    @da-woods has collected a list of CPython internals that we use in cython/cython#4635

    All of these (minus bugs) are guarded by version checks or feature flags (like CYTHON_USE_PYLIST_INTERNALS).

    I think the dict internals only use version checks, probably because the dict implementation tends to change every couple of releases. We might want to add a general feature flag for the dict internals as well, to make their access easy to disable for users. Looking a bit more through the implementation, we don't actually use "internals" in the sense of direct dict object struct access. We do use private C-API functions, though, in order to avoid performance disadvantages compared to CPython. For example, the _KnownHash functions are used when looking up interned strings for which the hash value has definitely been calculated at initialisation time and anything but a straight, unchecked lookup would just be useless overhead.

    In any case, there is always generic fallback code that only relies on public C-API features, usually for other Python implementations than CPython, or for the Limited API compilation mode.

  12. vstinner commented on Jul 24, 2023

    @vstinner
    vstinner
    on Jul 24, 2023
    MemberAuthor

    We do use private C-API functions, though, in order to avoid performance disadvantages compared to CPython

    I totally get it and I create this issue to discuss that: should we expose these private functions? If they help to make C extensions faster, maybe we should expose them so more C extensions can benefit from them.

  13. serhiy-storchaka commented on Aug 29, 2023

    @serhiy-storchaka
    serhiy-storchaka
    on Aug 29, 2023
    Member

    For example, the _KnownHash functions are used when looking up interned strings for which the hash value has definitely been calculated at initialisation time and anything but a straight, unchecked lookup would just be useless overhead.

    I afraid that it is a preliminary optimization, because every dict lookup function first at all checks whether the key is a string with the cached hash, and then does a straight lookup. The overhead is virtually just the comparison of two pointers.

    _KnownHash functions are more useful in case of slow hashing function, for example in lru_cache() implementation.

  14. vstinner commented on Dec 2, 2023

    @vstinner
    vstinner
    on Dec 2, 2023
    MemberAuthor

    See also #89410 "[C API] Add explicit support for Cython to the C API".

  15. vstinner commented on Dec 20, 2023

    @vstinner
    vstinner
    on Dec 20, 2023
    MemberAuthor

    I mark this issue as duplicate of #89410

  16. vstinner
    closed this as completedon Dec 20, 2023

  17. bje- commented on Jun 22, 2024

    @bje-
    bje-
    on Jun 22, 2024

    Similarly, nanosvg (bundled with wxPython) fails to compile with Python 3.13 because cython generates code that uses private Python API functions whose signatures have changed (namely, _PyLong_AsByteArray).

  18. vstinner commented on Jun 24, 2024

    @vstinner
    vstinner
    on Jun 24, 2024
    MemberAuthor

    Similarly, nanosvg (bundled with wxPython) fails to compile with Python 3.13 because cython generates code that uses private Python API functions whose signatures have changed (namely, _PyLong_AsByteArray).

    I suggest to use the new APIs:

  19. bje- commented on Jun 24, 2024

    @bje-
    bje-
    on Jun 24, 2024

    Similarly, nanosvg (bundled with wxPython) fails to compile with Python 3.13 because cython generates code that uses private Python API functions whose signatures have changed (namely, _PyLong_AsByteArray).

    Newer versions of Cython fix the problem, but I agree that it would be better for Cython to not be using private API functions where possible.

  20. vstinner
    closed this as a duplicateof [C API] Add explicit support for Cython to the C API #89410 on Oct 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    topic-C-APItype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions