Visitar URL original
Add Python 3.12 / 3.13 / 3.14 support (#122) · QuantConnect/pythonnet@cea3c89 · GitHub
Skip to content

Commit cea3c89

Browse files
Martin-Molinerofilmorclaude
authored
Add Python 3.12 / 3.13 / 3.14 support (#122)
* Initial 3.14 commit (cherry picked from commit caac33d) * Apply alignment fix (cherry picked from commit e10d333) * Disable problematic GC tests (cherry picked from commit e976558) * Set ht_token to NULL in Python 3.14 (cherry picked from commit 65af098) * Workaround for blocked PyObject_GenericSetAttr in metatypes Python 3.14 introduced a new assertion that prevents us from using PyObject_GenericSetAttr directly in our meta type. To work around this, we manipulate the type dict directly. This workaround is a simplified variant of Cython's workaround from cython/cython#6325. The relevant Python change is in python/cpython#118454 (cherry picked from commit 08550d0) * Use PyThreadState_GetUnchecked on Python 3.13 (cherry picked from commit f3face0) * Remove deprecated function call (cherry picked from commit 8dfe408) * Assign True instead of None to __clear_reentry_guard__ Not at all sure why this helps, but when assigning `None` instead, the object is gone at the time of garbage collection. (cherry picked from commit 8e0333d) * Move tp_clear workaround to .NET In Python 3.14, the objects __dict__ seems to already be half deconstructed, leading to crashes during garbage collection. Since gc in Python is single-threaded (I think :)), it should be fine to have a single static for this. If that is not true, we can always use a thread-local instead. (cherry picked from commit 908e13b) * Use non-BOM encodings (pythonnet#2370) * Use non-BOM encodings The documentation of the used `PyUnicode_DecodeUTF16` states that not passing `*byteorder` or passing a 0 results in the first two bytes, if they are the BOM (U+FEFF, zero-width no-break space), to be interpreted and skipped, which is incorrect when we convert a known "non BOM" string, which all strings from C# are. (cherry picked from commit 195cde6) * Preserve SyntaxError source line in message on Python 3.12+ Python 3.12 eagerly normalizes the error indicator, so PyErr_Fetch now hands us the SyntaxError instance (whose str() omits the offending source line) instead of the raw args tuple (whose str() included it). Callers that surface PythonException.Message for compile diagnostics therefore lost the offending source text on 3.12+. GetMessage now re-appends the SyntaxError 'text' attribute when present. This is a no-op on <=3.11 (there the fetched value is a tuple without the SyntaxError attributes) and only affects SyntaxError messages. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Make embed tests compatible with Python 3.12+ behavior changes Three CPython behavior changes surfaced as failures/crashes once the overload-resolution crash was fixed, all on 3.12+: - ClassManagerTests.BindsCorrectOverloadForClassName crashed the host with "Python memory allocator called without holding the GIL". TestClass2's Get(PyObject o) re-enters Python via ToPython() while MethodBinder has released the GIL (allow_threads) around the managed call. A managed callback that re-enters Python must re-acquire the GIL; tolerated on <=3.11, fatal on 3.12+. Wrap the body in using (Py.GIL()). - TestGetsPythonCodeInfoInStackTrace[ForNestedInterop]: 3.12+ adds caret indicator lines (e.g. "~~~^^^") under source lines in tracebacks, shifting the positional assertions. Drop caret-only lines before asserting (no-op on <=3.11). - Codecs.ExceptionDecodedNoInstance: 3.12 eagerly normalizes exceptions, so the error indicator always carries an instance ("value"); the instanceless scenario this decoder targets can no longer be produced. Guard the test to <3.12. Verified: full embed suite green on 3.11 (910/910) with these changes; the three previously-failing tests pass on 3.14. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Add Python 3.12 / 3.13 ABI offsets and CI jobs The fork resolves PyTypeObject field offsets from the hardcoded TypeOffset{major}{minor} tables (it does not run geninterop at build), so a missing table makes ABI.Initialize throw "Python ABI v... is not supported" and every test on that version fails at PythonEngine init. Only 3.6-3.11 and 3.14 tables were present. Vendor the 3.12 and 3.13 tables from pythonnet/pythonnet upstream (byte-identical to upstream master; same source as the already-present TypeOffset314) and add 3.12 + 3.13 to the CI matrix. Local verification (uv standalone CPython 3.13, this branch's fixes): embed suite ABI-initializes correctly and runs 847 passed / 0 failed (parity with 3.14). 3.12 table is vendored from the same authoritative source; CI exercises it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Support Python 3.13/3.14 re-init in tests; drop obsolete TestDomainReload The embed-test suite re-initializes the interpreter per fixture. On CPython 3.13/3.14 the suite aborted with "Failed to import encodings module" - not a filesystem problem (strace shows the file opens fine) but interpreter import state corrupted across re-initialization. Root cause isolated to a single test: TestPythonEngineProperties.SetPythonPath. It uses PythonEngine.PythonPath, which pins a fixed module search path via the deprecated Py_SetPath. CPython 3.13+ keeps that path config in _PyRuntime across Py_Finalize and offers no way to reset it back to auto-computation without the PyConfig API, so once this test runs every later re-initialization in the same process is forced onto the pinned path and eventually cannot bootstrap encodings. All other fixtures - including the normal Initialize/Shutdown cycles in pyinitialize and TestFinalizer - run fine in a single process. Run only SetPythonPath in its own test process so it cannot pollute the rest of the suite. Verified locally: full embed suite green on 3.11, 3.13 and 3.14 (main run 908 / SetPythonPath 1, 0 failures); no regression. Also delete TestDomainReload: AppDomain reload is not supported on modern .NET (single-domain), so those tests (MarshalByRefObject / AppDomain.CreateDomain) are obsolete. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Benedikt Reinartz <filmor@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent e360799 commit cea3c89

29 files changed

Lines changed: 684 additions & 466 deletions

‎.github/workflows/main.yml‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ jobs:
2222
strategy:
2323
fail-fast: false
2424
matrix:
25-
python: ["3.8", "3.9", "3.10", "3.11"]
25+
python: ["3.8", "3.9", "3.10", "3.11", "3.12", "3.13", "3.14"]
2626

2727
steps:
2828
- name: Checkout code
@@ -49,7 +49,16 @@ jobs:
4949
echo PYTHONHOME=$(python -c 'import sys; print(sys.prefix)') >> $GITHUB_ENV
5050
5151
- name: Embedding tests
52-
run: dotnet test --runtime any-x64 --logger "console;verbosity=detailed" src/embed_tests/
52+
run: dotnet test --runtime any-x64 --logger "console;verbosity=detailed" src/embed_tests/ --filter "FullyQualifiedName!~SetPythonPath"
53+
54+
# SetPythonPath exercises PythonEngine.PythonPath, which uses the deprecated Py_SetPath
55+
# to pin a fixed module search path. CPython 3.13+ keeps that path config in _PyRuntime
56+
# across Py_Finalize and provides no way to reset it back to auto-computation without the
57+
# PyConfig API, so once this test runs, every later interpreter re-initialization in the
58+
# same process is forced onto the pinned path and eventually fails to import encodings.
59+
# Run it in its own process so it cannot pollute the rest of the suite.
60+
- name: Embedding tests (SetPythonPath, isolated process)
61+
run: dotnet test --runtime any-x64 --logger "console;verbosity=detailed" src/embed_tests/ --filter "FullyQualifiedName~SetPythonPath"
5362

5463
- name: Python Tests (.NET Core)
5564
run: pytest --runtime netcore tests

‎pyproject.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ dependencies = [
1313
"clr_loader>=0.2.2,<0.3.0"
1414
]
1515

16-
requires-python = ">=3.7, <3.12"
16+
requires-python = ">=3.7, <3.15"
1717

1818
classifiers = [
1919
"Development Status :: 5 - Production/Stable",

‎src/embed_tests/ClassManagerTests.cs‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -965,7 +965,14 @@ private class TestClass2 : TestClass1
965965
{
966966
public PyObject Get(PyObject o)
967967
{
968-
return "PyObject Get(PyObject o)".ToPython();
968+
// This managed method is invoked by pythonnet with the GIL released
969+
// (MethodBinder uses allow_threads around managed calls). Re-entering
970+
// Python here - creating a str via ToPython() - requires re-acquiring
971+
// the GIL; on CPython 3.12+ allocating without the GIL is fatal.
972+
using (Py.GIL())
973+
{
974+
return "PyObject Get(PyObject o)".ToPython();
975+
}
969976
}
970977

971978
public dynamic Get(Type t)

‎src/embed_tests/Codecs.cs‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -361,6 +361,14 @@ from datetime import datetime
361361
[Test]
362362
public void ExceptionDecodedNoInstance()
363363
{
364+
if (Runtime.PyVersion >= new Version(3, 12))
365+
{
366+
// Python 3.12+ eagerly normalizes the error indicator, so an exception
367+
// always reaches the decoder with an instance ("value"). The instanceless
368+
// error scenario this decoder targets can no longer be produced by CPython.
369+
Assert.Ignore("Instanceless exceptions are not produced on Python 3.12+ (eager normalization).");
370+
}
371+
364372
PyObjectConversions.RegisterDecoder(new InstancelessExceptionDecoder());
365373
using var scope = Py.CreateScope();
366374
var error = Assert.Throws<ValueErrorWrapper>(() => PythonEngine.Exec(

0 commit comments

Comments
 (0)