Visitar URL original
Prepare Python 3.15 with rc3 libraries and native prerequisites by youknowdot · Pull Request #8954 · RustPython/RustPython · GitHub
Skip to content

Prepare Python 3.15 with rc3 libraries and native prerequisites - #8954

Draft
youknowdot wants to merge 71 commits into
RustPython:mainfrom
youknowdot:cpython-315-mega-integration
Draft

youknowdot wants to merge 71 commits into
RustPython:mainfrom
youknowdot:cpython-315-mega-integration

Conversation

@youknowdot

@youknowdot youknowdot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

RustPython's development target moves from 3.14 to 3.15, based on CPython v3.15.0rc3 (8a8eb0b90d40f0e03f8252e9f73cad156e1caca8). This mega upgrade combines startup/import/venv, the standard library and paired canonical tests, with the runtime prerequisites needed to execute them. Complete 3.15 compatibility remains work in progress.

The integration includes sentinel/frozendict, compile(module), atomic import ordering, lazy imports (including the exercised C-API exports), unpacking comprehensions, compiler/JIT/opcode updates, AST and diagnostics, typing, profiling, ctypes/C-API, SSL, SQLite and XML contracts. Full official rc3 test sources replace the previously incomplete test updates; canonical assertions were not rewritten to manufacture successful results. Proven stale native failure markers were removed after execution. Existing Rust diagnostic/opcode snapshots were updated with explicit user approval.

Two narrowly patched dependencies retain their official licenses and source/checksum provenance: the existing Ruff parser 0.16.5 with two official parser backports, and xml 1.3.0 with cumulative entity-expansion accounting. Cargo uses local [patch.crates-io] overrides without a broad dependency upgrade. The parser vendor adds roughly 1.1 MB of file content (1.7 MB on disk); the XML source adds roughly 300 KB. Their exact patches and removal guidance are included.

Added snippets were audited against rc3 behavior: the earlier reduction remains 39→27 files and 253→59 behavior groups, plus one standalone frozendict probe with three methods. A further six duplicate assertions were removed from the atomic-import snippet. Remaining snippets cover distinct native callback/reentry/ownership contracts; development differential probes remain outside the repository. Nine approved legacy snippets, including _imp.create_builtin, reflect rc3 behavior.

Latest rebased head and focused verification

Head 1aba241160f5762365a04c4c9ce9a11a3b0fbb98 is based on verified main 7536730ed6afb948ffa335fc113a0fcd9af0cdbf. Rebase preserves the migration and its merge topology, including the sentinel/frozendict resolution. Main's C-API recursion, CSV and specialized-list finalizer fixes are retained. The latest validation also exposed and fixed callable-pointer cast destinations and ordinary/null typed-pointer callback decoding, with matching CIF classification.

Fresh checks on this head, Linux x86-64, Rust 1.99.0:

  • Rust workspace: 1,351 passed /18 ignored; separate C-API: 121 passed /4 ignored. Both Clippy gates and the sqlite,capi release build pass without warnings; configured hooks pass
  • Ten Python families attempted: nine complete successfully; venv retains one platform error. Total 2,292 tests run, 164 skipped, 14 retained expected failures, 1 error. The families are list, CSV, site, full importlib, venv, lazy imports, type annotations, marshal, ctypes and exceptions
  • Site: 97 run /7 skipped /4 expected failures. Full importlib: 1,219 /54 /1. Lazy imports: 161 /0 /0
  • Venv: 42 run /10 skipped /4 expected failures, with BasicTest.test_multiprocessing failing when forkserver creates an AF_UNIX socket (PermissionError: [Errno 1] Operation not permitted); the recursion check and other exercised cases complete
  • Full snippets: 522 passed /12 platform failures, covering all 534 unique cases in disjoint native and reference/host selections. Native 261/4, exact rc3 reference 261/4, host harness 0/4. The same AF_UNIX and unmapped UID/GID failures remain visible; no source skips or assertions hide them
  • The callable-address/ownership and ordinary/null pointer callback probes pass. The rc3 reference was freshly built from the exact pinned source with the extensions needed by these checks

One additional implementation defect remains: marshalling a code object after assigning a lone-surrogate filename can panic. The canonical marshal family does not cover it, and its proposed correction remains held. The successful suites are therefore scoped evidence, not complete 3.15 compatibility. The full 183-family matrix below is explicitly historical, not a fresh run of every family on this head.

Independent Python 3.14 changes

The compatible fixes are published separately for review on the unchanged 3.14 target:

Each code extraction has standalone 3.14 build and test evidence in its PR. The broader typed-callback/py_object extraction is held because its canonical dllist path still produces an unraisable error. The cast-only PR excludes that path. These implementations remain in the mega upgrade while the standalone changes are reviewed. Existing atomic-import PR #8948 is reused.

Broader pre-rebase validation

Source fb0f7b67285e8a0aa25f51143952a19098caab10, Linux x86-64, Rust 1.99.0. Release checks use sqlite,capi; default build/startup is also verified with _testcapi absent. All commits run normal configured hooks.

  • Rust workspace: 1,351 passed /18 ignored /0 failed; separate C-API: 120 passed /4 ignored /0 failed. Both all-target Clippy gates finish without warnings
  • All 183 updated Python families now have bounded current attempts: 154 execute successfully, 12 wholly skip, 13 fail on demonstrated platform restrictions, 3 are resource-denied, and 1 is partial. Coverage includes 35,475 tests run, 2,971 skipped and 279 retained expected failures, with no unexpected successes or outer timeouts. Retained expected failures are still incompatibilities, not passes
  • The full-family sweep used the preceding release; 16 affected families were then rerun on the final runtime, totaling 5,689 tests with zero failures/errors, 349 skips and 49 expected failures. The other 167 rows retain the explicitly recorded preceding-runtime evidence; this is not a fresh 183-family run on the last binary
  • Site: 97 run, 7 skipped, 4 expected failures. Full importlib: 1,219 run, 54 skipped, 1 expected failure. Lazy imports: 161 run, no skips or failures. Venv: 42 run, 10 skipped, 4 expected failures, one AF_UNIX multiprocessing error
  • The former 40 peepholer opcode/optimization failures are fixed; its unchanged rc3 suite passes. The syntax/AST, PEP 798, strings, typing, XML, SSL and other affected canonical families run successfully with their recorded existing exclusions
  • Full snippets: 522 passed /12 failed; native 261/4, exact rc3 reference 261/4, host harness 0/4. The remaining failures are the same demonstrated AF_UNIX/UID-map restrictions. The original 18 actionable snippet failures are closed; GC/import completes under the recorded 180-second bound
  • Focused evidence includes 101 exact rc3 diagnostic-message/location comparisons, SQLite contracts, bounded XML accounting cases, callback pointer/lifetime/exception tests, and sparse pickle memo behavior

The source audit covers 554 changed test paths in 183 families (182 under Lib/test and IDLE). Across those complete families, 1,056 source/fixture paths were reviewed: 103 families are byte-exact rc3, 80 have reviewed native adaptations or retained extras, and no old canonical test body or missing canonical definition remains within that audited scope. This does not claim every untouched CPython test family has been migrated or executed.

Remaining limits

The 13 failing families are classified from the recorded AF_UNIX creation/interface-enumeration EPERM and unmapped UID/GID operations returning EINVAL, including resulting worker-startup failures. Some secondary cleanup errors, particularly logging’s NoneType.close, are diagnosed from the failed setup path and sibling permission error; those blocked behaviors have not been validated on an unrestricted host. No security settings were changed and no new source skips mask these results. The twelve wholly skipped families require unavailable platform/build capabilities; network/curses resource denials remain explicit. Socket's localized-DNS case, traceback's grapheme case, and remote-process profiling selections remain unverified.

Existing expected failures still cover unsupported runtime details. XML supports real preparse amplification limits, but live callback-time changes and external-child parsing remain unsupported; error positions/feed timing are not full Expat parity. The exercised callable-pointer ownership and ordinary/null typed-pointer callback contracts pass, and callbacks reattach an existing VM. General POINTER(py_object) roundtrips still lose ownership provenance through native callbacks; resolving that requires an explicit object-interop design. Automatic foreign-thread/cross-interpreter attachment also remains incomplete. The optional _testcapi implements the exercised genuine fixture subset, not all CPython testing APIs. Other platforms, raw mmap protection flags, and inherited edge discrepancies remain unverified/incomplete.

The automated library-upgrade workflow retains inherited 3.14 source/generated-pin drift. The separate inherited dependency-comment workflow also exceeds Linux's per-environment-string limit: 157,859-byte INPUT_MESSAGE versus 131,072 bytes. The bounded file-backed remedy is published separately in #8958. pull_request_target must receive it on trusted main before this PR benefits; its unchanged inherited Zizmor trigger finding remains disclosed there. CI requiring maintainer approval is not bypassed. No merge is performed.

Reproduction: cargo build --locked --release --features sqlite,capi; workspace tests/Clippy excluding wasm, venvlauncher and capi; separate tests/Clippy from crates/capi; release -m test -v with per-family 300-second bounds and the recorded resource selections. Snippets use exact-rc3-hosted pytest -v --timeout=180 with RUSTPYTHON set to the release. The full matrix, commands, binary/source hashes, initial failures and recovery are preserved separately.

This consolidates #8942, #8945, #8946, #8947, #8948, #8950 and #8951, which remain intact. AI assistance: implementation, testing, analysis and this description were prepared with Codex at the user's direction. Exact model metadata is unavailable; commits disclose Assisted-by: Codex:model-version-unavailable.

@coderabbitai

coderabbitai Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't add duplicated tests from Lib/test

@youknowdot youknowdot changed the title Prepare Python 3.15 with the rc2 standard library and runtime prerequisites Prepare Python 3.15 with rc3 libraries and native prerequisites Oct 4, 2026
@youknowdot
youknowdot force-pushed the cpython-315-mega-integration branch from fb0f7b6 to 85d2fb9 Compare October 4, 2026 12:26
@youknowdot
youknowdot force-pushed the cpython-315-mega-integration branch 2 times, most recently from 1aba241 to 16a2ef9 Compare October 5, 2026 01:42
@codspeed

codspeed Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Merging this PR will improve performance by ×6.3

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 2 improved benchmarks
✅ 60 untouched benchmarks
⏩ 4 skipped benchmarks1

Performance Changes

Benchmark BASE HEAD Efficiency
⚡ base64.py[rustpython] 742.7 ms 31.7 ms ×23
⚡ rustpython[20000] 14.5 ms 8.7 ms +68.05%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing youknowdot:cpython-315-mega-integration (70e0699) with main (3e0e401)

Open in CodSpeed

Footnotes

  1. 4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@youknowdot
youknowdot force-pushed the cpython-315-mega-integration branch 5 times, most recently from ae099f7 to 06d106b Compare October 8, 2026 01:02
youknowdot and others added 14 commits October 8, 2026 18:16
Add unique sentinel values with copy/pickle identity, caller module capture, union support, and cycle collection. Compare constructor behavior and six upstream tests with CPython v3.15.0rc2; retain the existing Python target and bundled libraries.

Assisted-by: Codex:model-version-unavailable
Expose _imp._set_lazy_attributes(modobj, name, /), used by CPython 3.15 importlib after eager loading. Match its empty pending-lazy-module registry behavior, including name type and string-subclass hashing checks. This does not implement lazy imports or change the target version or bundled libraries.

Assisted-by: Codex:model-version-unavailable
Add Python 3.15 compile(module=...) warning filtering. Move format-spec diagnostics to a fallible, literal-aware AST boundary; preserve nested literal warnings while excluding raw text and comments, map function-type fragments to source locations, and suppress duplicate type-comment emissions with per-call tracking.

Keep unrelated legacy warning-detection and exact diagnostic metadata gaps documented by differential probes. Add focused flag, source-location, exception, reentrancy, filtering and literal-boundary regression coverage.

Assisted-by: Codex:model-version-unavailable
Target Python 3.15.0alpha and import site, pkgutil, importlib and supporting test libraries from CPython v3.15.0rc2 (435c9e5a798c99653e3ab64ce29baed0e4f3dfee). Keep native atomic module-cache movement at the surviving importlib reorder sites and use the canonical builtin import callback. Preserve explicit temporary eager-import and readonly-table adaptations while their full PEP implementations remain separate. This starts the migration and does not claim complete 3.15 bytecode or runtime compatibility.

Assisted-by: Codex:model-version-unavailable
Import pinned rc2 test_import and its fixtures, preserving RustPython markers. The upstream tests remove obsolete module.__cached__ expectations. Match create_builtin validation for missing, non-string, empty, non-ASCII and unknown module names; unknown names must not succeed just because an unrelated module is cached. Add a CPython differential regression.

Assisted-by: Codex:model-version-unavailable
Use the PyStr reference directly after validating the ASCII name, fixing the compile error from the previous PyUtf8Str-only accessor.

Assisted-by: Codex:model-version-unavailable
Make the remaining compatibility work explicit in the project overview rather than implying that the initial target change completes support.

Assisted-by: Codex:model-version-unavailable
Import venv, its script assets and matching tests from CPython 435c9e5a798c99653e3ab64ce29baed0e4f3dfee. Preserve existing RustPython expected-failure markers and unchanged PowerShell line endings. Native comparison of unmodified rc2 tests reproduces the four known layout/zip-path differences plus the sandbox AF_UNIX limitation; offline pip bootstrap passes.

Assisted-by: Codex:model-version-unavailable
Add dictionary-owned frozen mapping storage, shared views and cached hashing, read-only execution globals, type namespace support, and stable dictionary C-API coverage. Add differential regressions against CPython 3.15rc2.

Assisted-by: Codex:model-version-unavailable
Remove the temporary MappingProxyType alias now that PEP 814 is integrated into this branch.

Assisted-by: Codex:model-version-unavailable
Sync enum and its regression tests from CPython v3.15.0rc2 (435c9e5a798c99653e3ab64ce29baed0e4f3dfee). Preserve the two existing RustPython expected-failure decorators. Keep this first migration slice limited to enum without changing the declared Python target or native runtime.

AI assistance: Codex/OpenAI reconstructed the upstream sync, reviewed the diff, and ran validation. The runtime did not expose the exact model version.

Assisted-by: Codex:model-version-unavailable
Resolve real symlink identity before generic home/python, retain explicit base-executable metadata for copied environments, and bound legacy creator traversal. This prevents copied aliases from producing dangling nested interpreters or silently selecting an unrelated CPython executable. The POSIX regression covers all copy/symlink combinations and deletion of the creator environment.

Assisted-by: Codex:model-version-unavailable
Consolidate the staged native prerequisites and pinned library sources into one migration branch. Preserve RustPython adaptations and transferred test markers. This is a local integration checkpoint before cold combined verification; feature and platform gaps remain recorded in the migration evidence.

Assisted-by: Codex:model-version-unavailable
Assisted-by: Codex:model-version-unavailable
Align shared opcode numbers with the pinned reference and reject future
metadata drift while preserving native-only instruction behavior. Correct
sleep conversion, code-name string subclasses, mapping-call protocols,
monitoring disable/reraise events, and raw mmap descriptor ownership.

Add structured unpacking diagnostics in the existing vendored parser with
its local patch and upstream provenance recorded. Keep the development
corpus external and preserve all canonical and existing native assertions.

Assisted-by: Codex:model-version-unavailable
Preserve function qualnames and callable side-effect ordering, repair ctypes
array-to-pointer metadata, and expose native numeric/code-location fixtures.
Share signed timestamp conversions with sleep and match rc3 wrapper conversion
and error order. Accept full starred generator operands with upstream-compatible
diagnostics, and refresh the complete pythoninfo helper from the pinned rc3.

Canonical assertions are unchanged; parser provenance retains the local patch.

Assisted-by: Codex:model-version-unavailable
Move the three retained string-subclass hash cases into the existing module snippet. Preserve their assertions and every pre-existing stdlib_imp check.

Assisted-by: Codex:model-version-unavailable
Remove the RustPython-only reorder dispatcher and restore the exact rc3 bootstrap. Retain the atomic dictionary primitive and its direct regression assertions, without claiming it fixes import-cache ordering without a caller.

Assisted-by: Codex:model-version-unavailable
Handle signal wakeup errors and interrupted opens, align signed filesystem
 timestamps and Windows platform exports, and provide native call, buffer,
audit, and thread fixtures for the implemented runtime contracts.
Preserve TLS session resumption while serializing initial signature failures
as a single fatal alert. Keep all canonical test families in CI, running
inspect and monitoring in their explicitly approved C API cohort.

Remaining optional internal C API gaps and restricted compatibility work are
not covered by this change; this does not claim a green full 3.15 aggregate.

Assisted-by: Codex:model-version-unavailable
Assisted-by: Codex:model-version-unavailable
Assisted-by: Codex:model-version-unavailable
Preserve stored hashes, frozen mapping behavior, atomic lazy replacement, and separate contains/set operations after rebasing the merged dictionary hashing changes.

Assisted-by: Codex:model-version-unavailable
The existing canonical assertion now passes after the descriptor flags are aligned. Preserve the remaining multiline-super expected failure.

Assisted-by: Codex:model-version-unavailable
The canonical frame C API case uses PyEval_GetFrameLocals. Move the whole family to the existing C API build alongside inspect and monitoring, preserving all422 families and failure propagation without changing tests.

Assisted-by: Codex:model-version-unavailable
Keep both canonical test bodies and all existing decorators unchanged. The two new expected-failure predicates require a RustPython Windows source build and identify the CPython buildbot or header layout they exercise.

Assisted-by: Codex:model-version-unavailable
Keep the existing nonnegative signed-64-bit FILETIME range while satisfying Windows Clippy's checked_conversions lint. The predicate preserves the same interval values, error path and low/high packing.

Assisted-by: Codex:model-version-unavailable
Explicitly include PyFrame_GetLocals as a non-Stable ABI compatibility exception. Preserve the Stable ABI inventory, source/compiled checks, existing implementation and canonical tests; the generated Windows export table gains exactly this one function.

Assisted-by: Codex:model-version-unavailable
Adopt main's parser-owned diagnostics and updated AST interfaces. Refresh the
source-only vendor from the published 0.16.10 crate, drop the two redundant
upstream backports, and retain reproducible rc3 parser patches with explicit
provenance. Keep module-aware warnings, incomplete-input behavior, and native
range conventions consistent across bytecode, AST, and symtable compilation.

The selected canonical suites, 1,425 core differential records, workspace
Rust tests, and C-API tests pass. Full-run platform and held-API limitations
remain recorded separately; this does not claim full 3.15 compatibility.
Canonical assertions and skip/xfail markers are unchanged.

AI assistance covered implementation, conflict resolution, review, and automated
validation under the user's direction. No human validation is claimed.

Assisted-by: Codex:model-version-unavailable
Keep the warning flag available to the existing cross-platform set_wakeup_fd
setter after reconciling upstream WASI signal support. WASI retains its
synchronous notification path; Unix and Windows retain queued error reporting.
The CI-feature WASI cargo check passes without changing canonical tests.

AI assistance covered conflict resolution, source review, and automated checks
under the user's direction. No human validation is claimed.

Assisted-by: Codex:model-version-unavailable
Represent default, ignored, and custom WASI handlers separately at the host
boundary. wasi-libc uses function pointers for SIG_IGN and SIG_ERR because
integer table index 1 can be a real callback. Preserve Python's disposition
values while translating only tagged dispositions and recognizing failed
installations before updating interpreter state.

This fixes an inherited upstream WASI mismatch exposed during the rebase:
ignored raises trapped and uncatchable signal registration appeared to succeed.
Direct ABI probes pass for custom delivery, ignore, error detection, and restore.
Unix/Windows behavior and canonical test assertions remain unchanged.

AI assistance covered implementation, source review, and automated validation
under the user's direction. No human validation is claimed.

Assisted-by: Codex:model-version-unavailable
Main's recursion guard now concatenates a caller-provided suffix so an empty
suffix no longer leaves trailing whitespace. Preserve the two lazy-import
guards' existing diagnostic text by supplying the required leading space.

The rebase preserves incoming __index__, signal-disposition, Windows host_env,
and recursion fixes. Existing canonical assertions and markers are unchanged.
All eight affected Python test modules pass, alongside workspace/C-API tests
and both Clippy cohorts. Full snippets retain the documented sandbox failures.

AI assistance covered source review, integration and automated validation
under the user's direction. No human validation is claimed.

Assisted-by: Codex:model-version-unavailable
Run native_scope_and_disable alone in a child test process. Its global
PY_START callback otherwise observes Python code executed concurrently by
other C-API tests sharing the main interpreter, contaminating its synthetic
native-event records. Preserve the original test body and every assertion;
propagate child failure through the parent wrapper.

The full C-API aggregate passes with eight test threads, followed by twenty
complete repetitions with sixteen threads. Each run confirms both the child
and parent test executed. Workspace tests, Clippy and affected canonical
Python suites also pass; known sandbox/held-API limitations remain separate.

AI assistance covered the implementation, source review and automated checks
under the user's direction. No human validation is claimed.

Assisted-by: Codex:model-version-unavailable
Give each Context, including copies, a non-reused cache identity instead
of using its heap address. Disable cache hits if the identity counter
saturates. This prevents an empty Context from reading a destroyed
Context's cached value while preserving existing locking and drop order.

Assisted-by: Codex:model-version-unavailable
Preserve the existing MAP_ANONYMOUS detection after the upstream flags binding rename.

Assisted-by: Codex:model-version-unavailable
Apply the existing test optimization level to rustpython-vm in dev builds. Unoptimized opcode dispatch reserves over 120 KiB per native frame and can exhaust normal stacks during stdlib imports, terminating multiprocessing manager accepter threads in coverage runs.

Retain debug assertions, native recursion guards, thread stack settings and all tests. Other workspace crates keep their existing dev profile.

Assisted-by: Codex:model-version-unavailable
Upstream build_staticmethod now returns PyNativeFunction directly. Set the existing builtins module metadata on that function rather than accessing the removed PyNativeMethod.func payload. Keep fixture call behavior and canonical tests unchanged.

Assisted-by: Codex:model-version-unavailable
Run the unchanged canonical manager XML-RPC case in an independent strict instrumented job. Retain the full coverage collection and use regrtest's normal 600-second timeout, with 30 minutes for the cold build and test. This makes the former accepter-thread stall visible without changing assertions or skip conditions.

Assisted-by: Codex:model-version-unavailable
@youknowdot
youknowdot force-pushed the cpython-315-mega-integration branch from 0d7489c to e14427a Compare October 8, 2026 10:08
Close raw descriptor numbers through the host CRT layer rather than constructing OwnedFd values for possibly closed descriptors. Preserve the half-open range, ignored close errors and Windows invalid-parameter handling.

Assisted-by: Codex:model-version-unavailable
Python os._exit, failed child exec and fatal/watchdog exits must bypass C exit handlers. Call the host CRT _exit on Unix, Windows and WASI while retaining the existing non-host fallback. Normal interpreter termination is unchanged.

Assisted-by: Codex:model-version-unavailable
Return EBADF for negative ordinary descriptors before calling rustix, avoiding debug-build assertions on canonical invalid-fd tests. Keep directory-fd sentinel and absolute-path handling unchanged.

Assisted-by: Codex:model-version-unavailable
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants