Repository navigation
Prepare Python 3.15 with rc3 libraries and native prerequisites - #8954
youknowdot wants to merge 71 commits into
Conversation
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
don't add duplicated tests from Lib/test
fb0f7b6 to
85d2fb9
Compare
1aba241 to
16a2ef9
Compare
Merging this PR will improve performance by ×6.3
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ⚡ | base64.py[rustpython] |
742.7 ms | 31.7 ms | ×23 |
| ⚡ | rustpython[20000] |
14.5 ms | 8.7 ms | +68.05% |
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing youknowdot:cpython-315-mega-integration (70e0699) with main (3e0e401)
Footnotes
-
4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
ae099f7 to
06d106b
Compare
Add unique sentinel values with copy/pickle identity, caller module capture, union support, and cycle collection. Compare constructor behavior and six upstream tests with CPython v3.15.0rc2; retain the existing Python target and bundled libraries. Assisted-by: Codex:model-version-unavailable
Expose _imp._set_lazy_attributes(modobj, name, /), used by CPython 3.15 importlib after eager loading. Match its empty pending-lazy-module registry behavior, including name type and string-subclass hashing checks. This does not implement lazy imports or change the target version or bundled libraries. Assisted-by: Codex:model-version-unavailable
Add Python 3.15 compile(module=...) warning filtering. Move format-spec diagnostics to a fallible, literal-aware AST boundary; preserve nested literal warnings while excluding raw text and comments, map function-type fragments to source locations, and suppress duplicate type-comment emissions with per-call tracking. Keep unrelated legacy warning-detection and exact diagnostic metadata gaps documented by differential probes. Add focused flag, source-location, exception, reentrancy, filtering and literal-boundary regression coverage. Assisted-by: Codex:model-version-unavailable
Target Python 3.15.0alpha and import site, pkgutil, importlib and supporting test libraries from CPython v3.15.0rc2 (435c9e5a798c99653e3ab64ce29baed0e4f3dfee). Keep native atomic module-cache movement at the surviving importlib reorder sites and use the canonical builtin import callback. Preserve explicit temporary eager-import and readonly-table adaptations while their full PEP implementations remain separate. This starts the migration and does not claim complete 3.15 bytecode or runtime compatibility. Assisted-by: Codex:model-version-unavailable
Import pinned rc2 test_import and its fixtures, preserving RustPython markers. The upstream tests remove obsolete module.__cached__ expectations. Match create_builtin validation for missing, non-string, empty, non-ASCII and unknown module names; unknown names must not succeed just because an unrelated module is cached. Add a CPython differential regression. Assisted-by: Codex:model-version-unavailable
Use the PyStr reference directly after validating the ASCII name, fixing the compile error from the previous PyUtf8Str-only accessor. Assisted-by: Codex:model-version-unavailable
Make the remaining compatibility work explicit in the project overview rather than implying that the initial target change completes support. Assisted-by: Codex:model-version-unavailable
Import venv, its script assets and matching tests from CPython 435c9e5a798c99653e3ab64ce29baed0e4f3dfee. Preserve existing RustPython expected-failure markers and unchanged PowerShell line endings. Native comparison of unmodified rc2 tests reproduces the four known layout/zip-path differences plus the sandbox AF_UNIX limitation; offline pip bootstrap passes. Assisted-by: Codex:model-version-unavailable
Add dictionary-owned frozen mapping storage, shared views and cached hashing, read-only execution globals, type namespace support, and stable dictionary C-API coverage. Add differential regressions against CPython 3.15rc2. Assisted-by: Codex:model-version-unavailable
Remove the temporary MappingProxyType alias now that PEP 814 is integrated into this branch. Assisted-by: Codex:model-version-unavailable
Sync enum and its regression tests from CPython v3.15.0rc2 (435c9e5a798c99653e3ab64ce29baed0e4f3dfee). Preserve the two existing RustPython expected-failure decorators. Keep this first migration slice limited to enum without changing the declared Python target or native runtime. AI assistance: Codex/OpenAI reconstructed the upstream sync, reviewed the diff, and ran validation. The runtime did not expose the exact model version. Assisted-by: Codex:model-version-unavailable
Resolve real symlink identity before generic home/python, retain explicit base-executable metadata for copied environments, and bound legacy creator traversal. This prevents copied aliases from producing dangling nested interpreters or silently selecting an unrelated CPython executable. The POSIX regression covers all copy/symlink combinations and deletion of the creator environment. Assisted-by: Codex:model-version-unavailable
Consolidate the staged native prerequisites and pinned library sources into one migration branch. Preserve RustPython adaptations and transferred test markers. This is a local integration checkpoint before cold combined verification; feature and platform gaps remain recorded in the migration evidence. Assisted-by: Codex:model-version-unavailable
Assisted-by: Codex:model-version-unavailable
Align shared opcode numbers with the pinned reference and reject future metadata drift while preserving native-only instruction behavior. Correct sleep conversion, code-name string subclasses, mapping-call protocols, monitoring disable/reraise events, and raw mmap descriptor ownership. Add structured unpacking diagnostics in the existing vendored parser with its local patch and upstream provenance recorded. Keep the development corpus external and preserve all canonical and existing native assertions. Assisted-by: Codex:model-version-unavailable
Preserve function qualnames and callable side-effect ordering, repair ctypes array-to-pointer metadata, and expose native numeric/code-location fixtures. Share signed timestamp conversions with sleep and match rc3 wrapper conversion and error order. Accept full starred generator operands with upstream-compatible diagnostics, and refresh the complete pythoninfo helper from the pinned rc3. Canonical assertions are unchanged; parser provenance retains the local patch. Assisted-by: Codex:model-version-unavailable
Move the three retained string-subclass hash cases into the existing module snippet. Preserve their assertions and every pre-existing stdlib_imp check. Assisted-by: Codex:model-version-unavailable
Remove the RustPython-only reorder dispatcher and restore the exact rc3 bootstrap. Retain the atomic dictionary primitive and its direct regression assertions, without claiming it fixes import-cache ordering without a caller. Assisted-by: Codex:model-version-unavailable
Handle signal wakeup errors and interrupted opens, align signed filesystem timestamps and Windows platform exports, and provide native call, buffer, audit, and thread fixtures for the implemented runtime contracts. Preserve TLS session resumption while serializing initial signature failures as a single fatal alert. Keep all canonical test families in CI, running inspect and monitoring in their explicitly approved C API cohort. Remaining optional internal C API gaps and restricted compatibility work are not covered by this change; this does not claim a green full 3.15 aggregate. Assisted-by: Codex:model-version-unavailable
Assisted-by: Codex:model-version-unavailable
Assisted-by: Codex:model-version-unavailable
Preserve stored hashes, frozen mapping behavior, atomic lazy replacement, and separate contains/set operations after rebasing the merged dictionary hashing changes. Assisted-by: Codex:model-version-unavailable
The existing canonical assertion now passes after the descriptor flags are aligned. Preserve the remaining multiline-super expected failure. Assisted-by: Codex:model-version-unavailable
The canonical frame C API case uses PyEval_GetFrameLocals. Move the whole family to the existing C API build alongside inspect and monitoring, preserving all422 families and failure propagation without changing tests. Assisted-by: Codex:model-version-unavailable
Keep both canonical test bodies and all existing decorators unchanged. The two new expected-failure predicates require a RustPython Windows source build and identify the CPython buildbot or header layout they exercise. Assisted-by: Codex:model-version-unavailable
Keep the existing nonnegative signed-64-bit FILETIME range while satisfying Windows Clippy's checked_conversions lint. The predicate preserves the same interval values, error path and low/high packing. Assisted-by: Codex:model-version-unavailable
Explicitly include PyFrame_GetLocals as a non-Stable ABI compatibility exception. Preserve the Stable ABI inventory, source/compiled checks, existing implementation and canonical tests; the generated Windows export table gains exactly this one function. Assisted-by: Codex:model-version-unavailable
Adopt main's parser-owned diagnostics and updated AST interfaces. Refresh the source-only vendor from the published 0.16.10 crate, drop the two redundant upstream backports, and retain reproducible rc3 parser patches with explicit provenance. Keep module-aware warnings, incomplete-input behavior, and native range conventions consistent across bytecode, AST, and symtable compilation. The selected canonical suites, 1,425 core differential records, workspace Rust tests, and C-API tests pass. Full-run platform and held-API limitations remain recorded separately; this does not claim full 3.15 compatibility. Canonical assertions and skip/xfail markers are unchanged. AI assistance covered implementation, conflict resolution, review, and automated validation under the user's direction. No human validation is claimed. Assisted-by: Codex:model-version-unavailable
Keep the warning flag available to the existing cross-platform set_wakeup_fd setter after reconciling upstream WASI signal support. WASI retains its synchronous notification path; Unix and Windows retain queued error reporting. The CI-feature WASI cargo check passes without changing canonical tests. AI assistance covered conflict resolution, source review, and automated checks under the user's direction. No human validation is claimed. Assisted-by: Codex:model-version-unavailable
Represent default, ignored, and custom WASI handlers separately at the host boundary. wasi-libc uses function pointers for SIG_IGN and SIG_ERR because integer table index 1 can be a real callback. Preserve Python's disposition values while translating only tagged dispositions and recognizing failed installations before updating interpreter state. This fixes an inherited upstream WASI mismatch exposed during the rebase: ignored raises trapped and uncatchable signal registration appeared to succeed. Direct ABI probes pass for custom delivery, ignore, error detection, and restore. Unix/Windows behavior and canonical test assertions remain unchanged. AI assistance covered implementation, source review, and automated validation under the user's direction. No human validation is claimed. Assisted-by: Codex:model-version-unavailable
Main's recursion guard now concatenates a caller-provided suffix so an empty suffix no longer leaves trailing whitespace. Preserve the two lazy-import guards' existing diagnostic text by supplying the required leading space. The rebase preserves incoming __index__, signal-disposition, Windows host_env, and recursion fixes. Existing canonical assertions and markers are unchanged. All eight affected Python test modules pass, alongside workspace/C-API tests and both Clippy cohorts. Full snippets retain the documented sandbox failures. AI assistance covered source review, integration and automated validation under the user's direction. No human validation is claimed. Assisted-by: Codex:model-version-unavailable
Run native_scope_and_disable alone in a child test process. Its global PY_START callback otherwise observes Python code executed concurrently by other C-API tests sharing the main interpreter, contaminating its synthetic native-event records. Preserve the original test body and every assertion; propagate child failure through the parent wrapper. The full C-API aggregate passes with eight test threads, followed by twenty complete repetitions with sixteen threads. Each run confirms both the child and parent test executed. Workspace tests, Clippy and affected canonical Python suites also pass; known sandbox/held-API limitations remain separate. AI assistance covered the implementation, source review and automated checks under the user's direction. No human validation is claimed. Assisted-by: Codex:model-version-unavailable
Give each Context, including copies, a non-reused cache identity instead of using its heap address. Disable cache hits if the identity counter saturates. This prevents an empty Context from reading a destroyed Context's cached value while preserving existing locking and drop order. Assisted-by: Codex:model-version-unavailable
Preserve the existing MAP_ANONYMOUS detection after the upstream flags binding rename. Assisted-by: Codex:model-version-unavailable
Apply the existing test optimization level to rustpython-vm in dev builds. Unoptimized opcode dispatch reserves over 120 KiB per native frame and can exhaust normal stacks during stdlib imports, terminating multiprocessing manager accepter threads in coverage runs. Retain debug assertions, native recursion guards, thread stack settings and all tests. Other workspace crates keep their existing dev profile. Assisted-by: Codex:model-version-unavailable
Upstream build_staticmethod now returns PyNativeFunction directly. Set the existing builtins module metadata on that function rather than accessing the removed PyNativeMethod.func payload. Keep fixture call behavior and canonical tests unchanged. Assisted-by: Codex:model-version-unavailable
Run the unchanged canonical manager XML-RPC case in an independent strict instrumented job. Retain the full coverage collection and use regrtest's normal 600-second timeout, with 30 minutes for the cold build and test. This makes the former accepter-thread stall visible without changing assertions or skip conditions. Assisted-by: Codex:model-version-unavailable
0d7489c to
e14427a
Compare
Close raw descriptor numbers through the host CRT layer rather than constructing OwnedFd values for possibly closed descriptors. Preserve the half-open range, ignored close errors and Windows invalid-parameter handling. Assisted-by: Codex:model-version-unavailable
Python os._exit, failed child exec and fatal/watchdog exits must bypass C exit handlers. Call the host CRT _exit on Unix, Windows and WASI while retaining the existing non-host fallback. Normal interpreter termination is unchanged. Assisted-by: Codex:model-version-unavailable
Return EBADF for negative ordinary descriptors before calling rustix, avoiding debug-build assertions on canonical invalid-fd tests. Keep directory-fd sentinel and absolute-path handling unchanged. Assisted-by: Codex:model-version-unavailable
RustPython's development target moves from 3.14 to 3.15, based on CPython v3.15.0rc3 (
8a8eb0b90d40f0e03f8252e9f73cad156e1caca8). This mega upgrade combines startup/import/venv, the standard library and paired canonical tests, with the runtime prerequisites needed to execute them. Complete 3.15 compatibility remains work in progress.The integration includes sentinel/frozendict, compile(module), atomic import ordering, lazy imports (including the exercised C-API exports), unpacking comprehensions, compiler/JIT/opcode updates, AST and diagnostics, typing, profiling, ctypes/C-API, SSL, SQLite and XML contracts. Full official rc3 test sources replace the previously incomplete test updates; canonical assertions were not rewritten to manufacture successful results. Proven stale native failure markers were removed after execution. Existing Rust diagnostic/opcode snapshots were updated with explicit user approval.
Two narrowly patched dependencies retain their official licenses and source/checksum provenance: the existing Ruff parser 0.16.5 with two official parser backports, and xml 1.3.0 with cumulative entity-expansion accounting. Cargo uses local
[patch.crates-io]overrides without a broad dependency upgrade. The parser vendor adds roughly 1.1 MB of file content (1.7 MB on disk); the XML source adds roughly 300 KB. Their exact patches and removal guidance are included.Added snippets were audited against rc3 behavior: the earlier reduction remains 39→27 files and 253→59 behavior groups, plus one standalone frozendict probe with three methods. A further six duplicate assertions were removed from the atomic-import snippet. Remaining snippets cover distinct native callback/reentry/ownership contracts; development differential probes remain outside the repository. Nine approved legacy snippets, including
_imp.create_builtin, reflect rc3 behavior.Latest rebased head and focused verification
Head
1aba241160f5762365a04c4c9ce9a11a3b0fbb98is based on verified main7536730ed6afb948ffa335fc113a0fcd9af0cdbf. Rebase preserves the migration and its merge topology, including the sentinel/frozendict resolution. Main's C-API recursion, CSV and specialized-list finalizer fixes are retained. The latest validation also exposed and fixed callable-pointer cast destinations and ordinary/null typed-pointer callback decoding, with matching CIF classification.Fresh checks on this head, Linux x86-64, Rust 1.99.0:
sqlite,capirelease build pass without warnings; configured hooks passBasicTest.test_multiprocessingfailing when forkserver creates an AF_UNIX socket (PermissionError: [Errno 1] Operation not permitted); the recursion check and other exercised cases completeOne additional implementation defect remains: marshalling a code object after assigning a lone-surrogate filename can panic. The canonical marshal family does not cover it, and its proposed correction remains held. The successful suites are therefore scoped evidence, not complete 3.15 compatibility. The full 183-family matrix below is explicitly historical, not a fresh run of every family on this head.
Independent Python 3.14 changes
The compatible fixes are published separately for review on the unchanged 3.14 target:
Each code extraction has standalone 3.14 build and test evidence in its PR. The broader typed-callback/
py_objectextraction is held because its canonical dllist path still produces an unraisable error. The cast-only PR excludes that path. These implementations remain in the mega upgrade while the standalone changes are reviewed. Existing atomic-import PR #8948 is reused.Broader pre-rebase validation
Source
fb0f7b67285e8a0aa25f51143952a19098caab10, Linux x86-64, Rust 1.99.0. Release checks usesqlite,capi; default build/startup is also verified with_testcapiabsent. All commits run normal configured hooks.The source audit covers 554 changed test paths in 183 families (182 under Lib/test and IDLE). Across those complete families, 1,056 source/fixture paths were reviewed: 103 families are byte-exact rc3, 80 have reviewed native adaptations or retained extras, and no old canonical test body or missing canonical definition remains within that audited scope. This does not claim every untouched CPython test family has been migrated or executed.
Remaining limits
The 13 failing families are classified from the recorded AF_UNIX creation/interface-enumeration EPERM and unmapped UID/GID operations returning EINVAL, including resulting worker-startup failures. Some secondary cleanup errors, particularly logging’s NoneType.close, are diagnosed from the failed setup path and sibling permission error; those blocked behaviors have not been validated on an unrestricted host. No security settings were changed and no new source skips mask these results. The twelve wholly skipped families require unavailable platform/build capabilities; network/curses resource denials remain explicit. Socket's localized-DNS case, traceback's grapheme case, and remote-process profiling selections remain unverified.
Existing expected failures still cover unsupported runtime details. XML supports real preparse amplification limits, but live callback-time changes and external-child parsing remain unsupported; error positions/feed timing are not full Expat parity. The exercised callable-pointer ownership and ordinary/null typed-pointer callback contracts pass, and callbacks reattach an existing VM. General
POINTER(py_object)roundtrips still lose ownership provenance through native callbacks; resolving that requires an explicit object-interop design. Automatic foreign-thread/cross-interpreter attachment also remains incomplete. The optional_testcapiimplements the exercised genuine fixture subset, not all CPython testing APIs. Other platforms, raw mmap protection flags, and inherited edge discrepancies remain unverified/incomplete.The automated library-upgrade workflow retains inherited 3.14 source/generated-pin drift. The separate inherited dependency-comment workflow also exceeds Linux's per-environment-string limit: 157,859-byte
INPUT_MESSAGEversus 131,072 bytes. The bounded file-backed remedy is published separately in #8958.pull_request_targetmust receive it on trusted main before this PR benefits; its unchanged inherited Zizmor trigger finding remains disclosed there. CI requiring maintainer approval is not bypassed. No merge is performed.Reproduction:
cargo build --locked --release --features sqlite,capi; workspace tests/Clippy excluding wasm, venvlauncher and capi; separate tests/Clippy from crates/capi; release-m test -vwith per-family 300-second bounds and the recorded resource selections. Snippets use exact-rc3-hostedpytest -v --timeout=180with RUSTPYTHON set to the release. The full matrix, commands, binary/source hashes, initial failures and recovery are preserved separately.This consolidates #8942, #8945, #8946, #8947, #8948, #8950 and #8951, which remain intact. AI assistance: implementation, testing, analysis and this description were prepared with Codex at the user's direction. Exact model metadata is unavailable; commits disclose
Assisted-by: Codex:model-version-unavailable.