Visitar URL original
Support compile(module=...) across AST warning paths by youknowdot · Pull Request #8947 · RustPython/RustPython · GitHub
Skip to content

Support compile(module=...) across AST warning paths - #8947

Open
youknowdot wants to merge 2 commits into
RustPython:mainfrom
youknowdot:cpython-315-compile-module
Open

youknowdot wants to merge 2 commits into
RustPython:mainfrom
youknowdot:cpython-315-compile-module

Conversation

@youknowdot

@youknowdot youknowdot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add Python 3.15's keyword-only compile(module=...) argument for SyntaxWarning filtering, including AST-only compilation and type comments. This is an independent native prerequisite for the 3.15 importlib/bootstrap migration; the advertised Python version and bundled libraries remain unchanged.

  • Validate module as str or None, keep diagnostic filenames separate, and preserve the existing Rust compile APIs through wrappers
  • Route existing AST format-spec diagnostics through a fallible, explicit-context boundary over the validated AST before preprocessing
  • Inspect literal spans so raw strings and comments do not become false SyntaxErrors, while nested nonraw warnings and distinct literals survive optimization
  • Map function-type argument/return fragments back to original source locations
  • Suppress duplicate TYPE_COMMENTS escape emissions using per-invocation offset tracking; warning callbacks can re-enter compile without sharing state
  • Preserve arbitrary warning-handler exceptions; convert escalated SyntaxWarning through the existing SyntaxError formatter

The implementation does not run a new whole-source warning scan before every AST parse or keep mutable warning context in VM-global/thread-local state.

Reference: CPython v3.15.0rc2, commit 435c9e5a798c99653e3ab64ce29baed0e4f3dfee. Its importlib source loader supplies this module context when compiling source.

Validation

Independent branch at a86db2711a26ece3781f39fea7b48c042f59bb3f:

  • Release build with SQLite enabled, source-isolation checks, and the expanded native regression snippet pass
  • The same snippet passes exact CPython 3.15.0rc2; its older-CPython comparison uses the existing version guard while RustPython always executes every assertion
  • -m test test_compile test_ast test_warnings test_site test_support test_importlib test_pkgutil --timeout 180: all seven suites successful, 2,156 run and 152 skipped
  • Independent 540-case differential corpus completed. Final membership cleanup produced byte-identical outputs to the reviewed candidate. 485 records match CPython exactly; 48 retain baseline detection/parser/traversal limitations and seven have the diagnostic differences below. All 180 optimization/folding controls match
  • Full bounded snippet aggregate: 469 passed, 13 failed. The failures are the unchanged executor/stress set: ten AF_UNIX socket permission failures, two chown errno differences, and one CPython 3.14 GC/import timeout. No new snippet failures
  • Workspace Rust tests: 1,345 passed, 18 ignored, no failures or warnings
  • Separate C-API tests: 115 passed, 4 ignored; workspace and separate C-API all-targets Clippy both passed without warnings
  • Normal configured commit hooks and diff checks pass

Remaining scope limits

This does not claim full compiler-warning parity. Existing missing warnings for ordinary AST-only literals and some ordinary-source/raw/template paths, and unrelated parser/traversal ordering, remain outside this change.

The shared diagnostic formatter also retains precise differences from CPython: three nested-template error cases point at the actual backslash rather than CPython's shifted position; three function-type cases append a newline to SyntaxError.text; one existing codegen case lacks end fields and differs in source text. These do not change module filtering, exception identity/type, valid-source acceptance, or duplicate suppression. They are recorded rather than hidden behind skips or CPython-specific caret adjustments.

AI assistance

OpenAI Codex assisted with implementation, CPython comparison, tests, review, and this description under maintainer direction. The runtime did not expose the exact model version. Results are automated Linux x86_64 verification; no additional human-use or untested-platform validation is claimed.

Summary by CodeRabbit

  • New Features
    • Added an optional keyword-only module argument to compile(). Its value is included in compiler warning messages, while source filenames remain unchanged.
    • Invalid escape warnings in format specifications are now reported during parsing.
  • Bug Fixes
    • Prevented duplicate invalid escape warnings across parsing and compilation.
    • Invalid values for module now raise a TypeError.

@coderabbitai

coderabbitai Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: RustPython/RustPython/.coderabbit.yml
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 7de6c703-fe73-4422-b64a-ed279876f597
📥 Commits

Reviewing files that changed from the base of the PR and between 1e72b22 and 3cc5388.

📒 Files selected for processing (5)
  • crates/vm/src/stdlib/_ast.rs
  • crates/vm/src/stdlib/_ast/string.rs
  • crates/vm/src/stdlib/builtins.rs
  • crates/vm/src/vm/compile.rs
  • extra_tests/snippets/builtin_compile_module.py

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.


📝 Walkthrough

Walkthrough

The compile() builtin now accepts an optional keyword-only module argument. Compilation passes that value to warning emission. AST parsing also collects format-spec escape warnings through callbacks and tracks their source offsets.

Changes

Compile module argument and warning propagation

Layer / File(s) Summary
Compile module argument and warning propagation
crates/vm/src/stdlib/builtins.rs, crates/vm/src/stdlib/_ast.rs, crates/vm/src/vm/compile.rs, extra_tests/snippets/builtin_compile_module.py
compile() accepts an optional keyword-only module value and rejects non-strings. The compiler passes the module name to warning emission. Tests cover module values, warning matching, and diagnostic locations.

AST format-spec warning collection

Layer / File(s) Summary
AST format-spec warning collection
crates/vm/src/stdlib/_ast.rs, crates/vm/src/stdlib/_ast/string.rs, crates/vm/src/vm/compile.rs, extra_tests/snippets/builtin_compile_module.py
AST parsing emits format-spec warnings through a fallible callback. The scanner reports invalid escapes by source offset, skips raw literals, and deduplicates emissions. Function-type parsing maps warning offsets to the original source. Tests cover AST parsing, function-type locations, duplicate diagnostics, and warning-handler errors.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant PythonCaller
  participant BuiltinsCompile
  participant Compiler
  participant WarningSystem
  PythonCaller->>BuiltinsCompile: Call compile with optional module
  BuiltinsCompile->>Compiler: Pass source and validated module
  Compiler->>WarningSystem: Emit warning with module and source location
Loading

Merge Risk: ⚪ Minimal · up to 3cc53

This change adds Python 3.15's keyword-only module argument to compile() and routes it into warning filtering. The changes include focused tests, and existing APIs are preserved. No concrete correctness or stability risk remains open, so the change appears ready to merge.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 36.36% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 44 functions across 5 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely identifies the main change: adding support for compile(module=...) across AST warning paths.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

youknowdot and others added 2 commits October 8, 2026 11:14
Add Python 3.15 compile(module=...) warning filtering. Move format-spec diagnostics to a fallible, literal-aware AST boundary; preserve nested literal warnings while excluding raw text and comments, map function-type fragments to source locations, and suppress duplicate type-comment emissions with per-call tracking.

Keep unrelated legacy warning-detection and exact diagnostic metadata gaps documented by differential probes. Add focused flag, source-location, exception, reentrancy, filtering and literal-boundary regression coverage.

Assisted-by: Codex:model-version-unavailable
Convert the single-mode multiple statements error into the generic
parse error type, and iterate f-string parts through FStringPartRef.

Assisted-by: Claude Code:claude-opus-5-5
@youknowone
youknowone force-pushed the cpython-315-compile-module branch from a86db27 to 3cc5388 Compare October 8, 2026 02:30
@youknowone
youknowone marked this pull request as ready for review October 8, 2026 02:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants