Visitar URL original
Implement _tokenize and update tokenize from v3.14.3 by youknowone · Pull Request #7392 · RustPython/RustPython · GitHub
Skip to content

Implement _tokenize and update tokenize from v3.14.3 #7392

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
youknowone merged 3 commits into from
Mar 10, 2026
Merged

Implement _tokenize and update tokenize from v3.14.3 #7392

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5c5aff8
Base implementation of _tokenize module
ShaharNaveh Mar 6, 2026
bf2b993
Update tokenize from v3.14.3
Mar 6, 2026
55737ed
Rewrite _tokenize with 2-phase model
youknowone Mar 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Base implementation of _tokenize module 
Port from PR #6240 by ShaharNaveh, adapted to current codebase.
Uses ruff_python_parser for tokenization via TokenizerIter.
  • Loading branch information
@ShaharNaveh @youknowone
ShaharNaveh authored and youknowone committed Mar 10, 2026
commit 5c5aff8fd9729050681697fc81a1c721b2199d92
4 changes: 4 additions & 0 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 5 additions & 0 deletions crates/stdlib/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,11 @@ rustpython-derive = { workspace = true }
rustpython-vm = { workspace = true, default-features = false, features = ["compiler"]}
rustpython-common = { workspace = true }

ruff_python_parser = { workspace = true }
ruff_python_ast = { workspace = true }
ruff_text_size = { workspace = true }
ruff_source_file = { workspace = true }

ahash = { workspace = true }
ascii = { workspace = true }
cfg-if = { workspace = true }
Expand Down
2 changes: 2 additions & 0 deletions crates/stdlib/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ mod pystruct;
mod random;
mod statistics;
mod suggestions;
mod tokenize;
// TODO: maybe make this an extension module, if we ever get those
// mod re;
#[cfg(all(feature = "host_env", not(target_arch = "wasm32")))]
Expand Down Expand Up @@ -225,6 +226,7 @@ pub fn stdlib_module_defs(ctx: &Context) -> Vec<&'static builtins::PyModuleDef>
ssl::module_def(ctx),
statistics::module_def(ctx),
suggestions::module_def(ctx),
tokenize::module_def(ctx),
#[cfg(all(feature = "host_env", unix, not(target_os = "redox")))]
syslog::module_def(ctx),
#[cfg(all(
Expand Down
Loading