Visitar URL original
Per-interpreter runtime state and a process-wide GC stop-the-world by youknowone · Pull Request #8517 · RustPython/RustPython · GitHub
Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
5c341d2
vm: add per-interpreter runtime state and interpreter registry
youknowone Aug 13, 2026
43f4b23
gc: stop every interpreter while collecting
youknowone Aug 13, 2026
ffd3d1e
vm: scope the interpreter registry per thread without threading
youknowone Aug 13, 2026
0079994
vm: keep the interpreter registry usable across bootstrap, drop and fork
youknowone Aug 13, 2026
8f108ff
vm: attach and detach thread slots when switching interpreters
youknowone Aug 13, 2026
696a8c1
vm: list only the current interpreter's subclasses
youknowone Aug 14, 2026
562b657
gc: give each interpreter its own collector state
youknowone Aug 14, 2026
30e2020
_queue, _thread: detach before taking locks held across waits
youknowone Aug 14, 2026
5ce062e
_io, _winapi: detach on the remaining stopped-holdable lock takes
youknowone Aug 14, 2026
87b6c68
gc: size the interpreter owner tag to the header padding
youknowone Aug 15, 2026
fd27902
vm: reuse cleared frame blocks and shorten interpreter hot paths
youknowone Aug 15, 2026
c3fcc70
gc, vm: address review notes on owner tags and interpreter docs
youknowone Aug 15, 2026
57428cd
vm: gate interpreter registration on stop-the-world admission
youknowone Aug 15, 2026
8d07831
gc: keep the tracking counters off the barrier path
youknowone Aug 15, 2026
4e518d3
dict: settle lookups and iteration steps under one read guard
youknowone Aug 15, 2026
eb94daf
vm: build a call's argument vector once
youknowone Aug 15, 2026
f70db35
vm: reach an instance dict without cloning it
youknowone Aug 15, 2026
4a76424
vm: shorten the per-instruction safepoint and the call preamble
youknowone Aug 15, 2026
b0f0dcc
vm: give KwArgs a zero-sized hasher
youknowone Aug 15, 2026
f9bbc6d
vm: promote borrowed stack refs before a yield
youknowone Aug 15, 2026
b4bb43c
vm: run the exact-args vectorcall on a heap frame again
youknowone Aug 15, 2026
98d2410
vm: release a tail call's function at the callee's return
youknowone Aug 16, 2026
08bd522
vm: stop copying a running frame's locals when it materializes
youknowone Aug 16, 2026
c7c1452
vm: key data stack frame reuse on the exact frame size
youknowone Aug 16, 2026
2d58fc8
vm: store the frame object payload address in the cross-thread slot
youknowone Aug 16, 2026
e24d831
vm: copy a foreign thread's frames instead of linking them
youknowone Aug 16, 2026
5d0d9be
vm: resolve the two rustdoc links this branch added
youknowone Aug 16, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
gc, vm: address review notes on owner tags and interpreter docs
Look up retired owner tags with a sorted binary search instead of a linear
scan, which every scanned object paid for once per dropped interpreter.

Drop the claim that clearing a tag frees it for reuse; `alloc_owner` only
ever hands out new tags. Also correct the tag space it mentions, which is
16-bit since the tag was sized to the header padding.

Document `is_main` as "top-level interpreter" rather than "the process
main": every top-level interpreter sets it, and only the first registered
one becomes the main `main_interpreter_id` reports.

Assert membership rather than an exact owned-interpreter count delta; the
owned table is process-global and other tests store into it in parallel.

Assisted-by: Claude
  • Loading branch information
youknowone committed Aug 15, 2026
commit c3fcc702e09d474abfdc10b56711114a41066a7d
24 changes: 18 additions & 6 deletions crates/vm/src/gc_state.rs
Original file line number Diff line number Diff line change
Expand Up @@ -306,7 +306,7 @@ impl GcState {
}

/// Reserve a tag for a new interpreter. Tags are never reused; exhausting
/// the 32-bit space falls back to `GC_NO_OWNER`, which costs isolation but
/// the tag space falls back to `GC_NO_OWNER`, which costs isolation but
/// stays correct, rather than aliasing a live interpreter.
fn alloc_owner(&self) -> GcOwner {
self.next_owner
Expand Down Expand Up @@ -536,11 +536,18 @@ impl GcState {
// Another interpreter's objects stay out of the candidate set, so they
// act as external roots: anything they reference survives this pass.
let owner = gc.owner;
let retired = self.retired.lock().clone();
// Sorted so that the test below, which every scanned object pays for,
// stays logarithmic in the number of interpreters that have been
// dropped instead of linear.
let retired = {
let mut retired = self.retired.lock().clone();
retired.sort_unstable();
retired
};
let mut collecting: HashSet<GcPtr> = HashSet::new();
for gen_list in &gen_locks {
for obj in gen_list.iter() {
if retired.contains(&obj.gc_owner()) {
if retired.binary_search(&obj.gc_owner()).is_ok() {
obj.set_gc_owner(GC_NO_OWNER);
}
if obj.strong_count() > 0 && is_owned_by(obj, owner) {
Expand All @@ -553,11 +560,15 @@ impl GcState {
// is where adoption finishes and the tags stop being tracked.
if generation == 2 && !retired.is_empty() {
for obj in self.permanent_list.read().iter() {
if retired.contains(&obj.gc_owner()) {
if retired.binary_search(&obj.gc_owner()).is_ok() {
obj.set_gc_owner(GC_NO_OWNER);
}
}
self.retired.lock().retain(|tag| !retired.contains(tag));
// Only the tags this scan saw: one retired while it ran still has
// objects nobody has adopted.
self.retired
.lock()
.retain(|tag| retired.binary_search(tag).is_err());
}

if collecting.is_empty() {
Expand Down Expand Up @@ -1281,7 +1292,8 @@ impl Drop for GcInterpreterState {
fn drop(&mut self) {
// Objects this interpreter tracked can outlive it (another interpreter
// may still hold one). Clearing the tag hands them to every collection
// instead of stranding them, and frees the tag for reuse.
// instead of stranding them. The tag itself is not handed back: it stays
// retired so that a later interpreter cannot inherit these objects.
gc_state().retire_owner(self.owner);
}
}
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Expand Down
10 changes: 7 additions & 3 deletions crates/vm/src/vm/interpreter.rs
Original file line number Diff line number Diff line change
Expand Up @@ -449,7 +449,10 @@ impl Interpreter {
self.global_state.whence
}

/// Whether this is the process main interpreter.
/// Whether this is a top-level interpreter rather than a subinterpreter.
///
/// Every top-level interpreter answers `true`; for *the* process main, use
/// [`Interpreter::is_process_main`].
#[inline]
#[must_use]
pub fn is_main(&self) -> bool {
Expand Down Expand Up @@ -1358,11 +1361,12 @@ mod tests {
let sub = main.create_subinterpreter();
let id = sub.id();

let before = runtime::owned_interpreter_count();
assert_eq!(runtime::store_owned_interpreter(sub), id);
assert!(runtime::is_owned_interpreter(id));
assert!(runtime::lookup_interpreter(id).is_some());
assert_eq!(runtime::owned_interpreter_count(), before + 1);
// The owned table is process-global and other tests store into it in
// parallel, so only this entry's own membership is deterministic.
assert!(runtime::owned_interpreter_count() >= 1);

// Reclaiming removes ownership but keeps the interpreter alive while the
// returned handle is held.
Expand Down
4 changes: 3 additions & 1 deletion crates/vm/src/vm/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -749,7 +749,9 @@ pub struct PyGlobalState {
pub interpreter_id: i64,
/// How this interpreter was created.
pub whence: runtime::InterpreterWhence,
/// True only for the process main interpreter.
/// True for every top-level (non-sub) interpreter, each of which keeps its
/// own signal and main-thread bookkeeping. Only the first one registered
/// becomes *the* process main — see [`runtime::main_interpreter_id`].
pub is_main: bool,
Comment thread
coderabbitai[bot] marked this conversation as resolved.
pub config: PyConfig,
pub module_defs: BTreeMap<&'static str, &'static builtins::PyModuleDef>,
Expand Down