[subinterpreters] PEP 554 implementation: add interpreters module

@ncoghlan

BPO	32604
Nosy	@ncoghlan, @abalkin, @pitrou, @vstinner, @pmp-p, @ericsnowcurrently, @zware, @zooba, @applio, @emilyemorehouse, @pablogsal, @miss-islington, @nanjekyejoannah
PRs	bpo-32604: Expose the subinterpreters C-API in a "private" stdlib module. #1748 bpo-32604: Add the _xxsubinterpreters extension module under Windows. #5436 bpo-32604: NULL-terminate kwlist in channel_drop_interpreter(). #5437 bpo-32604: Fix memory leaks in the new _xxsubinterpreters module. #5507 [3.7] bpo-32604: Fix memory leaks in the new _xxsubinterpreters module. (GH-5507) #5509 bpo-32604: Make _xxsubinterpreters build on Windows #5516 [3.7] bpo-32604: Make _xxsubinterpreters build on Windows (GH-5516) #5624 bpo-32604: Clean up created subinterpreters before runtime finalization. #5709 [3.7] bpo-32604: Clean up created subinterpreters before runtime finalization. (gh-5709) #5710 bpo-32604: Swap threads only if the interpreter is different. #5778 [3.7] bpo-32604: Swap threads only if the interpreter is different. (gh-5778) #5783 bpo-32604: Remove xid registry. #6813 bpo-32604: Improve subinterpreter tests. #6914 bpo-32604: Implement force-closing channels. #6937 bpo-33615: avoid extra decref #7251 bpo-33615: Temporarily disable a test that is triggering crashes on a few buildbots. #7288 bpo-33724: Use the right format code for int64_t in subinterpreters code. #7330 bpo-32604: [_xxsubinterpreters] Propagate exceptions. #19768 bpo-32604: Add support for a "default" arg in channel_recv(). #19770 bpo-32604: [_xxsubinterpreters] Add channel_send_wait(). #19829 Revert "bpo-32604: [_xxsubinterpreters] Propagate exceptions. (GH-19768)" #20089 gh-76785: Multiple Interpreters in the Stdlib (PEP 554) #18817 bpo-32604: PEP 554 for use in test suite #19985 Revert "bpo-32604: PEP 554 for use in test suite (GH-19985)" #20465 bpo-32604: Fix reference leak in select module #20600 bpo-32604: Recommit "bpo-32604: PEP 554 for use in test suite (GH-19985)" #20611 bpo-32604: Disable failing tests #20777 bpo-32604: Clean up test.support.interpreters. #20926

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ericsnowcurrently'
closed_at = None
created_at = <Date 2018-01-20.01:58:01.942>
labels = ['expert-subinterpreters', 'type-feature', '3.10', 'docs']
title = '[subinterpreters] PEP 554 implementation: add interpreters module'
updated_at = <Date 2020-06-17.00:42:48.323>
user = 'https://github.com/ericsnowcurrently'

bugs.python.org fields:

activity = <Date 2020-06-17.00:42:48.323>
actor = 'eric.snow'
assignee = 'eric.snow'
closed = False
closed_date = None
closer = None
components = ['Documentation', 'Subinterpreters']
creation = <Date 2018-01-20.01:58:01.942>
creator = 'eric.snow'
dependencies = []
files = []
hgrepos = []
issue_num = 32604
keywords = ['patch']
message_count = 66.0
messages = ['310314', '310316', '310317', '310318', '310448', '310562', '311210', '311211', '311212', '311217', '311218', '311219', '311220', '311222', '311224', '311299', '311533', '311534', '311536', '311538', '312001', '312004', '312260', '312262', '312288', '312362', '312367', '312368', '312369', '312370', '312372', '312443', '312447', '312837', '312888', '312975', '312976', '312979', '313010', '316657', '316717', '316850', '316936', '318333', '318479', '319560', '367573', '368349', '368395', '368846', '368849', '368851', '368899', '368902', '369378', '369454', '369455', '370130', '370140', '370631', '370660', '371157', '371173', '371176', '371704', '371707']
nosy_count = 13.0
nosy_names = ['ncoghlan', 'belopolsky', 'pitrou', 'vstinner', 'pmpp', 'eric.snow', 'zach.ware', 'steve.dower', 'davin', 'emilyemorehouse', 'pablogsal', 'miss-islington', 'nanjekyejoannah']
pr_nums = ['1748', '5436', '5437', '5507', '5509', '5516', '5624', '5709', '5710', '5778', '5783', '6813', '6914', '6937', '7251', '7288', '7330', '19768', '19770', '19829', '20089', '18817', '19985', '20465', '20600', '20611', '20777', '20926']
priority = None
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue32604'
versions = ['Python 3.10']

Linked PRs

In the interest of getting something landed for 3.7, so we can start using it in tests, I'm putting up a patch for a low-level interpreters module. In some ways this is a precursor for issue bpo-30439, which will add a proper public stdlib module in 3.8.

The module I'm adding draws from the ideas in PEP-554 (particularly for channels). Consequently, this will also give us an opportunity to try out some of the semantics from the PEP to give us better ideas for 3.8.

I expect to have some follow-on patches to facilitate simpler use in tests. This patch is big enough already. :)

@ned

@ned, it may be a little tight to land this given the time left before beta 1. However, this is meant as a tool for us to use in the test suite (particularly to test the subinterpreter C-API). So I'm arguing that, if necessary, it would still be okay to land this after the feature freeze. (I'm still hoping to get this in before the cutoff.) What do you think?

@nick

@nick, I may make the name change you suggested in issue bpo-30439 ("_subinterpreters").

FYI, there are a few things I need to clean up in the PR. However, I expect that those changes will be minor relative to the the whole patch, so I wanted to get the ball rolling on a review. :)

@eric

@eric, given the breadth of change introduced in the PR (including adding a new extension), I think it would be best if at all possible to get it in for beta 1 if we can resolve the review comments in time. If necessary and if there are no objections from other core developers, I would be willing to consider making an exception and allowing it into beta 2 as long as it remains a private interface. If it looks like it won't be in releasable shape by then, I think you should hold off for 3.8; doing otherwise would be unfair to others and to our downstream beta users / testers, for example, even if it is private, adding a new extension and setup.py changes potentially affect downstream packagers.

Sounds good, Ned. Thanks for taking a look. I should have everything finished up by Friday, so I'm hopeful for landing the change before the deadline. I may have a few minor tweaks to make after that, but I'll discuss that with you before making any changes if that happens.

New changeset 7f8bfc9 by Eric Snow in branch 'master':
bpo-32604: Expose the subinterpreters C-API in a "private" stdlib module. (gh-1748)
7f8bfc9

I've merged the patch without Windows support, which shouldn't be a problem given the purpose of the extension module. I've also added a PR for get the module building under Windows. I'd like to get that resolved ASAP.

Eric, looks like some buildbots are unhappy, for instance:

http://buildbot.python.org/all/#builders/13/builds/648

Yeah, I'm looking into it. Also, I noticed some refleaks that I'll be sorting out.

On 4 of the buildbots:

======================================================================
ERROR: test_drop_multiple_times (test.test__xxsubinterpreters.ChannelTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test__xxsubinterpreters.py", line 890, in test_drop_multiple_times
    interpreters.channel_drop_interpreter(cid, send=True, recv=True)
SystemError: More keyword list entries (7) than format specifiers (3)

======================================================================
ERROR: test_drop_single_user (test.test__xxsubinterpreters.ChannelTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test__xxsubinterpreters.py", line 848, in test_drop_single_user
    interpreters.channel_drop_interpreter(cid, send=True, recv=True)
SystemError: More keyword list entries (7) than format specifiers (3)

======================================================================
ERROR: test_drop_used_multiple_times_by_single_user (test.test__xxsubinterpreters.ChannelTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test__xxsubinterpreters.py", line 957, in test_drop_used_multiple_times_by_single_user
    interpreters.channel_drop_interpreter(cid, send=True, recv=True)
SystemError: More keyword list entries (7) than format specifiers (3)

======================================================================
ERROR: test_drop_with_unused_items (test.test__xxsubinterpreters.ChannelTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test__xxsubinterpreters.py", line 899, in test_drop_with_unused_items
    interpreters.channel_drop_interpreter(cid, send=True, recv=True)
SystemError: More keyword list entries (7) than format specifiers (3)

gh-76785: Minor Cleanup of "Cross-interpreter" Code (gh-126457)

gh-76785: Minor Cleanup of Exception-related Cross-interpreter State (g…

gh-76785: Improved Subinterpreters Compatibility with 3.12 (1/2) (gh-…

gh-76785: Improved Subinterpreters Compatibility with 3.12 (2/2) (gh-…

pythongh-76785: Minor Cleanup of "Cross-interpreter" Code (pythongh-1…

pythongh-76785: Minor Cleanup of Exception-related Cross-interpreter …

pythongh-76785: Improved Subinterpreters Compatibility with 3.12 (1/2) (

pythongh-76785: Improved Subinterpreters Compatibility with 3.12 (2/2) (

pythongh-76785: Minor Cleanup of "Cross-interpreter" Code (pythongh-1…

pythongh-76785: Minor Cleanup of Exception-related Cross-interpreter …

pythongh-76785: Improved Subinterpreters Compatibility with 3.12 (1/2) (

pythongh-76785: Improved Subinterpreters Compatibility with 3.12 (2/2) (

Now tracked in #134939.

A

Navigation Menu

Repository navigation

[subinterpreters] PEP 554 implementation: add interpreters module #76785

Linked PRs

213 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[subinterpreters] PEP 554 implementation: add interpreters module #76785

Description

Linked PRs

Activity

ericsnowcurrently commented on Jan 20, 2018

ericsnowcurrently commented on Jan 20, 2018

ericsnowcurrently commented on Jan 20, 2018

ericsnowcurrently commented on Jan 20, 2018

ned-deily commented on Jan 22, 2018

ericsnowcurrently commented on Jan 24, 2018

ericsnowcurrently commented on Jan 30, 2018

ericsnowcurrently commented on Jan 30, 2018

ned-deily commented on Jan 30, 2018

ericsnowcurrently commented on Jan 30, 2018

ericsnowcurrently commented on Jan 30, 2018

213 remaining items

AA-Turner commented on Aug 11, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions