Repository navigation
[feature request] PyTorch support for sub-interpreters with PEP 684 accepted and release in Python 3.12聽#102517
Description
馃殌 The feature, motivation and pitch
It appears that sub-interpretor support for Python in PEP 684 https://discuss.python.org/t/pep-684-a-per-interpreter-gil/ is advancing and will be merged soon (https://github.com/python/cpython/blob/main/Misc/NEWS.d/3.12.0b1.rst, python/cpython#104210, https://github.com/ericsnowcurrently/multi-core-python/wiki/0-The-Plan)
Any updates on eventual support of sub-interpreters in PyTorch?
Also, a question would be of zero-copy in-RAM IPC data exchange: maybe via named pipes? Some mmaps? shm?
I found older issues asking the same:
- upcoming PEP 554: how much effort we need to support sub-interpreter聽#23490
- Cannot run torch in different sub-interpreters聽#10950
This might be important for working in hosted multi-threaded environments where every thread runs its own Python / PyTorch code. This is also related to the current torch.compile / export / mini-interpreters story: https://discuss.pytorch.org/t/torch-compiles-deployment-story-to-non-python-host-processes/180943. This scenario was explicitly supported with TorchScript / TorchJit, but seems not yet developed for torch.compile, but it's an important usecase too (similar to API's OnnxRuntime is currently offering)
Alternatives
No response
Additional context
No response
Activity
albanD commentedon May 31, 2023
Would https://github.com/pytorch/multipy work for you? Any details on why it would or not?
Also I owuld be curious if there are more information on the questions in the other thread. In particular if our dependencies (like numpy) do support sub-interpreters?
vadimkantorov commentedon Jun 1, 2023
@albanD I guess
multipywould work! It would be great to see some tutorial of usingmultipytogether with torch.compile / triton-compiled AOT packagesBut maybe in the future, it's still important to track state-of-subinterpreters support for PyTorch/torch.compile and have a simple test or have a multipy/torch.package-like system leveraging subinterpreters but still allowing packaging only relevant code/modules in self-contained archives
vadimkantorov commentedon Jun 2, 2023
@albanD I would say this issue is not about python frontend, but more about "python backend" :)
albanD commentedon Jun 2, 2023
They are independent so should "just work" in theory :)
vadimkantorov commentedon Jun 2, 2023
Would be good to have a sample showcasing the two :) I think
multipyis relatively unknown. And torch.compile currently misses communication about inference time (which was different for torch::jit::load etc)vadimkantorov commentedon Oct 8, 2023
I guess Python 3.12 is released now. So at least it would be nice to have some note in the docs on the support / interaction with sub-interpreters (even if it's not supported yet, it would be better to state it explicitly to save time for the users)
albanD commentedon Oct 9, 2023
Note that we don't support 3.12 yet. But if you have a good grasp on sub-interpreters, feel free to write a blurb on it!
vadimkantorov commentedon Oct 29, 2023
Regarding "better off using multiprocessing", this scenario is mainly about embedding python in some other inference process which might need multiple sub-interpreters (and then maybe spawn is less convenient?). So some basic tests of having torch/domain libraries/numpy operating okay (or not okay) is important, as people are going to try this.
Also, for this kinds of inference, being able to host the readonly model weights in memory (I guess important for both CPU and CUDA - for RAM my understanding is that it used to have shm, but I don't know if CUDA memory can be shared in some similar way) only once and have them accessible between multiple threads / subinterpreters may need to be discussed...
albanD commentedon Oct 31, 2023
Yes you can and that will be done by default if you send a cuda Tensor over multiprocessing objects.
This is already possible for threads or processes based system today. Not sure what subinterpreter adds to the picture (beyond making PyObject ownership more complex since no object should be shared between interpreters)?
vadimkantorov commentedon Nov 11, 2023
E.g. maybe some bugs like this? #113520 When they were concealed, they might get hit if these multi-threaded interpreters will be used...
albanD commentedon Nov 16, 2023
Nope, these registration are happening on the C++ lib which would be shared and would fail the same way.
I would argue this problem would actually be worse because you would loose the ability to lock between these concurrent calls since they can't interact with each other.
vadimkantorov commentedon Jul 16, 2025
NumPy issue: