Visitar URL original
[subinterpreters] Get the current Python interpreter state from Thread Local Storage (autoTSSkey) · Issue #84702 · python/cpython · GitHub
Skip to content

[subinterpreters] Get the current Python interpreter state from Thread Local Storage (autoTSSkey) #84702

Closed
Closed
[subinterpreters] Get the current Python interpreter state from Thread Local Storage (autoTSSkey)#84702
Labels
3.9 (EOL)end of lifetopic-subinterpreters

Description

@vstinner
vstinner
opened on May 5, 2020
BPO 40522
Nosy @vstinner, @seberg, @corona10, @pablogsal, @shihai1991, @h-vetinari
PRs
  • bpo-40522: _PyThreadState_Swap() sets autoTSSkey #19939
  • bpo-40522: Store tstate in a Thread Local Storage #23976
  • bpo-40522: Replace PyThreadState_GET() with PyThreadState_Get() #24575
  • bpo-40522: Use PyThreadState_Get() in _lsprof.c #24596
  • bpo-40522: Use PyThreadState_Get() in decimal #24597
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2020-05-05.17:12:14.636>
labels = ['expert-subinterpreters', '3.9']
title = '[subinterpreters] Get the current Python interpreter state from Thread Local Storage (autoTSSkey)'
updated_at = <Date 2021-06-29.17:34:08.113>
user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2021-06-29.17:34:08.113>
actor = 'h-vetinari'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Subinterpreters']
creation = <Date 2020-05-05.17:12:14.636>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 40522
keywords = ['patch']
message_count = 11.0
messages = ['368182', '368188', '380324', '383894', '383895', '383899', '383900', '383940', '383967', '384057', '387312']
nosy_count = 6.0
nosy_names = ['vstinner', 'seberg', 'corona10', 'pablogsal', 'shihai1991', 'h-vetinari']
pr_nums = ['19939', '23976', '24575', '24596', '24597']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue40522'
versions = ['Python 3.9']

    Activity

    1. vstinner commented on May 5, 2020

      @vstinner
      vstinner
      on May 5, 2020
      MemberAuthor

      _PyThreadState_GET() gets the current Python thread state from _PyRuntime.gilstate.tstate_current atomic variable.

      When I experimented per-interpreter GIL (bpo-40512), I got issues with _PyThreadState_GET() which didn't return the expected Python thread state.

      I propose to modify _PyThreadState_GET() in the exprimental isolated subinterpreters mode to get and set the current Python thread state using a Thread Local Storage: autoTSSkey.

    2. vstinner
      added
      interpreter-core(Objects, Python, Grammar, and Parser dirs)
      3.9 (EOL)end of life
      on May 5, 2020

    3. vstinner commented on May 5, 2020

      @vstinner
      vstinner
      on May 5, 2020
      MemberAuthor

      New changeset e838a93 by Victor Stinner in branch 'master':
      bpo-40522: _PyThreadState_Swap() sets autoTSSkey (GH-19939)
      e838a93

    4. vstinner
      added
      topic-subinterpreters
      and removed
      interpreter-core(Objects, Python, Grammar, and Parser dirs)
      on May 15, 2020
    5. vstinner
      changed the title [-]Subinterpreters: get the current Python interpreter state from Thread Local Storage (autoTSSkey)[/-] [+][subinterpreters] Get the current Python interpreter state from Thread Local Storage (autoTSSkey)[/+] on May 15, 2020
    6. vstinner
      added
      topic-subinterpreters
      and removed
      interpreter-core(Objects, Python, Grammar, and Parser dirs)
      on May 15, 2020
    7. vstinner
      changed the title [-]Subinterpreters: get the current Python interpreter state from Thread Local Storage (autoTSSkey)[/-] [+][subinterpreters] Get the current Python interpreter state from Thread Local Storage (autoTSSkey)[/+] on May 15, 2020

    8. vstinner commented on Nov 4, 2020

      @vstinner
      vstinner
      on Nov 4, 2020
      MemberAuthor

      See also bpo-15751: "Make the PyGILState API compatible with subinterpreters".

    9. vstinner commented on Dec 28, 2020

      @vstinner
      vstinner
      on Dec 28, 2020
      MemberAuthor

      There are many ways to get the current interpreter (interp) and the current Python thread state (tstate).

      Public C API, opaque function call:

      • PyInterpreterState_Get() (new in Python 3.9)
      • PyThreadState_Get()

      Internal C API, static inline functions:

      • _PyInterpreterState_GET()
      • _PyThreadState_GET()

      There are so many variants that I wrote notes for myself:
      https://pythondev.readthedocs.io/pystate.html

      This issue is about optimizing _PyInterpreterState_GET() and _PyThreadState_GET() which are supposed to be the most efficient implementations.

      Currently, _PyInterpreterState_GET() is implemented as _PyThreadState_GET()->interp, and _PyThreadState_GET() is implemented as:

          _Py_atomic_load_relaxed(_PyRuntime.gilstate.tstate_current)

      --

      To find the _PyInterpreterState_GET() machine code, I read the PyInterpreterState_Get() assembly code (not optimized, it adds tstate==NULL test) and PyTuple_New() assembly code, since PyTuple_New() now needs to get the current interpreter:

      static struct _Py_tuple_state *
get_tuple_state(void)
{
    PyInterpreterState *interp = _PyInterpreterState_GET();
    return &interp->tuple;
}

      To find the _PyThreadState_GET() machine code, I read the PyThreadState_Get() assembly code.

      I looked at the x86-64 machine code generated by GCC -O3 (no LTO, no PGO, it should not be relevant here), using GCC 10.2.1 on Fedora 33.

      _PyThreadState_GET():

      mov rax,QWORD PTR [rip+0x2292b1] # 0x743118 <_PyRuntime+568>

      _PyInterpreterState_GET():

      mov rax,QWORD PTR [rip+0x22a7dd] # 0x743118 <_PyRuntime+568>
      mov rax,QWORD PTR [rax+0x10]

      By default, Python is built with -fPIC: _PyRuntime variable does not have a fixed address.

      $ objdump -t ./python|grep '\<_PyRuntime\>'
0000000000742ee0 g     O .bss	00000000000002a0              _PyRuntime

      The "[rip+0x2292b1] # 0x743118 <_PyRuntime+568>" indirection is needed by PIC.

    10. vstinner commented on Dec 28, 2020

      @vstinner
      vstinner
      on Dec 28, 2020
      MemberAuthor

      _PyInterpreterState_GET():
      mov rax,QWORD PTR [rip+0x22a7dd] # 0x743118 <_PyRuntime+568>
      mov rax,QWORD PTR [rax+0x10]

      While working on bpo-39465, I wrote PR 20767 to optimize _PyInterpreterState_GET(): single instruction instead of two:

      • Add _PyRuntimeState.interp_current member: atomic variable
      • _PyThreadState_Swap() sets _PyRuntimeState.interp_current

      But I failed to measure any performance difference.

    11. vstinner commented on Dec 28, 2020

      @vstinner
      vstinner
      on Dec 28, 2020
      MemberAuthor

      PR 23976 stores the currrent interpreter and the current Python thread state into a Thread Local Storage (TLS) using GCC/clang __thread keyword.

      On x86-64 using LTO and GCC -O3, _PyThreadState_GET() and _PyInterpreterState_GET() become a single MOV.

      Assembly code using LTO and gcc -O3.

      _PyThreadState_GET() in _PySys_GetObjectId():

      0x00000000004adabe <+14>: mov rbx,QWORD PTR fs:0xfffffffffffffff8

      _PyThreadState_GET() in PyThreadState_Get():

      0x000000000046b660 <+0>: mov rax,QWORD PTR fs:0xfffffffffffffff8

      _PyInterpreterState_GET() in PyTuple_New():

      0x000000000048dfcc <+12>: mov rax,QWORD PTR fs:0xfffffffffffffff0

      _PyInterpreterState_GET() in PyState_FindModule():

      0x000000000044bf20 <+16>: mov rax,QWORD PTR fs:0xfffffffffffffff0

      ---

      Note: Without LTO, sometimes there is an indirection:

      _PyThreadState_GET() in _PySys_GetObjectId(), 2 MOV (PIC indirection):

      mov rax,QWORD PTR [rip+0x1eb270] # 0x713fe0
      # rax = 0xfffffffffffffff0 (-16)
      mov r13,QWORD PTR fs:[rax]

      _PyInterpreterState_GET() in PyTuple_New(), 2 MOV (PIC indirection):

      mov rax,QWORD PTR [rip+0x294d95] # 0x713ff8
      mov rax,QWORD PTR fs:[rax]

      An optimized Python should always be built with LTO.

    12. vstinner commented on Dec 28, 2020

      @vstinner
      vstinner
      on Dec 28, 2020
      MemberAuthor

      PR 23976 should make _PyInterpreterState_GET() more efficient, and it prepares the code base for one GIL per interpreter (which is purpose of this issue ;-)).

    13. vstinner commented on Dec 28, 2020

      @vstinner
      vstinner
      on Dec 28, 2020
      MemberAuthor

      Mark Shannon experiment using __thread:

      He added " extern __thread struct _ts *_Py_tls_tstate;" to Include/cpython/pystate.h.

    14. pablogsal commented on Dec 29, 2020

      @pablogsal
      pablogsal
      on Dec 29, 2020
      Member

      An optimized Python should always be built with LTO.

      In MacOS is quite challenging to activate LTO, so normally optimized builds are only done with PGO. Also in Windows I am not sure is possible to use LTO. Same for many other platforms

    15. vstinner commented on Dec 30, 2020

      @vstinner
      vstinner
      on Dec 30, 2020
      MemberAuthor

      One GIL per interpreter requires to store the tstate per thread. I don't see any other option. We need to replace the global _PyRuntime atomic variable with a TLS variable. I'm trying to reduce the overhead, but it's heard to beat the performance of an atomic variable.

      That's also we I modified many functions to pass explicitly tstate to subfunctions in internal C functions, to avoid any possible overhead of getting tstate.

      https://vstinner.github.io/cpython-pass-tstate.html

      Pablo:

      In MacOS is quite challenging to activate LTO, so normally optimized builds are only done with PGO.

      Oh right, I forgot macOS. I should check how TLS is compiled on macOS. IMO wwo MOV instead of MOV is not a major performance bottleneck.

      The best would be to be able to avoid pthread_getspecific() function which is less efficient than a TLS variable. The glibc implementation uses an array for a few variables (first 32 variables?) and then a slower hash table.

      Pablo:

      Also in Windows I am not sure is possible to use LTO. Same for many other platforms.

      I will check how it's implemented on Windows.

      We cannot use TLS on all platforms, since it requires C11 features which are not available on all platforms. Also, the implementation depends on the architecture.

    16. vstinner commented on Feb 19, 2021

      @vstinner
      vstinner
      on Feb 19, 2021
      MemberAuthor

      New changeset 6207810 by Victor Stinner in branch 'master':
      bpo-40522: Replace PyThreadState_GET() with PyThreadState_Get() (GH-24575)
      6207810

    17. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    18. ezio-melotti
      added this to Subinterpreterson Apr 15, 2022
    19. ezio-melotti
      moved this to Todo in Subinterpreterson Apr 15, 2022

    20. vstinner commented on Nov 3, 2022

      @vstinner
      vstinner
      on Nov 3, 2022
      MemberAuthor

      Sadly, I don't have the bandwidth to work on this issue, I just close it.

    21. vstinner
      closed this as completedon Nov 3, 2022
    22. Repository owner moved this from Todo to Done in Subinterpreterson Nov 3, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.9 (EOL)end of lifetopic-subinterpreters

      Projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions