Support Python 3.11 #7143

Activity

1. The-Compiler

   added

   component: infrastructureIssues related to development scripts, CI, servers, etc.Issues related to development scripts, CI, servers, etc.

   priority: 0 - highIssues which are currently the primary focus.Issues which are currently the primary focus.

   on Apr 26, 2022
2. The-Compiler

   added a commit that references this issue on Apr 28, 2022

   ci: Comment out Python 3.11 entirely for now

   ...

   5a4ae36
3. twigleingrid

   added a commit that references this issue on May 13, 2022

   ci: Comment out Python 3.11 entirely for now

   ...

   d22d6d5
4. The-Compiler commented on May 26, 2022

   

   The-Compiler

   on May 26, 2022

   MemberAuthor

   More actions

   Gave this another try with the 3.11 branch (i.e. after Python 3.11.0b1) on qt6-v2. It looks better, though I needed some adjustments:

   • 9124c22 mkvenv: Remove sip packages from binary args
   • 543da7a mypy requirements: Skip lxml on Python 3.11
   • 4960c18 Patch PyQt6 QUrl flag behavior for Python 3.11
   • 64e8c71 tests: Ignore Flask/Werkzeug ResourceWarnings

   And reported/reproduced some upstream bugs:

   • Python/pystate.c:2218: _PyThreadState_PopFrame: Assertion `tstate->datastack_top >= base' failed. python/cpython#93252
   • Objects/codeobject.c:814: retreat: Assertion `bounds->ar_start > 0' failed python/cpython#93249
   • Default enum.FlagBoundary should be KEEP? python/cpython#93250
   • Crash on AST with misordered linenos python/cpython#92597
   • ResourceWarning for unclosed sockets with development server pallets/werkzeug#2421

   CPython patches:

   From ea116aaa1c9904149befdd8a45a985d550cad2aa Mon Sep 17 00:00:00 2001
   From: Florian Bruhin <me@the-compiler.org>
   Date: Thu, 26 May 2022 14:40:23 +0200
   Subject: [PATCH] Disable broken asserts
   
   ---
    Python/compile.c | 2 +-
    Python/pystate.c | 2 +-
    2 files changed, 2 insertions(+), 2 deletions(-)
   
   diff --git a/Python/compile.c b/Python/compile.c
   index cc0d76e038..42507532a5 100644
   --- a/Python/compile.c
   +++ b/Python/compile.c
   @@ -7531,7 +7531,7 @@ write_location_info_long_form(struct assembler* a, struct instr* i, int length)
        assert(length > 0 &&  length <= 8);
        write_location_first_byte(a, PY_CODE_LOCATION_INFO_LONG, length);
        write_location_signed_varint(a, i->i_lineno - a->a_lineno);
   -    assert(i->i_end_lineno >= i->i_lineno);
   +    //assert(i->i_end_lineno >= i->i_lineno);
        write_location_varint(a, i->i_end_lineno - i->i_lineno);
        write_location_varint(a, i->i_col_offset+1);
        write_location_varint(a, i->i_end_col_offset+1);
   diff --git a/Python/pystate.c b/Python/pystate.c
   index 3e28a6ab69..c801cdcd8b 100644
   --- a/Python/pystate.c
   +++ b/Python/pystate.c
   @@ -2215,7 +2215,7 @@ _PyThreadState_PopFrame(PyThreadState *tstate, _PyInterpreterFrame * frame)
        }
        else {
            assert(tstate->datastack_top);
   -        assert(tstate->datastack_top >= base);
   +        //assert(tstate->datastack_top >= base);
            tstate->datastack_top = base;
        }
    }
   -- 
   2.36.1
   

   This all results in:

   FAILED tests/end2end/features/test_downloads_bdd.py::test_answering_a_question_for_a_cancelled_download_415 - end2end.fixtures.testprocess.WaitForTimeout: Timed out after 5000ms waiting for {'category': 'commands', 'module': 'command',...
   FAILED tests/end2end/features/test_downloads_bdd.py::test_answering_a_question_for_a_cancelled_download_415 - AssertionError: assert False
   FAILED tests/end2end/features/test_tabs_bdd.py::test_undo_with_window_and_count - end2end.fixtures.testprocess.WaitForTimeout: Timed out while waiting for about:blank to be loaded
   FAILED tests/unit/utils/test_debug.py::TestQEnumKey::test_qenum_key[Qt-1024-None-0x0400] - TypeError: Can't guess enum class of an int!
   FAILED tests/unit/utils/test_debug.py::TestQFlagsKey::test_qflags_key[Qt-4352-AlignmentFlag-AlignBaseline|0x1000] - AttributeError: 'int' object has no attribute 'name'
   ERROR tests/end2end/features/test_downloads_bdd.py::test_downloading_with_infinite_redirect - end2end.fixtures.testprocess.InvalidLine: 
   ERROR tests/end2end/features/test_downloads_bdd.py::test_quitting_with_finished_downloads_and_confirm_quitdownloads_issue_846 - end2end.fixtures.testprocess.InvalidLine: 
   ERROR tests/end2end/features/test_downloads_bdd.py::test_retrying_a_failed_download - end2end.fixtures.testprocess.InvalidLine: 
   ERROR tests/end2end/features/test_hints_bdd.py::test_using_rapid_number_hinting_twice - end2end.fixtures.testprocess.InvalidLine: 
   ERROR tests/end2end/features/test_hints_bdd.py::test_using_hintsauto_follow__uniquematch_in_letter_mode - end2end.fixtures.testprocess.WaitForTimeout: Timed out after 5000ms waiting for {'category': 'ipc', 'module': 'ipc', 'function': ...
   ERROR tests/end2end/features/test_tabs_bdd.py::test_undo_with_window_and_count - end2end.fixtures.testprocess.ProcessExited
   

   when running:

   .venv-py311/bin/pytest  -- tests/end2end/features/test_downloads_bdd.py tests/end2end/features/test_hints_bdd.py tests/end2end/features/test_tabs_bdd.py tests/unit/utils/test_debug.py     
   

   It seems like mostly timeout issues (perhaps due to the thread issue above?) and this:

   E           Exception ignored in: <function IOBase.__del__ at 0x7f7f228c4f70>
   E           Traceback (most recent call last):
   E             File "/home/florian/proj/cpython/Lib/_pyio.py", line 436, in __del__
   E               self.close()
   E               ^^^^^^^^^^^^
   E             File "/home/florian/proj/cpython/Lib/_pyio.py", line 1338, in close
   E               self.flush()
   E               ^^^^^^^^^^^^
   E             File "/home/florian/proj/cpython/Lib/_pyio.py", line 1299, in flush
   E               self._flush_unlocked()
   E               ^^^^^^^^^^^^^^^^^^^^^^
   E             File "/home/florian/proj/qutebrowser/git/.venv-py311/lib/python3.11/site-packages/cheroot/makefile.py", line 45, in _flush_unlocked
   E               n = self.raw.write(bytes(self._write_buf))
   E                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   E             File "/home/florian/proj/cpython/Lib/socket.py", line 723, in write
   E               return self._sock.send(b)
   E                      ^^^^^^^^^^^^^^^^^^
   E           OSError: [Errno 9] Bad file descriptor
   E           Exception ignored in: <function IOBase.__del__ at 0x7f7f228c4f70>
   E           Traceback (most recent call last):
   E             File "/home/florian/proj/cpython/Lib/_pyio.py", line 436, in __del__
   E               self.close()
   E               ^^^^^^^^^^^^
   E             File "/home/florian/proj/cpython/Lib/_pyio.py", line 1338, in close
   E               self.flush()
   E               ^^^^^^^^^^^^
   E             File "/home/florian/proj/cpython/Lib/_pyio.py", line 1299, in flush
   E               self._flush_unlocked()
   E               ^^^^^^^^^^^^^^^^^^^^^^
   E             File "/home/florian/proj/qutebrowser/git/.venv-py311/lib/python3.11/site-packages/cheroot/makefile.py", line 45, in _flush_unlocked
   E               n = self.raw.write(bytes(self._write_buf))
   E                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   E             File "/home/florian/proj/cpython/Lib/socket.py", line 723, in write
   E               return self._sock.send(b)
   E                      ^^^^^^^^^^^^^^^^^^
   E           OSError: [Errno 9] Bad file descriptor
   

   Should probably retry with a non-debug Python and the master branch too.
5. The-Compiler commented on Jun 1, 2022

   

   The-Compiler

   on Jun 1, 2022

   MemberAuthor

   More actions

   Tried with a release build of Python 3.11b2 with python/cpython#93397 on top. The only remaining blocker with that is python/cpython#93252 I think.
6. The-Compiler commented on Jul 10, 2022

   

   The-Compiler

   on Jul 10, 2022

   MemberAuthor

   More actions

   With python/cpython#94708 and python/cpython#93895 I believe we can finally get back to this once 3.11.0b4 is released. Not sure what the exact plan is for that, see Mailman 3 [Release] Status of Python 3.11 release - Python-Dev - python.org.

   It looks like there are some new failing tests though, due to some kind of coverage/tracing changes:

   >       assert covtest.check() == [expected]
   E       AssertionError: assert [Message(typ=...h coverage!')] == [Message(typ=...h coverage!')]
   E         At index 0 diff: Message(typ=<MsgType.insufficient_coverage: 1>, filename='module.py', text='module.py has 71.43% line and 50.00% branch coverage!') != Message(typ=<MsgType.insufficient_coverage: 1>, filename='module.py', text='module.py has 83.33% line and 50.00% branch coverage!')
   

   and others:

   FAILED tests/unit/scripts/test_check_coverage.py::test_tested_no_branches - AssertionError: assert [Message(typ=...h coverage!')] == []
   FAILED tests/unit/scripts/test_check_coverage.py::test_tested_with_branches - AssertionError: assert [Message(typ=...h coverage!')] == []
   FAILED tests/unit/scripts/test_check_coverage.py::test_untested - AssertionError: assert [Message(typ=...h coverage!')] == [Message(typ=...h coverage!')]
   FAILED tests/unit/scripts/test_check_coverage.py::test_untested_floats - AssertionError: assert [Message(typ=...h coverage!')] == [Message(typ=...h coverage!')]
   FAILED tests/unit/scripts/test_check_coverage.py::test_untested_branches - AssertionError: assert [Message(typ=...h coverage!')] == [Message(typ=...h coverage!')]
   FAILED tests/unit/scripts/test_check_coverage.py::test_tested_unlisted - AssertionError: assert [] == [Message(typ=...fect_files!')]
   

   Also, the "Bad file descriptor" issue in cheroot still seems to be around:

   E           end2end.fixtures.testprocess.InvalidLine: 
   E           Exception ignored in: <function IOBase.__del__ at 0x7ff4c226cec0>
   E           Traceback (most recent call last):
   E             File "/home/florian/proj/cpython/Lib/_pyio.py", line 436, in __del__
   E               self.close()
   E               ^^^^^^^^^^^^
   E             File "/home/florian/proj/cpython/Lib/_pyio.py", line 1338, in close
   E               self.flush()
   E               ^^^^^^^^^^^^
   E             File "/home/florian/proj/cpython/Lib/_pyio.py", line 1299, in flush
   E               self._flush_unlocked()
   E               ^^^^^^^^^^^^^^^^^^^^^^
   E             File "/home/florian/proj/qutebrowser/git/.venv-py311/lib/python3.11/site-packages/cheroot/makefile.py", line 45, in _flush_unlocked
   E               n = self.raw.write(bytes(self._write_buf))
   E                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   E             File "/home/florian/proj/cpython/Lib/socket.py", line 723, in write
   E               return self._sock.send(b)
   E                      ^^^^^^^^^^^^^^^^^^
   E           OSError: [Errno 9] Bad file descriptor
   

   in various somewhat random tests:

   ERROR tests/end2end/features/test_downloads_bdd.py::test_not_remembering_the_last_download_directory - end2end.fixtures.testprocess.InvalidLine: 
   ERROR tests/end2end/features/test_hints_bdd.py::test_yanking_javascript_link_to_clipboard - end2end.fixtures.testprocess.InvalidLine: 
   ERROR tests/end2end/features/test_private_bdd.py::test_adblocking_after_reiniting_private_profile - end2end.fixtures.testprocess.InvalidLine: 
   
7. The-Compiler commented on Jul 12, 2022

   

   The-Compiler

   on Jul 12, 2022

   Last edited by The-Compiler

   MemberAuthor

   More actions

   3.11.0b4 is out now. The coverage issue is coveragepy/coveragepy#1419 which has been fixed in coveragepy/coveragepy@0d64498 - and indeed our tests work fine with that.

   That means only the OSError: [Errno 9] Bad file descriptor is remaining now. Only happens with --with-pydebug builds, though. Maybe it's not even specific to Python 3.11 either.

   edit: Indeed, also happens with a Python 3.10 built with --with-pydebug.

   Probably time to re-enable Python 3.11 on CI and close this! 🎉 (at least when there has been a coverage.py release)
8. The-Compiler

   closed this as completedin d1e6531on Jul 12, 2022
9. The-Compiler commented on Jul 12, 2022

   

   The-Compiler

   on Jul 12, 2022

   MemberAuthor

   More actions

   Oh, looks like GitHub Actions is not quite ready yet, probably waiting for actions/python-versions#177.
10. The-Compiler

    reopened this on Jul 12, 2022
11. The-Compiler commented on Jul 13, 2022

    

    The-Compiler

    on Jul 13, 2022

    MemberAuthor

    More actions

    Works fine now: https://github.com/qutebrowser/qutebrowser/runs/7319313602?check_suite_focus=true ✨
12. The-Compiler

    closed this as completedon Jul 13, 2022
13. markstos

    added a commit that references this issue on Feb 3, 2024

    ci: Re-add Python 3.11

    ...

    This issue will close once commit d1e6531 is merged into the 'master' branch.

    d1e6531