Test suite skips failing tests when setUp[Class] fails #90681

Activity

1. sobolevn commented on Jan 25, 2022

   

   sobolevn

   on Jan 25, 2022

   MemberAuthor

   More actions

   Here's what happened. We had an error in test_typing.py, which was silently ignored.

    ======================================================================
   ERROR: setUpClass (test.test_typing.NewTypeTests)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "D:\a\cpython\cpython\lib\test\test_typing.py", line 3917, in setUpClass
       UserId = NewType('UserId', int)
   NameError: name 'NewType' is not defined
   
   ----------------------------------------------------------------------
   Ran 396 tests in 0.085s
   
   FAILED (errors=1, skipped=1)
   test test_typing failed
   

   Link: https://github.com/python/cpython/runs/4902363883?check_suite_focus=true

   But, later the suite runner tried to rerun it:

   0:09:12 load avg: 6.37 Re-running failed tests in verbose mode
   0:09:12 load avg: 6.37 Re-running test_typing in verbose mode (matching: setUpClass)
   
   1 re-run test:
       test_typing
   
   1 test run no tests:
       test_typing
   

   And since nothing matched setUpClass - no tests were executed and the CI went green instead of red.

   What can we do?

   1. Only schedule real test_ item to be rerun, fail for everything else
   2. Convert setupClass failure into the whole class rerun
   3. Other options?

   I would like to work on this, when we will decide which way is best.
2. sobolevn

   added

   3.9 (EOL)end of lifeend of life

   3.10 (EOL)end of lifeend of life

   3.11only security fixesonly security fixes

   testsTests in the Lib/test dirTests in the Lib/test dir

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Jan 25, 2022
3. Fidget-Spinner commented on Jan 25, 2022

   

   Fidget-Spinner

   on Jan 25, 2022

   Member

   More actions

   I vote for option 2.
4. Fidget-Spinner

   changed the title [-]Test suite skips failing tests[/-] [+]Test suite skips failing tests when setUp[Class] fails[/+] on Jan 25, 2022
5. Fidget-Spinner

   changed the title [-]Test suite skips failing tests[/-] [+]Test suite skips failing tests when setUp[Class] fails[/+] on Jan 25, 2022
6. ezio-melotti

   transferred this issue fromon Apr 10, 2022
7. furkanonder commented on May 8, 2023

   

   furkanonder

   on May 8, 2023

   Contributor

   More actions

   @sobolevn The issue seems to have been solved. We can close the issue.
8. sobolevn

   closed this as completedon May 9, 2023