Repository navigation
Make xpass a failure again #11467
Description
What's the problem this feature will solve?
Some years ago we accepted a change which made xpass not a failure
I recently hit multiple cases where it was missed that a test was fixed
That's not an acceptable default
Activity
nicoddemus commentedon Sep 25, 2023
This means changing the default of the
strictparameter toTrue?RonnyPfannschmidt commentedon Sep 25, 2023
That's would be the starting point, In the long run it should go away
TanyaAgarwal28 commentedon Oct 9, 2023
Hi, How we can test this scenario, I can't see xpass as a marker in pytest.
RonnyPfannschmidt commentedon Oct 9, 2023
@TanyaAgarwal28 xpass is a test outcome for when a xfail marked test passed "unexpectedly"
TanyaAgarwal28 commentedon Oct 10, 2023
Yeah Got it so what we are facing issue here?
Isin't a feature that if xfail testcase is passed then it will came as xpassed in outcome?
import pytest
@pytest.mark.xfail
def test_example_expected_fail():
assert 3 + 3 == 6
RonnyPfannschmidt commentedon Oct 10, 2023
@TanyaAgarwal28 the problem is that its a misfeature - it hides important change for a non-flaky testsuite - fixing a previously failing test by accident
when non-strict xfail was introduced, it was a major mistake not to use a new mark
TanyaAgarwal28 commentedon Oct 11, 2023
So basically we have to revert this change and if xfail testcase is by mistakly fixed then the outcome should be passed??
And for non_strict_xfail we should have new mark??
TanyaAgarwal28 commentedon Oct 11, 2023
Overview of 4 scenarios for xfail and non_strict_xfail
xfail testcase which is expectedly fail:
import pytest
@pytest.mark.xfail
def test_example_expected_fail():
assert 3 + 3 == 7 # This test is marked as expected fail
xfail testcase which is passed:
import pytest
@pytest.mark.xfail
def test_example_expected_fail():
assert 3 + 3 == 6 # This test is marked as expected fail
non_strict_xfail testcase is passed:
import pytest
@pytest.mark.non_strict_xfail
def test_example_expected_fail():
assert 3 + 3 == 6 # This test is marked as expected fail
nicoddemus commentedon Oct 11, 2023
IMHO what should be done:
pytest.mark.xfailhas astrictparameter, which defaults toFalsetoday. We want to issue a warning that this default will change toTruein the future, but only if thestrictflag has not been passed explicitly.Trueand remove the warning.Is this what you have in mind @RonnyPfannschmidt ?
RonnyPfannschmidt commentedon Oct 11, 2023
Correct, once the default is switched I'd like to ensure xpass is either a warning or a failure in any case
TanyaAgarwal28 commentedon Oct 12, 2023
okay Got it
Thanks
Make xpass a failure again pytest-dev#11467
Make xpass a failure again pytest-dev#11467
Make xpass a failure again pytest-dev#11467
Make xpass a failure again pytest-dev#11467
Make xpass a failure again pytest-dev#11467
Make xpass a failure again pytest-dev#11467
bluetech commentedon Nov 5, 2025
We now have strict mode which enables
strict_xfail. Maybe that's enough to close this issue?RonnyPfannschmidt commentedon Nov 5, 2025
this issue is about making it default to true