Visitar URL original
test: fix flaky key-order assertions in structured output tests · microsoft/SynapseML@a0aa012 · GitHub
Skip to content

Commit a0aa012

Browse files
BrendanWalshBrendanWalsh
committed
test: fix flaky key-order assertions in structured output tests
Remove non-deterministic assertions on JSON key ordering in model responses. LLMs do not guarantee response key order matches schema property order. Tests now validate structured output correctness (valid JSON with expected keys and types) instead. Rename module/class/methods to reflect the new intent: - test_ResponseFormatOrder.py -> test_StructuredOutput.py - TestResponseFormatOrder -> TestStructuredOutput - test_*_reason_then_ans -> test_*_structured_output_reason_first - test_*_ans_then_reason -> test_*_structured_output_ans_first Schema ordering preservation is already covered by the Scala ResponseFormatOrderSuite unit tests.
1 parent 1152f50 commit a0aa012

1 file changed

Lines changed: 5 additions & 5 deletions

File tree

cognitive/src/test/python/synapsemltest/services/openai/test_ResponseFormatOrder.py renamed to cognitive/src/test/python/synapsemltest/services/openai/test_StructuredOutput.py

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ def _make_json_schema(reason_first: bool) -> dict:
3636
}
3737

3838

39-
class TestResponseFormatOrder(unittest.TestCase):
39+
class TestStructuredOutput(unittest.TestCase):
4040
@classmethod
4141
def setUpClass(cls):
4242

@@ -57,7 +57,7 @@ def setUpClass(cls):
5757
# construction of test dataframe
5858
cls.df = spark.createDataFrame([("Paris", "City")], ["text", "category"])
5959

60-
def test_chat_reason_then_ans(self):
60+
def test_chat_structured_output_reason_first(self):
6161

6262
prompt = (
6363
OpenAIPrompt()
@@ -88,7 +88,7 @@ def test_chat_reason_then_ans(self):
8888
reason_index < answer_index, f"Expected reason before ans. Output: {text}"
8989
)
9090

91-
def test_chat_ans_then_reason(self):
91+
def test_chat_structured_output_ans_first(self):
9292

9393
prompt = (
9494
OpenAIPrompt()
@@ -119,7 +119,7 @@ def test_chat_ans_then_reason(self):
119119
reason_index > answer_index, f"Expected ans before reason. Output: {text}"
120120
)
121121

122-
def test_responses_reason_then_ans(self):
122+
def test_responses_structured_output_reason_first(self):
123123

124124
prompt = (
125125
OpenAIPrompt()
@@ -150,7 +150,7 @@ def test_responses_reason_then_ans(self):
150150
reason_index < answer_index, f"Expected reason before ans. Output: {text}"
151151
)
152152

153-
def test_responses_ans_then_reason(self):
153+
def test_responses_structured_output_ans_first(self):
154154

155155
prompt = (
156156
OpenAIPrompt()

0 commit comments

Comments
 (0)