Visitar URL original
Behavioural decontamination, schema 2: an own input a reference did n… · trestoncuzzort/dawnr@26564bc · GitHub
Skip to content

Commit 26564bc

Browse files
Behavioural decontamination, schema 2: an own input a reference did not answer leaves the count, and no pair is left with neither a verdict nor a reading
The review of 2026-09-25 found that verdict_of contradicted the rule it wrote into the policy: the rule said an input a reference did not answer in time says nothing, but a duplicate needed own_agreed == own_inputs, where own_inputs counted the timed-out own inputs. One 1 s CPU timeout on an own input (find_lucas, the naive recursion, on an AtCoder problem's N=86) made a duplicate impossible whatever the other 105 inputs said, and eight pairs that agreed everywhere both references answered, with no difference anywhere, stayed admissible to every future corpus as "insufficient": Lucas numbers, binomial coefficients, Catalan numbers, a remainder. This commit measures again under a corrected rule and replaces the checked-in policy. What changed, each with a test that failed before: - An own input a reference did not answer, not in time, not within its recursion or memory limit (RecursionError and MemoryError are now scored like a timeout), or a t value the other problem's argument shape cannot hold, leaves the own-input count (own_unanswered); a duplicate needs every answered own input agreed, at least one of them the held-out problem's own, and the draws as before. HyClone, https://arxiv.org/html/2508.01357v1, drops such inputs the same way ("retaining only those producing valid outputs"); we stay stricter on a one-sided raise, which remains a difference. The unrenderable case had been counted as a one-sided raise; no pair carries such a witness now (25 did). - A reference that stops answering one shape's draws in a pair is still asked the other shape's, and an answer already given is never skipped; the first version skipped every draw after two timeouts anywhere, so a held-out reference that could not answer a train problem's two large examples never saw its own 50 draws. The recursion limit is pinned per call, so a corpus solution that raises it at load time no longer changes what a later reference in the same worker can answer (find_lucas(-10) timed out in one worker and raised at once in another). - A pair the rule cannot decide is "undecided", listed with both statements and why, and `write` refuses to produce the policy until each has a reading by hand (--read-same / --read-different, recorded in the JSON with the counts); a pair read as the same function is excluded and the loader checks exclude_train_ids against duplicates plus those readings, and refuses a file with an unread pair. This run left none to read. - The timeout is a BaseException, so a reference's own `except Exception` cannot swallow it; a late return after a fired timer is a timeout; a bare except that swallows even that makes the wall-clock backstop name the reference on stderr and in a sidecar next to the progress file and exit the worker, as pytest-timeout's thread method does (https://raw.githubusercontent.com/pytest-dev/pytest-timeout/main/README.rst, "often the only sure way to interrupt a hanging test is by terminating the entire process"); the next run refuses that reference by name. The parent takes results with an inactivity limit and names the held-out ids it never got instead of waiting forever on a lost worker, which multiprocessing.Pool would (python/cpython#66587, still open). - A prelude supplies the names LeetCode and Codewars solutions use without importing (gcd, lru_cache, Counter, ...): 12 of pool v5's 32 non-running references now run; 20 v5 references and 31 stdin scripts still do not, each named in the JSON, and the rule says nothing about them. Measured on the lab CPU at nice 19, four workers, 1 s CPU per call: pool v5 48,734 pairs in 994 s, pool v6 66,891 pairs in 896 s with v5's pairs copied. 77 train ids are excluded (72 before): the five that join are apps_3579 choose (203579) against binomial_Coeff 28, apps_4032 solve (204032) against catalan_number 583, apps_4712 lucasnum (204712) and apps_stdin_4681 (304681) against find_lucas 389, and apps_4714 remainder (204714) against find 502; nothing left the list. 16 of the 21 same-task exclusions of 2026-09-21 are rediscovered, 61 are new (13 MBPP, 3 HumanEval, 37 function-shaped APPS, 8 stdin-shaped), 83 duplicate pairs, 85 minimal pairs, 0 undecided, 49 held-out ids have a twin in training (28 and 389 join the 47). The merged policy excludes 82 ids (77 before). The 14 pairs schema 1 called insufficient are all decided: 5 duplicates and 9 minimal pairs. Three of those minimal pairs differ only at 0, outside both problems' stated domains, and are the same function everywhere the problems look: the AtCoder Lucas script 3762772 against find_lucas 389 (N=0), the AtCoder lcm script 304176 against lcm 876 (lcm(0, 11)), and sum_of_squares 201645 against get_Min_Squares 325 (n=0). The rule keeps them, as it keeps 76 against 347; they belong on the hand list of 2026-09-21. Dev ids 199, 372 and 955 stay on the exclusion list; the dev draw and its test still read the 2026-09-21 file alone. The count assertions in t/test_r12_problem_id.py move to 77 behavioural, 16 in both lists, 82 merged, 49 held-out ids with a twin. Callers not edited here: t/r12_data_queue.sh step_build's done-marker inputs should include t/decontamination-behavioural-2026-09-25.json, and its dev-ids step and t/test_r12_data_queue.py's dev-id guard should read the merged policy; t/score_heldout.py can report held-out scores split by behavioural_overlap_eval_ids.
1 parent 75b6a07 commit 26564bc

6 files changed

Lines changed: 2922 additions & 1756 deletions

‎t/DECONTAMINATION-BEHAVIOURAL-2026-09-25.md‎

Lines changed: 160 additions & 81 deletions
Large diffs are not rendered by default.

‎t/behavioural_decontam.py‎

Lines changed: 500 additions & 95 deletions
Large diffs are not rendered by default.

0 commit comments

Comments
 (0)