Repository navigation
Commit e614321
committed
Behavioural decontamination, schema 2: an own input a reference did not answer leaves the count, and no pair is left with neither a verdict nor a reading
The review of 2026-09-25 found that verdict_of contradicted the rule it
wrote into the policy: the rule said an input a reference did not answer in
time says nothing, but a duplicate needed own_agreed == own_inputs, where
own_inputs counted the timed-out own inputs. One 1 s CPU timeout on an own
input (find_lucas, the naive recursion, on an AtCoder problem's N=86) made a
duplicate impossible whatever the other 105 inputs said, and eight pairs that
agreed everywhere both references answered, with no difference anywhere,
stayed admissible to every future corpus as "insufficient": Lucas numbers,
binomial coefficients, Catalan numbers, a remainder. This commit measures
again under a corrected rule and replaces the checked-in policy.
What changed, each with a test that failed before:
- An own input a reference did not answer, not in time, not within its
recursion or memory limit (RecursionError and MemoryError are now scored
like a timeout), or a t value the other problem's argument shape cannot
hold, leaves the own-input count (own_unanswered); a duplicate needs every
answered own input agreed, at least one of them the held-out problem's own,
and the draws as before. HyClone, https://arxiv.org/html/2508.01357v1,
drops such inputs the same way ("retaining only those producing valid
outputs"); we stay stricter on a one-sided raise, which remains a
difference. The unrenderable case had been counted as a one-sided raise;
no pair carries such a witness now (25 did).
- A reference that stops answering one shape's draws in a pair is still
asked the other shape's, and an answer already given is never skipped; the
first version skipped every draw after two timeouts anywhere, so a
held-out reference that could not answer a train problem's two large
examples never saw its own 50 draws. The recursion limit is pinned per
call, so a corpus solution that raises it at load time no longer changes
what a later reference in the same worker can answer (find_lucas(-10)
timed out in one worker and raised at once in another).
- A pair the rule cannot decide is "undecided", listed with both statements
and why, and `write` refuses to produce the policy until each has a
reading by hand (--read-same / --read-different, recorded in the JSON with
the counts); a pair read as the same function is excluded and the loader
checks exclude_train_ids against duplicates plus those readings, and
refuses a file with an unread pair. This run left none to read.
- The timeout is a BaseException, so a reference's own `except Exception`
cannot swallow it; a late return after a fired timer is a timeout; a bare
except that swallows even that makes the wall-clock backstop name the
reference on stderr and in a sidecar next to the progress file and exit
the worker, as pytest-timeout's thread method does
(https://raw.githubusercontent.com/pytest-dev/pytest-timeout/main/README.rst,
"often the only sure way to interrupt a hanging test is by terminating the
entire process"); the next run refuses that reference by name. The parent
takes results with an inactivity limit and names the held-out ids it
never got instead of waiting forever on a lost worker, which
multiprocessing.Pool would (python/cpython#66587,
still open).
- A prelude supplies the names LeetCode and Codewars solutions use without
importing (gcd, lru_cache, Counter, ...): 12 of pool v5's 32 non-running
references now run; 20 v5 references and 31 stdin scripts still do not,
each named in the JSON, and the rule says nothing about them.
Measured on the lab CPU at nice 19, four workers, 1 s CPU per call: pool v5
48,734 pairs in 994 s, pool v6 66,891 pairs in 896 s with v5's pairs copied.
77 train ids are excluded (72 before): the five that join are apps_3579
choose (203579) against binomial_Coeff 28, apps_4032 solve (204032) against
catalan_number 583, apps_4712 lucasnum (204712) and apps_stdin_4681 (304681)
against find_lucas 389, and apps_4714 remainder (204714) against find 502;
nothing left the list. 16 of the 21 same-task exclusions of 2026-09-21 are
rediscovered, 61 are new (13 MBPP, 3 HumanEval, 37 function-shaped APPS, 8
stdin-shaped), 83 duplicate pairs, 85 minimal pairs, 0 undecided, 49
held-out ids have a twin in training (28 and 389 join the 47). The merged
policy excludes 82 ids (77 before). The 14 pairs schema 1 called
insufficient are all decided: 5 duplicates and 9 minimal pairs. Three of
those minimal pairs differ only at 0, outside both problems' stated domains,
and are the same function everywhere the problems look: the AtCoder Lucas
script 3762772 against find_lucas 389 (N=0), the AtCoder lcm script 304176
against lcm 876 (lcm(0, 11)), and sum_of_squares 201645 against
get_Min_Squares 325 (n=0). The rule keeps them, as it keeps 76 against 347;
they belong on the hand list of 2026-09-21. Dev ids 199, 372 and 955 stay on
the exclusion list; the dev draw and its test still read the 2026-09-21 file
alone.
The count assertions in t/test_r12_problem_id.py move to 77 behavioural, 16
in both lists, 82 merged, 49 held-out ids with a twin. Callers not edited
here: t/r12_data_queue.sh step_build's done-marker inputs should include
t/decontamination-behavioural-2026-09-25.json, and its dev-ids step and
t/test_r12_data_queue.py's dev-id guard should read the merged policy;
t/score_heldout.py can report held-out scores split by
behavioural_overlap_eval_ids.1 parent 4e9400f commit e614321
6 files changed
Lines changed: 2922 additions & 1756 deletions
Large diffs are not rendered by default.
Large diffs are not rendered by default.
0 commit comments