fix(hle): compute cerr at attempt level for epochs > 1 - #2132

Thanks — both points addressed in 17c26508:

1. attempt_preserving_mean's parameter renamed to to_float (no longer shadows the module-level value_to_float import, matching cerr's pattern).
2. Added test_cerr_skips_none_attempts: 100 samples each with one scored attempt and one None attempt → cerr reflects only the scored attempts (1.0), pinning the unscored-accounting behavior.

HLE suite 19 passed, ruff/mypy clean.

Claude Code Review

Summary

Reviewed the HLE attempt-level cerr change (metrics.py, judge.py, hle.py, tests, README/changelog, version bump). The core mechanism is sound and the version bump, package changelog fragment, and README changelog entry are all present. Four issues below, the first of which is a claim in the changelog that the code does not support.

Issues Found

Scoring and Metrics — documented behaviour not implemented

Issue: Both the package changelog and the reducer docstring claim this PR fixes unscored accounting, but nothing consumes the None entries, and the behaviour for unscored epochs is byte-for-byte identical to main.

Trace a sample with epochs [scored(CORRECT, 90), unscored]:

• Before: mean skips the root-NaN epoch → reduced {"score": 1.0, "confidence": 90} → cerr sees one pair (1.0, 0.9).
• After: attempts = [{"score": 1.0, "confidence": 90.0}, None] → the isinstance(attempt, dict) guard in _score_confidence_pairs skips the None → cerr sees one pair (1.0, 0.9).

So changelog.d/20260812_000000_hle_attempt_preserving_reducer.md:3 ("...fixing distorted calibration and unscored accounting") and metrics.py:125-126 ("unscored epochs are recorded as None so downstream metrics can count them") both describe capability that does not exist. test_cerr_skips_none_attempts pins the opposite — that they are not counted. The None stash is currently write-only machinery.

Location: changelog.d/20260812_000000_hle_attempt_preserving_reducer.md:3, src/inspect_evals/hle/metrics.py:125-126

Recommendation: Pick one: either drop the "unscored accounting" claim from the changelog and reword the docstring to say the None placeholders preserve positional alignment (and are skipped by cerr), or actually surface an unscored-attempt count as a metric. Note the calibration fix alone already justifies the 4-B bump, so no version consequence either way.

Scoring and Metrics — cerr reports perfect calibration on no data

Issue: metrics.py:77-78 returns 0.0 when pairs is empty. 0.0 is the "perfectly calibrated" end of the scale, so an eval where every sample was unscored (all grader calls failed) now reports cerr 0.000 — indistinguishable from a flawless run. On main this case raised ZeroDivisionError, which was worse as a crash but at least not silently wrong. This path is also untested.

Location: src/inspect_evals/hle/metrics.py:77-78

Recommendation: Return float("nan") for the no-data case (metrics render NaN as absent rather than as a value), and add a unit test covering an all-None-attempts sample list.

Code Quality — unreachable branch

Issue: if not bins: return 0.0 at metrics.py:90-91 cannot fire. Past the if not pairs guard, len(pairs) >= 1, so BIN_SIZE = min(100, len(pairs)) >= 1 and len(confidence) // BIN_SIZE >= 1 for every input (when len(pairs) < 100, BIN_SIZE == len(pairs) and the floor division is exactly 1). Dead code implying a reachable state that isn't.

Location: src/inspect_evals/hle/metrics.py:90-91

Recommendation: Remove it.

Documentation, Environments, and Tooling — README Scoring section not updated

Issue: README.md:18-20 still says only "Accuracy and calibration error are reported over the dataset." Two user-visible facts introduced here are documented nowhere a user will look: (1) at --epochs > 1, cerr is now computed over n_samples × n_epochs attempts while accuracy/stderr remain over reduced samples, so the two headline metrics no longer share a denominator; (2) passing --epochs-reducer replaces the task-pinned reducer, at which point cerr silently reverts to the epoch-averaged semantics this PR exists to fix. The metric cannot detect that override (it has no visibility into the epoch count), so documentation is the only mitigation available.

Location: src/inspect_evals/hle/README.md:18-20

Recommendation: Add a short paragraph to the Scoring section covering both points. The changelog entry and PR body are not a substitute — a user reading the README to interpret their cerr number will not find either.

Notes

• test_attempt_level_cerr_end_to_end: the cerr == 0.4 assertion at tests/hle/test_epoch_compatibility.py:160 does not discriminate the attempt path from the fallback — the reduced value {"score": 0.5, "confidence": 90} yields |0.9 - 0.5| = 0.4 too. The docstring acknowledges this and compensates with the _score_confidence_pairs spy, so coverage is real, but it makes the test dependent on a private function name. A fixture where attempt-level and reduced cerr differ numerically (as test_cerr_uses_attempt_level_pairs does at unit level) would let the e2e test assert on the metric alone.
• test_hle_epoch_compatibility's docstring ("HLE stores confidence in Score.value... which is properly reduced across epochs") now describes the fallback path rather than how the real HLE task behaves, since that test builds its own Task without the pinned reducer. Worth a one-line note in the docstring so a future reader doesn't take it as the reference behaviour.
• Delegating to mean_score rather than reimplementing the mean is the right call — it keeps NaN handling and the answer/explanation carry rules aligned with inspect, and the new dict is built fresh so scores[0].metadata is not mutated.
• CI: Mypy (3.11/3.12/3.13), README check, and Render docs were still pending at review time. Worth confirming mypy is clean — cerr_value is annotated float but accumulates np.float64 from np.square(...).

Reviewer Feedback Status

No human reviewer feedback is unaddressed.

This is an automatic review performed by Claude Code. Any issues raised here should be fixed or justified, but a human review is still required in order for the PR to be merged.

_{Maintainers: comment /claude <instruction> on this PR and Claude will push a fix. To batch multiple changes, submit a review with body /claude and inline comments — Claude will address them all in one run. Single inline comments starting with /claude also work.}