Visitar URL original
Add opt-in Bayes@N summaries to score reporting by mohsenhariri · Pull Request #162 · LiveCodeBench/LiveCodeBench · GitHub
Skip to content

Add opt-in Bayes@N summaries to score reporting - #162

Open
mohsenhariri wants to merge 1 commit into
LiveCodeBench:mainfrom
mohsenhariri:feat/bayes-at-n
Open

mohsenhariri wants to merge 1 commit into
LiveCodeBench:mainfrom
mohsenhariri:feat/bayes-at-n

Conversation

@mohsenhariri

Copy link
Copy Markdown

Add --bayes_at_n to the documented compute_scores command. It reports the posterior mean and posterior standard deviation of average success probability over the selected code-generation questions. --bayes_output_file saves overall and nonempty difficulty-group summaries, including cohort sizes, as JSON.

python -m lcb_runner.evaluation.compute_scores --eval_all_file results_eval_all.json \
    --bayes_at_n --bayes_output_file posterior.json

The implementation uses each question's repeated binary graded_list after the existing date/platform filters. It follows the binary, uniform-prior case of Bayes@N, with one Beta(1, 1) prior per question and no new runtime dependency. Each subset's uncertainty is recomputed from its raw rows. Complete files from resumed runs therefore use one corpus calculation, without averaging chunk standard deviations. Runner aggregate metrics and default score reporting are unchanged.

The option rejects empty selections, missing/duplicate question IDs, missing or nonbinary grades, unequal sample counts and unsupported scenarios. It assumes independent generations under a fixed configuration and a complete intended cohort. Historical observations, categorical rubrics and dependent self-repair trials are outside this scope.

Validation in a dedicated Python 3.11 environment:

  • python -m pytest tests/test_bayes.py -q: 26 passed.
  • Public CLI tests cover filters, difficulty summaries, JSON output, input preservation, defaults and invalid configurations.
  • Independent exact moments distinguish pooled counts from per-question inference; combined old/new evidence matches a one-shot calculation.
  • python -m compileall -q lcb_runner/evaluation/bayes.py lcb_runner/evaluation/compute_scores.py tests/test_bayes.py: passed.

Existing pass@k reporting emits NumPy empty-slice warnings for empty difficulty subsets; the posterior report omits empty difficulty groups. 24 seeded workloads matched exact Fraction moments and the pinned Scorio reference within 1e-14. Independent review completed. I am a coauthor of the Bayes@N paper.

@mohsenhariri

Copy link
Copy Markdown
Author

Could a maintainer review compute_scores as the entry point for this option and advise on any additional checks needed? GitHub currently reports no checks or workflow runs for head 5f219f5.

The flag uses each question's binary graded_list after the date/platform filters and recomputes uncertainty for each difficulty subset. The recorded local validation passed 26 tests covering the CLI, filters and JSON output, plus 24 exact/reference workloads. Self-repair trials are excluded from this integration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant