Repository navigation
Add opt-in Bayes@N summaries to score reporting - #162
Open
mohsenhariri wants to merge 1 commit into
Open
mohsenhariri wants to merge 1 commit into
mohsenhariri wants to merge 1 commit into
Conversation
Author
|
Could a maintainer review The flag uses each question's binary |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add
--bayes_at_nto the documentedcompute_scorescommand. It reports the posterior mean and posterior standard deviation of average success probability over the selected code-generation questions.--bayes_output_filesaves overall and nonempty difficulty-group summaries, including cohort sizes, as JSON.python -m lcb_runner.evaluation.compute_scores --eval_all_file results_eval_all.json \ --bayes_at_n --bayes_output_file posterior.jsonThe implementation uses each question's repeated binary
graded_listafter the existing date/platform filters. It follows the binary, uniform-prior case of Bayes@N, with one Beta(1, 1) prior per question and no new runtime dependency. Each subset's uncertainty is recomputed from its raw rows. Complete files from resumed runs therefore use one corpus calculation, without averaging chunk standard deviations. Runner aggregate metrics and default score reporting are unchanged.The option rejects empty selections, missing/duplicate question IDs, missing or nonbinary grades, unequal sample counts and unsupported scenarios. It assumes independent generations under a fixed configuration and a complete intended cohort. Historical observations, categorical rubrics and dependent self-repair trials are outside this scope.
Validation in a dedicated Python 3.11 environment:
python -m pytest tests/test_bayes.py -q: 26 passed.python -m compileall -q lcb_runner/evaluation/bayes.py lcb_runner/evaluation/compute_scores.py tests/test_bayes.py: passed.Existing pass@k reporting emits NumPy empty-slice warnings for empty difficulty subsets; the posterior report omits empty difficulty groups. 24 seeded workloads matched exact Fraction moments and the pinned Scorio reference within
1e-14. Independent review completed. I am a coauthor of the Bayes@N paper.