Repository navigation
feat(eval_service): return per-item scores in Eval response - #646
Open
erinlimbogoogle wants to merge 3 commits into
Open
erinlimbogoogle wants to merge 3 commits into
erinlimbogoogle wants to merge 3 commits into
Conversation
The Eval RPC with summary_in_response only returned run-level counts
({"total", "scores": {metric: correct}}). Callers that wanted to slice
results by dataset item (e.g. per prompt template) had no per-item data
and were left to approximate it, which produces upward-biased numbers.
_process_results now also returns an item_scores dict, built from the
same scores_df that is written to the scores table:
{item_id: {<comparator>: 0.0..1.0,
"executable": 0|1, # generated SQL ran without error
"golden_failed": bool, # golden SQL failed: zero is unfair
"scorer_errors": [...]}} # comparators that raised
Eval includes it as a top-level "item_scores" key next to "summary".
Interact and AgentInteract discard it; their stream payloads are
unchanged. Multi-trial runs return {} since per-row ids are
"<id>_trial_<n>" and the headline is computed over trials.
score rows now also carry golden_error (BigQuery table uses
ALLOW_FIELD_ADDITION, so the new column is picked up automatically).
This is what lets golden_failed distinguish a dataset/infra zero from
a model zero.
Scores stored to reporters are unchanged; item_scores is built after
reporter.store().
erinlimbogoogle
force-pushed
the
feat/item-scores-in-response
branch
from
October 5, 2026 21:46
e22f23f to
20c2887
Compare
erinlimbogoogle
marked this pull request as ready for review
October 6, 2026 15:28
erinlimbogoogle
requested review from
IsmailMehdi,
helloeve and
prernakakkar-google
as code owners
October 6, 2026 15:28
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Return per-item scores in Eval response
With summary_in_response, Eval only returned run-level counts. Callers wanting a per-item breakdown (e.g. by prompt template) had to approximate it from the headline, which is upward-biased.
_process_results now also returns item_scores, built from the same scores_df that goes to the scores table, and Eval includes it top-level:
{"job_id": "...", "summary": {...},
"item_scores": {"400": {"exact_match": 1.0, "llmrater": 1.0, "executable": 1.0, "golden_failed": false}}}
• executable — generated SQL ran without a DB error (same signal as the headline metric).
• golden_failed — golden SQL errored, so the zero is a dataset/infra problem, not a model one. Requires the new golden_error field on score rows (BQ picks it up via ALLOW_FIELD_ADDITION).
• scorer_errors — comparators that raised.
Interact / AgentInteract payloads are unchanged. Multi-trial runs return {} (per-row ids are _trial_; a per-item view is ambiguous). Reporter output is unchanged. ~40 KB for a 460-item run.