Visitar URL original
Evaluation pairs every ID-less tool call with the last tool response · Issue #8132 · google/adk-python · GitHub
Skip to content

Evaluation pairs every ID-less tool call with the last tool response #8132

Description

@1aifanatic

Describe the bug

Evaluation transcripts with missing function-call IDs pair every tool call with the last ID-less response. get_all_tool_calls_with_responses constructs a dictionary keyed by response ID, so all missing IDs collide under None. The incorrect pairs reach the rubric tool-use judge prompt.

Minimal reproduction

from google.adk.evaluation.eval_case import IntermediateData, get_all_tool_calls_with_responses
from google.genai import types

data = IntermediateData(
    tool_uses=[types.FunctionCall(name='search'), types.FunctionCall(name='book')],
    tool_responses=[
        types.FunctionResponse(name='search', response={'result': 'found'}),
        types.FunctionResponse(name='book', response={'result': 'booked'}),
    ],
)
print([(call.name, response.name) for call, response in get_all_tool_calls_with_responses(data)])

Actual: [('search', 'book'), ('book', 'book')].
Expected: [('search', 'search'), ('book', 'book')].

Scope and proposed fix

This affects restored or hand-authored legacy IntermediateData and ID-less InvocationEvents, not normal Runner traces that supply call IDs. Match missing-ID records by name and occurrence, consuming each response once; retain explicit-ID matching. For multiple calls of the same name, recorded order is the only available correlation information.

Reproduced on main at 12aab4aaa, Python 3.12.10, Windows. A JSON-round-tripped Invocation was also evaluated through the real RubricBasedToolUseV1Evaluator with a scripted external judge; its prompt contained the incorrect pairs. No external API credentials are needed.

Prepared with AI assistance; reproduction and validation were executed locally.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions