Visitar URL original
fix(vertexai): record async stream text after the stream ends by 1fanwang · Pull Request #4542 · traceloop/openllmetry · GitHub
Skip to content

fix(vertexai): record async stream text after the stream ends - #4542

Open
1fanwang wants to merge 2 commits into
traceloop:mainfrom
1fanwang:1fanwang/vertexai-async-stream-accumulated-text
Open

1fanwang wants to merge 2 commits into
traceloop:mainfrom
1fanwang:1fanwang/vertexai-async-stream-accumulated-text

Conversation

@1fanwang

@1fanwang 1fanwang commented Oct 2, 2026 •

Copy link
Copy Markdown
  • I have added tests that cover my changes.
  • If adding a new instrumentation or changing an existing one, I've added screenshots from some observability platform showing the change. Not applicable: a local async generator exercises the same code path without a hosted backend.
  • PR name follows conventional commits format: feat(instrumentation): ... or fix(instrumentation): ....
  • (If applicable) I have updated the documentation accordingly.

Problem

At current main, commit be49830, Vertex AI async streaming calls can finish with missing or incorrect response text in the recorded span. The streaming wrapper does accumulate the full text while it drains the stream, but the code that finalizes the span gets the exhausted generator object instead of the text that was just collected.

Fix

Pass the accumulated async stream text into the response-finalization path once the drain completes, and add a regression test around the real async streaming wrapper so this does not regress.

Testing Done

The red and green probe uses a local async generator substitute rather than a live Vertex AI account. This bug is pure response-plumbing logic inside the instrumentation layer, so the substitute is enough to drive the real code path honestly. The substitute yields chunk objects with the same fields this path reads from the SDK.

Red

Command:

cd packages/opentelemetry-instrumentation-vertexai
PYTHONWARNINGS=ignore::FutureWarning uv run python /tmp/vertexai_async_stream_probe.py

Output before this patch:

observed ['hello ', 'world']
completion_attr_type async_generator
completion_attr_repr <async_generator object _stream_chunks at 0x10cf37ec0>
ended True

Green

Command:

cd packages/opentelemetry-instrumentation-vertexai
PYTHONWARNINGS=ignore::FutureWarning uv run python /tmp/vertexai_async_stream_probe.py

Output with this patch:

observed ['hello ', 'world']
completion_attr_type str
completion_attr_repr 'hello world'
ended True
Reproducer source: vertexai_async_stream_probe.py
import asyncio
from types import SimpleNamespace
from unittest.mock import patch
from opentelemetry.instrumentation.vertexai import _abuild_from_streaming_response
from opentelemetry.semconv._incubating.attributes import gen_ai_attributes as GenAIAttributes

class RecordingSpan:
    def __init__(self):
        self.attributes = {}
        self.statuses = []
        self.ended = False
    def is_recording(self):
        return True
    def set_attribute(self, name, value):
        self.attributes[name] = value
    def set_status(self, status):
        self.statuses.append(status)
    def end(self):
        self.ended = True

def _chunk(text, usage_metadata=None):
    return SimpleNamespace(text=text, usage_metadata=usage_metadata)

async def _stream_chunks():
    usage_metadata = SimpleNamespace(total_token_count=7, candidates_token_count=4, prompt_token_count=3, cached_content_token_count=None)
    yield _chunk('hello ')
    yield _chunk('world', usage_metadata=usage_metadata)

async def main():
    span = RecordingSpan()
    with patch('opentelemetry.instrumentation.vertexai.should_emit_events', return_value=False), patch('opentelemetry.instrumentation.vertexai.span_utils.should_send_prompts', return_value=True):
        response = _abuild_from_streaming_response(span=span, event_logger=None, response=_stream_chunks(), llm_model='gemini-2.5-pro')
        observed = [chunk.text async for chunk in response]
    print('observed', observed)
    print('completion_attr_type', type(span.attributes[f"{GenAIAttributes.GEN_AI_COMPLETION}.0.content"]).__name__)
    print('completion_attr_repr', repr(span.attributes[f"{GenAIAttributes.GEN_AI_COMPLETION}.0.content"]))
    print('ended', span.ended)

asyncio.run(main())

Signed-off-by: 1fanwang <1fannnw@gmail.com>
@coderabbitai

coderabbitai Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Important

Review skipped

Review was skipped as selected files did not have any reviewable changes.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: fd62a23c-5661-4456-903f-32efeabc000d
Commits

Reviewing files that changed from the base of the PR and between 074510e and 08a55f5.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review

No actionable comments were generated in the recent review.

Walkthrough

The async streaming handler now passes accumulated response text to handle_streaming_response. A test verifies that chunks are yielded in order, the combined text is recorded as completion content, and the span ends.

Changes: Async streaming completion

Layer / File(s) Summary
Accumulated response handling
packages/opentelemetry-instrumentation-vertexai/opentelemetry/instrumentation/vertexai/__init__.py, packages/opentelemetry-instrumentation-vertexai/tests/test_async_streaming.py
The handler passes accumulated response text to handle_streaming_response instead of the response iterator. The test checks chunk order, recorded completion content, and span completion.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Merge Risk: Minimal · previous review 07451

The async stream records its accumulated response text while continuing to yield chunks and end the span. No actionable merge-blocking risk remains.

Pre-merge checks: Passed 4

Check name Status Explanation
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly describes the main change: recording accumulated text after the Vertex AI async stream ends.
Finishing Touches

🧪 Generate unit tests (beta)

  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: c0b8fa38-5408-4df9-82ef-326ff2f0945b

📥 Commits

Reviewing files that changed from the base of the PR and between be49830 and 074510e.

📒 Files selected for processing (2)

  • packages/opentelemetry-instrumentation-vertexai/opentelemetry/instrumentation/vertexai/__init__.py
  • packages/opentelemetry-instrumentation-vertexai/tests/test_async_streaming.py

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

…sync-stream-text

Signed-off-by: 1fanwang <1fannnw@gmail.com>
@1fanwang 1fanwang changed the title Record Vertex AI async stream text after the stream ends fix(vertexai): record async stream text after the stream ends Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant