Visitar URL original
🐛 Bug Report: Qdrant async spans end before the operation runs · Issue #4658 · traceloop/openllmetry · GitHub
Skip to content

🐛 Bug Report: Qdrant async spans end before the operation runs #4658

Open

Description

@GaliFFun

Which component is this bug for?

Qdrant Instrumentation

📜 Description

Awaited AsyncQdrantClient operations produce spans that end before the operation executes. This makes request durations nearly zero, loses exceptions raised during the await, and attaches spans created inside the request to the caller instead of the Qdrant span.

The instrumentor registers the synchronous _wrap callback for both client classes. For a coroutine method, wrapped(*args, **kwargs) returns the coroutine immediately; the wrapper marks that truthy object as successful and exits the span context. wrapt invokes the supplied callback directly; it does not automatically await the coroutine or extend the context manager's lifetime.

👟 Reproduction steps

  1. Use opentelemetry-instrumentation-qdrant 0.62.4 and qdrant-client 1.11.3.
  2. Start a local Jaeger instance accepting OTLP/HTTP on http://127.0.0.1:4318/v1/traces.
  3. Save the script below as reproduce.py, then run python reproduce.py before. The before argument only labels the exported traces; run against the unchanged instrumentation.
  4. Inspect the success and error traces in Jaeger, or the exported evidence/before-spans.json.

The script uses the real AsyncQdrantClient.upsert method with httpx.MockTransport: it waits 250 ms, then returns either HTTP 200 or HTTP 500. No Qdrant server or credentials are needed.

Standalone reproducer
import argparse
import asyncio
import importlib.metadata as metadata
import json
from pathlib import Path

import httpx
from opentelemetry.instrumentation.qdrant import QdrantInstrumentor
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from qdrant_client import AsyncQdrantClient, models
from qdrant_client.http.exceptions import UnexpectedResponse


parser = argparse.ArgumentParser()
parser.add_argument('phase', choices=['before', 'after'])
parser.add_argument('--output-dir', type=Path, default=Path(__file__).parent / 'evidence')
args = parser.parse_args()
output = args.output_dir
output.mkdir(parents=True, exist_ok=True)
provider = TracerProvider(resource=Resource.create({
    'service.name': f'qdrant-async-spans-{args.phase}',
    'demo.transport': 'httpx.MockTransport; synthetic 250 ms latency',
}))
memory = InMemorySpanExporter()
provider.add_span_processor(SimpleSpanProcessor(memory))
provider.add_span_processor(SimpleSpanProcessor(OTLPSpanExporter(endpoint='http://127.0.0.1:4318/v1/traces')))
tracer = provider.get_tracer('qdrant-async-spans-reproduction')
QdrantInstrumentor().instrument(tracer_provider=provider)


async def scenario(fail):
    async def handler(request):
        with tracer.start_as_current_span('HTTP transport (simulated 250 ms)'):
            await asyncio.sleep(0.25)
            if fail:
                return httpx.Response(500, json={'status': {'error': 'controlled demo failure'}})
            return httpx.Response(200, json={
                'result': {'operation_id': 1, 'status': 'completed'}, 'status': 'ok', 'time': 0.25,
            })

    client = AsyncQdrantClient(url='http://localhost:6333', transport=httpx.MockTransport(handler))
    try:
        try:
            with tracer.start_as_current_span(f'Async upsert: {"error" if fail else "success"}'):
                await client.upsert(collection_name='demo_collection', points=[models.PointStruct(id=1, vector=[0.1, 0.2])])
        except UnexpectedResponse:
            pass
    finally:
        await client.close()


async def main():
    await scenario(False)
    await scenario(True)


try:
    asyncio.run(main())
    provider.force_flush()
    spans = [json.loads(span.to_json()) for span in memory.get_finished_spans()]
    (output / f'{args.phase}-spans.json').write_text(json.dumps(spans, indent=2) + '\n')
    summaries = []
    for span in memory.get_finished_spans():
        summaries.append({
            'name': span.name, 'trace_id': f'{span.context.trace_id:032x}',
            'span_id': f'{span.context.span_id:016x}',
            'parent_id': f'{span.parent.span_id:016x}' if span.parent else None,
            'duration_ms': round((span.end_time-span.start_time)/1e6, 3),
            'status': span.status.status_code.name,
        })
    print(json.dumps({'phase': args.phase, 'versions': {n: metadata.version(n) for n in
        ['qdrant-client', 'wrapt', 'opentelemetry-sdk']}, 'spans': summaries}, indent=2))
    (output / f'{args.phase}-summary.json').write_text(json.dumps(summaries, indent=2) + '\n')
finally:
    QdrantInstrumentor().uninstrument()
    provider.shutdown()

👍 Expected behavior

The Qdrant span stays open and current while the operation is awaited. Its duration covers the request, transport spans are its children, and an exception raised during the await sets ERROR and records an exception event.

Calling an async method without awaiting it should not export a completed successful operation span.

👎 Actual Behavior with Screenshots

In the reproduction:

  • Success: the Qdrant span lasts 15 microseconds while the simulated transport takes about 251 ms. The transport span is its sibling.
  • HTTP 500: the Qdrant span lasts 14 microseconds and remains OK with no exception event, even though the awaited operation raises UnexpectedResponse.
  • Merely creating the coroutine exports a completed OK span before any request runs.

Jaeger screenshots

Captured from Jaeger 2.6.0 with the real client and instrumentor, using httpx.MockTransport with a controlled 250 ms delay. These show instrumentation lifecycle behavior, not live Qdrant server latency.

Success, before: the Qdrant span ends after 15 microseconds; the transport is a sibling.

Success before the fix

Success, with the proposed fix: the Qdrant span lasts 262.125 ms and parents the transport span.

Success after the fix

HTTP 500, before: the Qdrant span ends after 14 microseconds with status OK and no exception event.

Error before the fix

HTTP 500, with the proposed fix: the Qdrant span lasts 261.740 ms, has status ERROR, and records one exception event.

Error after the fix

🤖 Python Version

3.11.9 on macOS arm64; regression checks also run on Python 3.10 and 3.12.

📃 Provide any additional context for the Bug.

Reproduced against upstream commit 8d9c93f37b352131f5871dcb6ecf5e79c19aeb40, instrumentation 0.62.4, Qdrant client 1.11.3, OpenTelemetry SDK 1.45.1, and wrapt 2.5.1. Also reproduced with wrapt 1.17.3.

An async wrapper should await the operation inside the span context. Wrapper selection needs to inspect the actual method: AsyncQdrantClient also exposes synchronous upload methods. This is separate from missing method coverage and compatibility with newer Qdrant client releases.

👀 Have you spent some time to check if this bug has been raised before?

  • I checked and didn't find similar issue

Are you willing to submit PR?

Yes I am willing to submit a PR!

Activity

  1. GaliFFun commented on Oct 25, 2026

    @GaliFFun
    Author

    The fix is ready for review in #4659. It keeps async Qdrant spans open through the awaited operation and adds regression coverage for duration, context propagation, exceptions, cancellation, suppression, and the async client's synchronous upload methods. Before/after Jaeger screenshots are included in this issue and the PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions