Repository navigation
Repeated cancellation can interrupt root cleanup and leave HTTP connections in use #8106
Description
Describe the bug
When a task consuming an ADK run is cancelled repeatedly, Runner._cleanup_root_task cancels the root task and then awaits it directly. A subsequent cancellation of the consuming task can propagate into the root task again and interrupt its resource cleanup.
If the root task is using httpx/httpcore, this can leave a connection marked in use. In a long-lived application, affected connections can accumulate until the pool is exhausted.
Steps to reproduce
Install google-adk, httpx, and httpcore, then run this script. It uses a local HTTP server and does not require Gemini credentials.
import asyncio
from contextlib import suppress
import google.adk
import httpcore
import httpx
from google.adk.agents import Agent
from google.adk.runners import Runner
from google.adk.sessions import InMemorySessionService
async def main():
connected = asyncio.Event()
received = asyncio.Event()
disconnected = asyncio.Event()
async def serve(reader, writer):
try:
await reader.readuntil(b"\r\n\r\n")
writer.write(
b"HTTP/1.1 200 OK\r\n"
b"Transfer-Encoding: chunked\r\n\r\n"
b"3\r\none\r\n"
)
await writer.drain()
connected.set()
await reader.read()
disconnected.set()
finally:
writer.close()
await writer.wait_closed()
server = await asyncio.start_server(serve, "127.0.0.1", 0)
port = server.sockets[0].getsockname()[1]
runner = Runner(
agent=Agent(name="test"),
app_name="test",
session_service=InMemorySessionService(),
)
async with httpx.AsyncClient(timeout=10) as client:
async def consume():
async with client.stream(
"GET", f"http://127.0.0.1:{port}/"
) as response:
async for _ in response.aiter_bytes():
received.set()
root = asyncio.create_task(consume())
await asyncio.wait_for(connected.wait(), 2)
await asyncio.wait_for(received.wait(), 2)
# Isolate the same cleanup path used by Runner.run_async().
owner = asyncio.create_task(
runner._cleanup_root_task(root, "test")
)
await asyncio.sleep(0)
# Simulate repeated cancellation of the consuming task.
for _ in range(100):
if owner.done():
break
owner.cancel()
await asyncio.sleep(0)
with suppress(asyncio.CancelledError):
await owner
await asyncio.sleep(0.1)
pool = client._transport._pool
busy = sum(not connection.is_idle() for connection in pool.connections)
print(
f"google-adk={google.adk.__version__} "
f"httpx={httpx.__version__} httpcore={httpcore.__version__}"
)
print(
f"pool_in_use={busy}, "
f"server_saw_close={disconnected.is_set()}"
)
await runner.close()
server.close()
await server.wait_closed()
asyncio.run(main())The reproduction deliberately calls a private method to isolate the cleanup path. We also reproduced the behavior through an HTTP route consuming Runner.run_async().
Expected behavior
Cancellation of the consuming task should not interrupt the root task's resource cleanup. The HTTP connection should be released, and the server should observe the socket close.
Observed behavior
With ADK 2.13.0 and 2.14.0, the minimal reproduction produced the following result in three runs of each version:
pool_in_use=1, server_saw_close=False
In a separate controlled HTTP-route reproduction, 20 cancelled streams left 20 connections in use.
Environment details
- ADK: 2.13.0 and 2.14.0
- Python: 3.13.5
- OS: macOS
- httpx: 0.28.1
- httpcore: 1.0.9
- anyio: 4.14.2
- LiteLLM: no
- Model: not applicable to the minimal reproduction
Possible cause and approach
_cleanup_root_task awaits the cancelled root task directly. Repeated cancellation of the consuming task can therefore cancel the root task again while httpcore is closing its response.
A local prototype cancels the root task immediately and shields the subsequent cleanup from repeated cancellation. With that prototype, the minimal reproduction produced this result in three runs:
pool_in_use=0, server_saw_close=True
The prototype retains pending cleanup tasks and waits for them during runner shutdown. We would welcome maintainer feedback on the approach and would be happy to contribute a PR.
Additional context
There are related httpcore reports: #658 and #830.
We also found a separate early-stream-close issue in google-genai and plan to report it independently.
AI assistance was used to help investigate and draft this report; the reproduction and outputs were run locally.
Activity
surajksharma07 commentedon Nov 12, 2026
@auwilly. Reproduced this on 2.14.0 and current main through run_async() with an LlmAgent: 20 cancelled streams left 20 connections stuck in the pool.
Your approach looks right. Shielding the await in _cleanup_root_task and having close() wait on the pending cleanups took it to 0/20 locally, and caller cancellation plus root exceptions still propagate. Could you try that on your HTTP-route setup as a workaround and confirm it holds there? Please test it thoroughly before you open the PR.
One gap: a custom BaseAgent root still leaks, since it runs inline in the caller’s task and never reaches _cleanup_root_task.
@Binary-Refinery a regression test that cancels the caller twice would be great. Maybe sync with @auwilly so it can go in the same PR.
auwilly commentedon Nov 12, 2026
Thanks both! I’ll test the proposed shielding approach in our HTTP-route setup and report back before opening a PR.
@Binary-Refinery, the regression test would be very helpful. Would you be happy to share a branch or patch so we can include it in the same PR?
auwilly commentedon Nov 12, 2026
Thanks @Binary-Refinery for preparing the regression test and sharing the commit! I’ll include it in the fix branch and preserve your attribution if the commits are squashed.
@surajksharma07, I tested the shielding prototype locally against upstream main at df9f9d0, using our HTTP route with a fake Gemini server:
I will not fix the custom BaseAgent case as it would complexity my PR. I'll open the PR, most likely tomorrow, thanks for answering me swiftly today.
I'll keep you posted !
surajksharma07 commentedon Nov 14, 2026
Thanks @auwilly, those numbers line up with what I saw, and the cancelled-while-waiting-for-headers case is a good one to have covered. Leaving the custom BaseAgent case out is fine, we can track that separately.
A few things for the PR. _cleanup_root_task is also used by run_live in live/_runner_utils.py, where it runs twice in a row (agent_task, then queue_task). If the caller gets cancelled during the first one, queue_task never gets cancelled, so it’s worth cancelling both before the shielded wait.
For the pending cleanups, drop finished tasks with a done callback, read their exceptions so asyncio doesn’t log “never retrieved”, and put a timeout on the wait in close() so a stuck root task can’t hang shutdown.
Main (2.14.1) still has the old cleanup, so this isn’t fixed yet. Please add “Fixes #8106” to the PR so it links here, and run the full runner/live tests along with @Binary-Refinery’s test.