Don't await tasks when a scope is closed by GeneratorExit - #106

Problem

A common pattern is an async generator that holds a scope open across yield:

async def generate():
    async with duet.new_scope() as scope:
        async for x in duet.pmap_aiter(scope, func, items):
            yield x

If the caller stops iterating early (break, or an exception) without calling aclose(), Python closes the generator synchronously when it is garbage collected. That throws GeneratorExit into new_scope, which on main handles it like any other error: it interrupts the remaining tasks and then awaits them. Three things go wrong:

1. A generator can't yield while handling GeneratorExit, so the close fails with an unraisable RuntimeError: async generator ignored GeneratorExit.
2. The abandoned new_scope frame has already set main_task.interruptible = False and never restores it, so the caller's task can no longer be interrupted, e.g. by a timeout: a later await duet.sleep(1) in the caller hangs forever.
3. The interrupted tasks are still attached to the caller's task. When they eventually fail (with ScopeExit), they interrupt the caller, which has moved on.

We hit this with Ctrl-C hangs in a downstream project.

Reproduction

import duet

async def child():
    await duet.AwaitableFuture()  # Waits until interrupted.

async def generate():
    async with duet.new_scope() as scope:
        scope.spawn(child)
        yield 1
        yield 2

async def main():
    gen = generate()
    async for _ in gen:
        break
    del gen                 # main: "RuntimeError: async generator ignored GeneratorExit"
    await duet.sleep(0.05)  # main: hangs, this task is no longer interruptible

duet.run(main)

Fix

• new_scope gets a separate except GeneratorExit: branch. It interrupts the remaining tasks with ScopeExit without awaiting them, then re-raises.
• A new impl.Task.detach(error) sets the task's main_task to a private sentinel, impl._DETACHED, whose interrupt() refuses every interrupt (returns False), and interrupts the task. A detached task still runs until it handles the interrupt, and duet.run still waits for it before returning, but its failure is no longer delivered to the caller.
• Since ScopeExit is a BaseException, an except Exception: in a detached task can't swallow it, including inside the task's own nested scopes (e.g. duet.sleep).

We considered keeping the tasks attached and having the caller ignore the ScopeExit they echo back, but a pending echoed interrupt would make Task.interrupt drop a real one (e.g. a timeout) arriving in the same tick.

Behaviour change: with await agen.aclose(), the generator's remaining tasks are now interrupted but finish in the background, instead of before aclose() returns. new_scope can't tell an awaited aclose() from a synchronous close (both throw GeneratorExit at the same point), and awaiting is only safe in the first case. We chose to always detach, so nothing hangs or raises unraisable errors. Detached tasks still have to finish before duet.run returns, but they're no longer bounded by an enclosing scope. If you'd rather handle this differently (e.g. a way to tell an awaited aclose() apart), happy to rework it.

• _DETACHED refuses interrupts like a task that can't be interrupted. With Don't drop SIGINT that a task can't take right away #104, which re-posts a KeyboardInterrupt that a main task refuses, a Ctrl-C that reaches a detached task is then delivered to another task instead of being lost.

Prerequisites: please merge #103 and #104 first. Detached tasks are exactly the kind of tasks no scope waits for, so without #103 a root-task failure in the same tick as a detached task becoming ready makes duet.run hang; without #104 a Ctrl-C that reaches a detached task is lost.

Alternative considered: installing async-generator finalizer hooks (sys.set_asyncgen_hooks) in duet.run, as asyncio does, so that a garbage-collected generator is closed by an awaited aclose() in a new task instead of synchronously. That would let aclose() keep waiting for the scope's tasks, but it means installing and restoring per-thread global hooks in a re-entrant duet.run (and alongside asyncio in the same thread), and the generator's tasks would still finish outside the caller's scope. Detaching keeps the change local to new_scope.

Tests

• New: TestScope.test_async_generator_with_scope_closed_when_caller_fails: a pmap_aiter inside a generator, with the caller raising inside async for. filterwarnings("error::pytest.PytestUnraisableExceptionWarning") makes the unraisable RuntimeError fail the test. Fails on main, passes here.
• New: TestScope.test_async_generator_with_scope_closed_early[del]: synchronous close via del gen; afterwards the caller must still be interruptible and must not be interrupted by the closed scope's child. Fails on main, passes here.
• New: TestScope.test_async_generator_with_scope_closed_early[aclose]: the same with await gen.aclose(). It passes on main too (there aclose() awaits the child), but fails if the fix interrupts the tasks without detaching them, so it covers detach.