Turbopack: terminate failed plugin worker threads - #96592

When a worker-thread plugin runtime dies unexpectedly (uncaught exception,
process.exit, OOM), nothing told the Rust worker pool about it: the dead
Worker stayed in loaderWorkers, the operation waiting for its response
hung forever, and every later evaluation could be handed the dead worker,
wedging the dev server's compilation permanently. This is the crash-time
sibling of #96592, which covered evaluation and IPC errors reported over
the channel.

JavaScript now attaches 'error'/'exit' listeners to pooled workers and
reports unexpected deaths through a new workerDied binding (intentional
terminations remove the worker from the map first, so they are never
reported; the thread id is captured at creation since it resets to -1 by
the time 'exit' fires). On the Rust side, handle_worker_death closes the
in-flight task's channel so its pending recv fails and surfaces through
the existing subprocess-crash path in pull_operation (synthesized issue +
disallow_reuse + fresh worker on the next evaluation), removes idle dead
workers from the pool, and fails the oldest pending creation when a worker
dies while booting before worker_created.

Verified end to end with turbopackPluginRuntimeStrategy: 'workerThreads':

- before: a loader calling process.exit(1) hangs the request forever and
  wedges all later compilations;
- after: the request fails in ~1s with "Node.js subprocess crashed while
  evaluating loaders", and the next request is served by a fresh worker in
  ~50ms. A worker that dies between tasks (idle) is reaped the same way.

The new development test covers the crash path: baseline (stock binary)
fails by hanging; candidate fails the evaluation and recovers. The #96592
error-cleanup test still passes.

Validation: pnpm build-all (TS + native), cargo fmt --check, cargo clippy
-p turbopack-node --all-targets -D warnings, cargo test -p turbopack-node
--lib, candidate integration test pass, exact baseline binary fails by
hanging.

Co-Authored-By: Marcos Hernanz <96699542+marcoshernanz@users.noreply.github.com>

When a worker-thread plugin runtime dies unexpectedly (uncaught exception,
process.exit, OOM), nothing told the Rust worker pool about it: the dead
Worker stayed in loaderWorkers, the operation waiting for its response
hung forever, and every later evaluation could be handed the dead worker,
wedging the dev server's compilation permanently. This is the crash-time
sibling of #96592, which covered evaluation and IPC errors reported over
the channel.

JavaScript now attaches 'error'/'exit' listeners to pooled workers and
reports any death while the worker is still registered (intentional
terminations remove the worker from the map first, so they are never
reported; the thread id is captured at creation since it resets to -1 by
the time 'exit' fires). On the Rust side, a dead-worker set records
reaped ids and handle_worker_death classifies the death:

- busy: closes the in-flight task's channel (via a watch channel, which
  cannot lose the wakeup), so its pending recv fails and surfaces through
  the existing subprocess-crash path in pull_operation - synthesized
  issue, disallow_reuse, fresh worker on the next evaluation;
- idle: removed from the idle list and pool statistics;
- created but unassigned (in transit to an acquirer or routed to a
  waiter): total worker count decremented;
- boot failure: the pending creation for the same pool is failed (so
  create_worker errors instead of waiting forever) and the booting
  statistics are balanced - no phantom workers after a boot crash.

Every acquisition and return path checks the dead set so a reaped id is
never handed out again: acquire_worker retries after dead candidates
(idle pop, waiter handoff, fresh boot), and WorkerOperation's drop skips
returning a worker that died while checked out.

Verified end to end with turbopackPluginRuntimeStrategy: 'workerThreads':

- before: a loader calling process.exit(1) hangs the request forever and
  wedges all later compilations;
- after: the request fails in ~1s with "Node.js subprocess crashed while
  evaluating loaders", and the next request is served by a fresh worker
  in ~50ms. A worker that dies between tasks (idle) is reaped the same
  way.

The new development test covers the crash path: baseline (stock binary)
fails by hanging; candidate fails the evaluation and recovers. The #96592
error-cleanup test still passes. Autoreview panel findings addressed:
watch-channel close, dead-set validation on all handoff paths,
options-tagged pending creations, clean-exit reaping.

Validation: pnpm build-all (TS + native), cargo fmt --check, cargo clippy
-p turbopack-node --all-targets -D warnings, cargo test -p turbopack-node
--lib, candidate integration test pass, exact baseline binary fails by
hanging.

Co-Authored-By: Marcos Hernanz <96699542+marcoshernanz@users.noreply.github.com>

When a worker-thread plugin runtime dies unexpectedly (uncaught exception,
process.exit, OOM), nothing told the Rust worker pool about it: the dead
Worker stayed in loaderWorkers, the operation waiting for its response
hung forever, and every later evaluation could be handed the dead worker,
wedging the dev server's compilation permanently. This is the crash-time
sibling of #96592, which covered evaluation and IPC errors reported over
the channel.

JavaScript now attaches 'error'/'exit' listeners to pooled workers and
reports any death while the worker is still registered (intentional
terminations remove the worker from the map first, so they are never
reported; the thread id is captured at creation since it resets to -1 by
the time 'exit' fires). On the Rust side, a dead-worker set records
reaped ids and handle_worker_death classifies the death:

- busy: closes the in-flight task's channel (via a watch channel, which
  cannot lose the wakeup), so its pending recv fails and surfaces through
  the existing subprocess-crash path in pull_operation - synthesized
  issue, disallow_reuse, fresh worker on the next evaluation;
- idle: removed from the idle list and pool statistics;
- created but unassigned (in transit to an acquirer or routed to a
  waiter): total worker count decremented;
- boot failure: the pending creation for the same pool is failed (so
  create_worker errors instead of waiting forever) and the booting
  statistics are balanced - no phantom workers after a boot crash.

Every acquisition and return path checks the dead set so a reaped id is
never handed out again: acquire_worker retries after dead candidates
(idle pop, waiter handoff, fresh boot), and WorkerOperation's drop skips
returning a worker that died while checked out.

Verified end to end with turbopackPluginRuntimeStrategy: 'workerThreads':

- before: a loader calling process.exit(1) hangs the request forever and
  wedges all later compilations;
- after: the request fails in ~1s with "Node.js subprocess crashed while
  evaluating loaders", and the next request is served by a fresh worker
  in ~50ms. A worker that dies between tasks (idle) is reaped the same
  way.

The new development test covers the crash path: baseline (stock binary)
fails by hanging; candidate fails the evaluation and recovers. The #96592
error-cleanup test still passes. Autoreview panel findings addressed:
watch-channel close, dead-set validation on all handoff paths,
options-tagged pending creations, clean-exit reaping.

Validation: pnpm build-all (TS + native), cargo fmt --check, cargo clippy
-p turbopack-node --all-targets -D warnings, cargo test -p turbopack-node
--lib, candidate integration test pass, exact baseline binary fails by
hanging.

Co-Authored-By: Marcos Hernanz <96699542+marcoshernanz@users.noreply.github.com>

When a worker-thread plugin runtime dies unexpectedly (uncaught exception,
process.exit, OOM), nothing told the Rust worker pool about it: the dead
Worker stayed in loaderWorkers, the operation waiting for its response
hung forever, and every later evaluation could be handed the dead worker,
wedging the dev server's compilation permanently. This is the crash-time
sibling of #96592, which covered evaluation and IPC errors reported over
the channel.

JavaScript now attaches 'error'/'exit' listeners to pooled workers and
reports any death while the worker is still registered (intentional
terminations remove the worker from the map first, so they are never
reported; the thread id is captured at creation since it resets to -1 by
the time 'exit' fires). On the Rust side, a dead-worker set records
reaped ids and handle_worker_death classifies the death:

- busy: closes the in-flight task's channel (via a watch channel, which
  cannot lose the wakeup), so its pending recv fails and surfaces through
  the existing subprocess-crash path in pull_operation - synthesized
  issue, disallow_reuse, fresh worker on the next evaluation;
- idle: removed from the idle list and pool statistics;
- created but unassigned (in transit to an acquirer or routed to a
  waiter): total worker count decremented;
- boot failure: the pending creation for the same pool is failed (so
  create_worker errors instead of waiting forever) and the booting
  statistics are balanced - no phantom workers after a boot crash.

Every acquisition and return path checks the dead set so a reaped id is
never handed out again: acquire_worker retries after dead candidates
(idle pop, waiter handoff, fresh boot), and WorkerOperation's drop skips
returning a worker that died while checked out.

Verified end to end with turbopackPluginRuntimeStrategy: 'workerThreads':

- before: a loader calling process.exit(1) hangs the request forever and
  wedges all later compilations;
- after: the request fails in ~1s with "Node.js subprocess crashed while
  evaluating loaders", and the next request is served by a fresh worker
  in ~50ms. A worker that dies between tasks (idle) is reaped the same
  way.

The new development test covers the crash path: baseline (stock binary)
fails by hanging; candidate fails the evaluation and recovers. The #96592
error-cleanup test still passes. Autoreview panel findings addressed:
watch-channel close, dead-set validation on all handoff paths,
options-tagged pending creations, clean-exit reaping.

Validation: pnpm build-all (TS + native), cargo fmt --check, cargo clippy
-p turbopack-node --all-targets -D warnings, cargo test -p turbopack-node
--lib, candidate integration test pass, exact baseline binary fails by
hanging.

Co-Authored-By: Marcos Hernanz <96699542+marcoshernanz@users.noreply.github.com>

When a worker-thread plugin runtime dies unexpectedly (uncaught exception,
process.exit, OOM), nothing told the Rust worker pool about it: the dead
Worker stayed in loaderWorkers, the operation waiting for its response
hung forever, and every later evaluation could be handed the dead worker,
wedging the dev server's compilation permanently. This is the crash-time
sibling of #96592, which covered evaluation and IPC errors reported over
the channel.

JavaScript now attaches 'error'/'exit' listeners to pooled workers and
reports any death while the worker is still registered (intentional
terminations remove the worker from the map first, so they are never
reported; the thread id is captured at creation since it resets to -1 by
the time 'exit' fires). On the Rust side, a dead-worker set records
reaped ids and handle_worker_death classifies the death:

- busy: closes the in-flight task's channel (via a watch channel, which
  cannot lose the wakeup), so its pending recv fails and surfaces through
  the existing subprocess-crash path in pull_operation - synthesized
  issue, disallow_reuse, fresh worker on the next evaluation;
- idle: removed from the idle list and pool statistics;
- created but unassigned (in transit to an acquirer or routed to a
  waiter): total worker count decremented;
- boot failure: the pending creation for the same pool is failed (so
  create_worker errors instead of waiting forever) and the booting
  statistics are balanced - no phantom workers after a boot crash.

Every acquisition and return path checks the dead set so a reaped id is
never handed out again: acquire_worker retries after dead candidates
(idle pop, waiter handoff, fresh boot), and WorkerOperation's drop skips
returning a worker that died while checked out.

Verified end to end with turbopackPluginRuntimeStrategy: 'workerThreads':

- before: a loader calling process.exit(1) hangs the request forever and
  wedges all later compilations;
- after: the request fails in ~1s with "Node.js subprocess crashed while
  evaluating loaders", and the next request is served by a fresh worker
  in ~50ms. A worker that dies between tasks (idle) is reaped the same
  way.

The new development test covers the crash path: baseline (stock binary)
fails by hanging; candidate fails the evaluation and recovers. The #96592
error-cleanup test still passes. Autoreview panel findings addressed:
watch-channel close, dead-set validation on all handoff paths,
options-tagged pending creations, clean-exit reaping.

Validation: pnpm build-all (TS + native), cargo fmt --check, cargo clippy
-p turbopack-node --all-targets -D warnings, cargo test -p turbopack-node
--lib, candidate integration test pass, exact baseline binary fails by
hanging.

Co-Authored-By: Marcos Hernanz <96699542+marcoshernanz@users.noreply.github.com>

When a worker-thread plugin runtime dies unexpectedly (uncaught exception,
process.exit, OOM), nothing told the Rust worker pool about it: the dead
Worker stayed in loaderWorkers, the operation waiting for its response
hung forever, and every later evaluation could be handed the dead worker,
wedging the dev server's compilation permanently. This is the crash-time
sibling of #96592, which covered evaluation and IPC errors reported over
the channel.

JavaScript now attaches 'error'/'exit' listeners to pooled workers and
reports any death while the worker is still registered (intentional
terminations remove the worker from the map first, so they are never
reported; the thread id is captured at creation since it resets to -1 by
the time 'exit' fires). On the Rust side, a dead-worker set records
reaped ids and handle_worker_death classifies the death:

- busy: closes the in-flight task's channel (via a watch channel, which
  cannot lose the wakeup), so its pending recv fails and surfaces through
  the existing subprocess-crash path in pull_operation - synthesized
  issue, disallow_reuse, fresh worker on the next evaluation;
- idle: removed from the idle list and pool statistics;
- created but unassigned (in transit to an acquirer or routed to a
  waiter): total worker count decremented;
- boot failure: the pending creation for the same pool is failed (so
  create_worker errors instead of waiting forever) and the booting
  statistics are balanced - no phantom workers after a boot crash.

Every acquisition and return path checks the dead set so a reaped id is
never handed out again: acquire_worker retries after dead candidates
(idle pop, waiter handoff, fresh boot), and WorkerOperation's drop skips
returning a worker that died while checked out.

Verified end to end with turbopackPluginRuntimeStrategy: 'workerThreads':

- before: a loader calling process.exit(1) hangs the request forever and
  wedges all later compilations;
- after: the request fails in ~1s with "Node.js subprocess crashed while
  evaluating loaders", and the next request is served by a fresh worker
  in ~50ms. A worker that dies between tasks (idle) is reaped the same
  way.

The new development test covers the crash path: baseline (stock binary)
fails by hanging; candidate fails the evaluation and recovers. The #96592
error-cleanup test still passes. Autoreview panel findings addressed:
watch-channel close, dead-set validation on all handoff paths,
options-tagged pending creations, clean-exit reaping.

Validation: pnpm build-all (TS + native), cargo fmt --check, cargo clippy
-p turbopack-node --all-targets -D warnings, cargo test -p turbopack-node
--lib, candidate integration test pass, exact baseline binary fails by
hanging.

Co-Authored-By: Marcos Hernanz <96699542+marcoshernanz@users.noreply.github.com>

When a worker-thread plugin runtime dies unexpectedly (uncaught exception,
process.exit, OOM), nothing told the Rust worker pool about it: the dead
Worker stayed in loaderWorkers, the operation waiting for its response
hung forever, and every later evaluation could be handed the dead worker,
wedging the dev server's compilation permanently. This is the crash-time
sibling of #96592, which covered evaluation and IPC errors reported over
the channel.

JavaScript now attaches 'error'/'exit' listeners to pooled workers and
reports any death while the worker is still registered (intentional
terminations remove the worker from the map first, so they are never
reported; the thread id is captured at creation since it resets to -1 by
the time 'exit' fires). On the Rust side, a dead-worker set records
reaped ids and handle_worker_death classifies the death:

- busy: closes the in-flight task's channel (via a watch channel, which
  cannot lose the wakeup), so its pending recv fails and surfaces through
  the existing subprocess-crash path in pull_operation - synthesized
  issue, disallow_reuse, fresh worker on the next evaluation;
- idle: removed from the idle list and pool statistics;
- created but unassigned (in transit to an acquirer or routed to a
  waiter): total worker count decremented;
- boot failure: the pending creation for the same pool is failed (so
  create_worker errors instead of waiting forever) and the booting
  statistics are balanced - no phantom workers after a boot crash.

Every acquisition and return path checks the dead set so a reaped id is
never handed out again: acquire_worker retries after dead candidates
(idle pop, waiter handoff, fresh boot), and WorkerOperation's drop skips
returning a worker that died while checked out.

Verified end to end with turbopackPluginRuntimeStrategy: 'workerThreads':

- before: a loader calling process.exit(1) hangs the request forever and
  wedges all later compilations;
- after: the request fails in ~1s with "Node.js subprocess crashed while
  evaluating loaders", and the next request is served by a fresh worker
  in ~50ms. A worker that dies between tasks (idle) is reaped the same
  way.

The new development test covers the crash path: baseline (stock binary)
fails by hanging; candidate fails the evaluation and recovers. The #96592
error-cleanup test still passes. Autoreview panel findings addressed:
watch-channel close, dead-set validation on all handoff paths,
options-tagged pending creations, clean-exit reaping.

Validation: pnpm build-all (TS + native), cargo fmt --check, cargo clippy
-p turbopack-node --all-targets -D warnings, cargo test -p turbopack-node
--lib, candidate integration test pass, exact baseline binary fails by
hanging.

Co-Authored-By: Marcos Hernanz <96699542+marcoshernanz@users.noreply.github.com>

## What

Terminate a worker-thread plugin runtime when an evaluation or IPC error
marks
its `WorkerOperation` non-reusable.

`disallow_reuse()` already removed the worker from pool statistics and
disabled
the callback that returns it to the pool, but it did not stop the
underlying
Node.js `Worker`. The worker then stayed strongly owned by
`loaderWorkers`, kept
its V8 isolate/module graph alive, and waited forever for another task
that
could never arrive.

This calls the existing worker terminator used by `wait_or_kill`,
scale-down,
and scale-to-zero. It removes the routed native channel and dispatches
the
existing JavaScript `Worker.terminate()` callback, which also deletes
the map
entry.

## Why this matters for v0

v0's planned worker-thread runtime turns transient loader/PostCSS errors
during
agent edits into a deterministic lifecycle leak. On the real v0
`/button`
route, each unique Tailwind/PostCSS failure retained one complete worker
runtime
on the baseline.

The balanced eight-process A/B panel (80 unique failures per lane)
measured:

| Lane | `WorkerThread` slope | Private-memory slope | Median failure |
Median recovery | >5 s failures |
|---|---:|---:|---:|---:|---:|
| baseline | **+1.000/error** | **+49.81 MiB/error** | 108.04 ms |
329.26 ms | 3 / 80 |
| candidate | **0.000/error** | **+5.12 MiB/error** | 108.15 ms | 317.65
ms | 2 / 80 |

That is a **100% elimination of leaked-worker growth** and an **89.7%
reduction
in retained private-memory slope** (exact process-bootstrap 95%
interval:
85.7%–99.5%). Every candidate process was below every baseline process;
exact
4-vs-4 process-label permutation `p=0.0143` one-sided / `0.0286`
two-sided.
Median failure latency changed by only +0.11 ms.

The independent 100-cycle-per-lane endurance confirmation showed the
same
causal result:

| Lane | `WorkerThread` cycle 1→100 | Private memory cycle 1→100 |
Private slope | Median failure | Median recovery |
|---|---:|---:|---:|---:|---:|
| baseline | **38→137** | **4.28→7.34 GiB** | **+27.02 MiB/error** |
104.45 ms | 331.32 ms |
| candidate | **32→32** | **3.93→4.10 GiB** | **+0.98 MiB/error** |
102.94 ms | 297.24 ms |

That long run eliminated all observed worker growth and reduced the
fitted
private-memory slope 96.4%. Every one of the 200 unique failures
recovered to
the complete real route, and the source hash was restored after every
cycle.

The benchmark used fresh `.next` state per process, five error/recovery
warmups,
a five-minute quiescence window, unique uncached errors, exact source
restoration
after every cycle, and within-process slopes. Baseline and candidate
native
binaries were built from the same Next.js base, with only this lifecycle
change
affecting candidate native code, and selected by SHA-verified
`NEXT_TEST_NATIVE_DIR` paths.

This is independent of vercel#96433. That PR coordinates explicit multi-file
edit
transactions; this PR fixes the lifecycle of a worker that has already
failed.
It also does not change idle-worker scale-down policy: failed workers
are absent
from the idle pool, so scale-down cannot see them.

## Regression test

The new development test schedules a one-second filesystem marker inside
a
custom loader, then throws an evaluation error under
`turbopackPluginRuntimeStrategy: 'workerThreads'`:

- baseline native binary: fails because the orphan worker remains alive
and
  writes the marker;
- candidate native binary: passes because the worker is actually
terminated;
- candidate then renders a recovered value, proving replacement/recovery
works.

## Validation

- `pnpm build-all`
- `cargo fmt --all -- --check`
- `cargo check -p turbopack-node --all-targets`
- `cargo test -p turbopack-node --lib`
- `cargo clippy -p turbopack-node --all-targets -- -D warnings`
- focused Prettier and ESLint checks for the new test
- candidate integration test pass; exact baseline integration test fails
at the
  intended liveness assertion
- GPT-5.6 Sol xhigh + Claude Opus 5 xhigh autoreview panel: zero
actionable
  findings, patch correct at 0.96 confidence

Exact optimized native SHA-256 values used by the benchmark:

- baseline:
`609eeeed41e1425f06136b54b7997e5b1b1add992169b17a797889dfb830dd7f`
- candidate:
`141211112ae47b8bd814aaab437060114e92f60a1bf6f23e1b0e153b1e4837cc`

Both stripped binaries were 205,863,784 bytes. Raw per-process JSON,
`/proc`
samples, server logs, harness source, analysis output, and A/B plus
endurance
graphs are retained in the investigation archive.

Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>