run_tests reports false failure or gets stuck when a domain reload lands mid-job (Windows) — related to #1390 #1394

Summary

run_tests (both EditMode and PlayMode, though PlayMode far more often) intermittently reports a false job-level failure — "Test job failed to initialize (tests did not start within timeout)", progress.total: null — while the underlying Unity test run actually completes and passes, confirmed via the NUnit TestResults.xml written to disk. In a smaller number of cases the job instead gets stuck at completed: 0 with total already known, never resolving without run_tests(clear_stuck: true). Once, the bridge itself stopped responding entirely (Unity session not ready ... (ping not answered) on every tool, including execute_code) until the Editor was restarted — the same message reported in #1390.

We ran a structured, 23-call reproduction across four conditions and cross-checked every call's job result against TestResults.xml. Root cause appears to be a domain reload landing during/around a test job, which triggers an MCP↔Unity bridge reconnect; TestJobManager's progress/completion callback seems to stay bound to the pre-reload session and never receives the real completion signal.

Filing this as its own issue (rather than a comment) since the reproduction data is fairly large and platform differs — see "Relation to #1390" below.

Environment

Symptom A — false failure, real run completes (most common)

run_tests(mode="PlayMode")
  -> status: running
  -> ... ~120s ...
  -> status: failed
     error: "Test job failed to initialize (tests did not start within timeout)"
     progress.total: null, progress.completed: 0

TestResults.xml on disk for the same time window shows a real, complete, passing run (e.g. testcasecount="80" result="Passed" total="80" passed="80"). Console shows [TestJobManager] Job <id> failed to initialize within 120000ms, auto-failing alongside [TestRunnerNoThrottle] Applied No Throttling → Restored Interaction Mode after test run — i.e. Unity's own test runner lifecycle completed normally.

Reproduced 10 / 23 calls (both EditMode and PlayMode, see table below).

Symptom B — stuck at completed: 0 (less common, more severe)

Same setup, but the job never reaches failed or succeeded — get_test_job returns identical last_update_unix_ms / completed: 0 across 6 consecutive polls (60s wait_timeout each) with total already populated. run_tests(clear_stuck: true) is required to unblock. Reproduced 2 / 23 calls:

• EditMode: console showed [TestRunnerNoThrottle] Recovered from domain reload - reapplying No Throttling twice during the single stuck job, then Restored Interaction Mode after test run — the real run did finish (TestResults.xml: 447/447) but the tracker never saw it.
• PlayMode: same pattern, but this time TestResults.xml was never updated at all and editor/state.editor.active_scene stayed pinned to the ephemeral InitTestSceneXXXXXXXX scene Unity creates for PlayMode test runs — i.e. that run genuinely never returned control.

Symptom C — full bridge lockup (once, matches #1390's transport symptom)

After several PlayMode retries, run_tests returned "Unity plugin session <id> disconnected while awaiting command_result", then "TimeoutError" on retry. From then on every tool call (run_tests(clear_stuck), execute_code, the editor/state MCP resource) failed with "Unity session not ready for '<op>' (ping not answered); please retry" — repeated 6+ times over several minutes, with each run_tests retry surfacing a different session id. The Editor process itself stayed up and interactive; only restarting it (not killing it) restored the bridge. This is the same (ping not answered) message reported in #1390, though our Editor did not hard-hang (no SIGKILL needed).

Reproduction method

We instrumented every run_tests call with a before/after snapshot of the editor/state MCP resource (compilation.last_domain_reload_after_unix_ms, advice.ready_for_tools, advice.blocking_reasons, staleness.is_stale), the job's get_test_job progress, a console excerpt (TestJobManager, TestRunnerNoThrottle, bridge/reconnect lines), and cross-referenced TestResults.xml's start-time/total/passed for the same window. Four conditions, EditMode + PlayMode:

One of the "no incident" C2 EditMode calls is worth flagging separately: the run_tests call itself returned a transport error ("dropped mid-call", no job_id at all) rather than a job-level failure, while editor/state.tests.last_run and TestResults.xml both confirmed the underlying run had actually completed and passed. An immediate retry succeeded normally. This reads as a milder, self-resolving instance of the same transport hiccup that escalated into Symptom C later on.

In every A/B row, compilation.last_domain_reload_after_unix_ms in the post-call editor/state snapshot fell inside [job.started_unix_ms, job.finished_unix_ms] — including C0/C1 rows, where we did not explicitly request a reload; Unity appears to trigger a reload on its own near test-job start reasonably often (and PlayMode's own Play-mode transition is itself a reload, which is presumably why PlayMode rows dominate).

Root-cause hypothesis (evidence, not verified against source)

1. Domain reload lands during/around job start. Supported by the timestamp overlap above; for PlayMode this is close to unavoidable since entering Play Mode triggers one.
2. Bridge reconnects as a result. Console shows [HTTP Reload] Resume attempt 1/6 → Resume succeeded → [WebSocket] Preparing to register 35 tool(s) with the bridge immediately following an injected reload (sometimes twice for a single job). In the Symptom C lockup, run_tests retries each surfaced a new Unity plugin session id, consistent with repeated reconnect attempts that never fully complete.
3. The editor/state resource cache stalls during the reconnect window — advice.ready_for_tools: false, blocking_reasons: ["stale_status"], with observed_at_unix_ms frozen (not advancing at all) for anywhere from a few seconds up to ~400s across our samples, before self-healing (except in Symptom C, where it never did).
4. TestJobManager's progress/completion callback appears bound to the pre-reload bridge session and never receives the real completion signal that arrives after reconnect — hence the job either times out (false failure, Symptom A) or is left at completed: 0 forever (Symptom B).

This reads as the same underlying weakness reported in #1390 (job tracking not surviving a Play-mode-triggered domain reload / bridge disruption), but with different severity: #1390 is a hard, 3/3-reproducible deadlock on Linux/Vulkan requiring SIGKILL, whereas ours is a Windows-side race that mostly self-heals via retry, escalating to a full lockup (Symptom C) only once in 23 calls — resolved by an Editor restart rather than a process kill.

Current workaround (client-side, not a fix)

Before treating a run_tests failure as real: check progress.total == null, check whether editor/state.compilation.last_domain_reload_after_unix_ms falls inside the job's [started_unix_ms, finished_unix_ms] window, and check the console for [TestRunnerNoThrottle] Restored Interaction Mode after test run after the reported failure. If all three hold, confirm advice.ready_for_tools == true and retry once. For the stuck case (Symptom B), run_tests(clear_stuck: true) followed by the same readiness check. For the full-lockup case (Symptom C), an Editor restart was the only thing that worked in our one occurrence.

Trimmed the raw per-call data (JSON snapshots, console excerpts, XML timestamps) down to the summary above; happy to answer follow-up questions about specific data points if it helps triage, but the project this was reproduced in is private so I'd rather not post the full log verbatim.