Visitar URL original
Keep deadlock and concurrency tests off the thread pool by dwcullop · Pull Request #1226 · reactivemarbles/DynamicData · GitHub
Skip to content

Keep deadlock and concurrency tests off the thread pool - #1226

Open
dwcullop wants to merge 2 commits into
u/dacullop/main/cross_feeding_deadlock_testfrom
u/dacullop/main/deterministic_stress_tests
Open

dwcullop wants to merge 2 commits into
u/dacullop/main/cross_feeding_deadlock_testfrom
u/dacullop/main/deterministic_stress_tests

Conversation

@dwcullop

@dwcullop dwcullop commented Nov 2, 2026

Copy link
Copy Markdown
Member

Depends on #1213. This PR is based on #1213's branch, so the diff shows only the two commits here. Once #1213 merges, I'll retarget this PR to main.

Summary

Several deadlock and concurrency tests block thread-pool threads and then guard the wait with an in-test timer. On an oversubscribed CI runner, the pool is too starved to start the threads the test is waiting for, the timer fires, and the test reports a deadlock that never happened.

This PR applies the pattern from #1213 to the rest of the suite.

Root cause

DeadlockTortureTest.MultiplePairs_Simultaneous_NoDeadlock failed this way on #1166. It starts eleven bidirectional runs at once. Each parks 2 or 3 Task.Run writers at a Barrier, so it holds up to 33 pool threads and then waits for the pool to inject more.

Locally, on one core with DOTNET_PROCESSOR_COUNT=8:

  • It takes 11 to 16 s instead of under 1 s, and all of the difference is spent waiting for threads.
  • With this change it takes 3 to 5 s.

Changes

Every changed test follows the same rules:

  • Any thread a test blocks runs on a dedicated thread.
  • In-test timers are removed: Task.WhenAny/Task.Delay guards, timed Waits, WaitAsync(timeout), and .Result.
  • A hang is bounded only by the xUnit Timeout attribute.
  • Tests that block threads or carry a Timeout live in a nested IntegrationTests class, so they run in the non-parallel collection. xUnit v2 only honours Timeout there.

New helper

DedicatedThread.Run(Action) in DynamicData.Tests/Utilities wraps Task.Factory.StartNew(..., TaskCreationOptions.LongRunning, ...). I found no existing equivalent. The name is open to rename.

Fixtures changed

  • DeadlockTortureTest and CrossCacheDeadlockStressTest: dedicated writer threads, guards removed, [Fact(Timeout = 60_000)]. The slowest test takes 9 s. The now-dead completionNames diagnostics are removed too.
  • Tests moved, unchanged apart from the rules above, into new or existing IntegrationTests classes:
    • TransformAsyncFixture: the two forced-transform serialization tests.
    • DeliveryQueueFixture: OnlyOneDelivererAtATime (104 blocked Task.Run contenders) and the two dispose tests.
    • SharedDeliveryQueueFixture: the three parked-drainer receipt-order tests, which ignored the result of a 5 s Wait.
    • DeliveryQueueMergeFixture: the concurrent producers test.
  • Already in the integration collection, converted in place: SourceCacheFixture, SuspendNotificationsFixture, WhenPropertyChangedFixture and CacheParentSubscriptionFixture.

Test-only change: no library code or public API is affected.

Known leftovers, not changed here

  • SuspendNotificationsFixture.IntegrationTests.StaleResumeSignalIsSuppressedByConcurrentReSuspend keeps a 5 s Wait and always takes the full 5 s.
    • The parked subscriber holds the cache lock, so the window its comments describe never opens.
    • Replacing the wait with an untimed Wait() makes the test hang.
    • It needs a redesign. It is slow but not flaky.
  • These widen race windows rather than gate pass/fail, so I kept them:
    • Thread.Sleep and Thread.SpinWait in the TransformAsyncFixture serialization tests.
    • Thread.SpinWait in DeliveryQueueFixture.
    • Task.Delay(100) in DisposeWaitsForInFlightDelivery.
  • TransformAsyncFixture.RemoveFlowsToTheEnd and WithMaxConcurrency still depend on timing.
  • CrossCacheDeadlockStressTest asserts Or = Merged, which can fail rarely until Serialize static Or, And, Except and Xor delivery across sources #1224 (the combiner fix) lands.

Validation

  • Full suite (net9.0): 2631 passed, 0 failed, 2648 total. The total is unchanged, so no tests were lost or duplicated by the moves.
  • Parallel-phase tests under CPU oversubscription: 0 failures.

DeadlockTortureTest.MultiplePairs_Simultaneous_NoDeadlock failed on PR #1166 with five of its runs reporting a
timeout. Each run parks two or three Task.Run writers at a Barrier, and MultiplePairs starts eleven runs at once, so
it blocks up to 33 pool threads at a time and then waits for the pool to inject more. Under load that injection is
slower than the timer inside the test: on one core with DOTNET_PROCESSOR_COUNT=8 the test takes 11 to 16 s instead
of under 1 s, all of it waiting for threads.

Writers in DeadlockTortureTest and CrossCacheDeadlockStressTest now run on dedicated threads through a new
DedicatedThread.Run test helper, and the Task.WhenAny/Task.Delay guards are gone: each test awaits its work directly
and a hang is reported by the Timeout attribute, well above the slowest run (9 s). Under the same oversubscription
MultiplePairs now takes 3 to 5 s.
…rallel collection

Several concurrency tests parked Task.Run threads at a Barrier, inside a
blocked observer, or on an event, then waited for other pool threads to
arrive under an in-test timer (Task.WhenAny with Task.Delay, or a timed
Wait). On an oversubscribed CI runner the parked threads starve the pool,
the threads the test is waiting for are not injected in time, and the
timer fires: a false failure. xUnit v2 also only honours a Timeout
attribute for tests that run with parallelization disabled.

Every thread a test blocks now runs on a dedicated thread through
DedicatedThread.Run, the in-test timers and timed waits are gone, waits on
the test thread are awaited instead of blocked on, and a hang is bounded
only by the xUnit Timeout attribute. Tests that block threads or carry a
Timeout live in the nested IntegrationTests classes, which run in the
non-parallel collection.

Fixtures changed:
- TransformAsyncFixture: forced transform serialization tests moved to
  TransformAsyncFixture.IntegrationTests.cs
- DeliveryQueueFixture: OnlyOneDelivererAtATime and the two dispose tests
  moved to DeliveryQueueFixture.IntegrationTests.cs
- SharedDeliveryQueueFixture: the parked drainer receipt order tests moved
  into the existing IntegrationTests class
- DeliveryQueueMergeFixture: the concurrent producers test moved to
  DeliveryQueueMergeFixture.IntegrationTests.cs
- SourceCacheFixture, SuspendNotificationsFixture, WhenPropertyChangedFixture
  and CacheParentSubscriptionFixture integration tests: dedicated threads,
  no in-test timers, Timeout attributes

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant