Repository navigation
Conversation
DeadlockTortureTest.MultiplePairs_Simultaneous_NoDeadlock failed on PR #1166 with five of its runs reporting a timeout. Each run parks two or three Task.Run writers at a Barrier, and MultiplePairs starts eleven runs at once, so it blocks up to 33 pool threads at a time and then waits for the pool to inject more. Under load that injection is slower than the timer inside the test: on one core with DOTNET_PROCESSOR_COUNT=8 the test takes 11 to 16 s instead of under 1 s, all of it waiting for threads. Writers in DeadlockTortureTest and CrossCacheDeadlockStressTest now run on dedicated threads through a new DedicatedThread.Run test helper, and the Task.WhenAny/Task.Delay guards are gone: each test awaits its work directly and a hang is reported by the Timeout attribute, well above the slowest run (9 s). Under the same oversubscription MultiplePairs now takes 3 to 5 s.
…rallel collection Several concurrency tests parked Task.Run threads at a Barrier, inside a blocked observer, or on an event, then waited for other pool threads to arrive under an in-test timer (Task.WhenAny with Task.Delay, or a timed Wait). On an oversubscribed CI runner the parked threads starve the pool, the threads the test is waiting for are not injected in time, and the timer fires: a false failure. xUnit v2 also only honours a Timeout attribute for tests that run with parallelization disabled. Every thread a test blocks now runs on a dedicated thread through DedicatedThread.Run, the in-test timers and timed waits are gone, waits on the test thread are awaited instead of blocked on, and a hang is bounded only by the xUnit Timeout attribute. Tests that block threads or carry a Timeout live in the nested IntegrationTests classes, which run in the non-parallel collection. Fixtures changed: - TransformAsyncFixture: forced transform serialization tests moved to TransformAsyncFixture.IntegrationTests.cs - DeliveryQueueFixture: OnlyOneDelivererAtATime and the two dispose tests moved to DeliveryQueueFixture.IntegrationTests.cs - SharedDeliveryQueueFixture: the parked drainer receipt order tests moved into the existing IntegrationTests class - DeliveryQueueMergeFixture: the concurrent producers test moved to DeliveryQueueMergeFixture.IntegrationTests.cs - SourceCacheFixture, SuspendNotificationsFixture, WhenPropertyChangedFixture and CacheParentSubscriptionFixture integration tests: dedicated threads, no in-test timers, Timeout attributes
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #1213. This PR is based on #1213's branch, so the diff shows only the two commits here. Once #1213 merges, I'll retarget this PR to
main.Summary
Several deadlock and concurrency tests block thread-pool threads and then guard the wait with an in-test timer. On an oversubscribed CI runner, the pool is too starved to start the threads the test is waiting for, the timer fires, and the test reports a deadlock that never happened.
This PR applies the pattern from #1213 to the rest of the suite.
Root cause
DeadlockTortureTest.MultiplePairs_Simultaneous_NoDeadlockfailed this way on #1166. It starts eleven bidirectional runs at once. Each parks 2 or 3Task.Runwriters at aBarrier, so it holds up to 33 pool threads and then waits for the pool to inject more.Locally, on one core with
DOTNET_PROCESSOR_COUNT=8:Changes
Every changed test follows the same rules:
Task.WhenAny/Task.Delayguards, timedWaits,WaitAsync(timeout), and.Result.Timeoutattribute.Timeoutlive in a nestedIntegrationTestsclass, so they run in the non-parallel collection. xUnit v2 only honoursTimeoutthere.New helper
DedicatedThread.Run(Action)inDynamicData.Tests/UtilitieswrapsTask.Factory.StartNew(..., TaskCreationOptions.LongRunning, ...). I found no existing equivalent. The name is open to rename.Fixtures changed
DeadlockTortureTestandCrossCacheDeadlockStressTest: dedicated writer threads, guards removed,[Fact(Timeout = 60_000)]. The slowest test takes 9 s. The now-deadcompletionNamesdiagnostics are removed too.IntegrationTestsclasses:TransformAsyncFixture: the two forced-transform serialization tests.DeliveryQueueFixture:OnlyOneDelivererAtATime(104 blockedTask.Runcontenders) and the two dispose tests.SharedDeliveryQueueFixture: the three parked-drainer receipt-order tests, which ignored the result of a 5 sWait.DeliveryQueueMergeFixture: the concurrent producers test.SourceCacheFixture,SuspendNotificationsFixture,WhenPropertyChangedFixtureandCacheParentSubscriptionFixture.Test-only change: no library code or public API is affected.
Known leftovers, not changed here
SuspendNotificationsFixture.IntegrationTests.StaleResumeSignalIsSuppressedByConcurrentReSuspendkeeps a 5 sWaitand always takes the full 5 s.Wait()makes the test hang.Thread.SleepandThread.SpinWaitin theTransformAsyncFixtureserialization tests.Thread.SpinWaitinDeliveryQueueFixture.Task.Delay(100)inDisposeWaitsForInFlightDelivery.TransformAsyncFixture.RemoveFlowsToTheEndandWithMaxConcurrencystill depend on timing.CrossCacheDeadlockStressTestassertsOr = Merged, which can fail rarely until Serialize static Or, And, Except and Xor delivery across sources #1224 (the combiner fix) lands.Validation