Visitar URL original
Make the cross-feeding CacheParentSubscription test deterministic by dwcullop · Pull Request #1213 · reactivemarbles/DynamicData · GitHub
Skip to content

Make the cross-feeding CacheParentSubscription test deterministic - #1213

Open
dwcullop wants to merge 2 commits into
mainfrom
u/dacullop/main/cross_feeding_deadlock_test
Open

dwcullop wants to merge 2 commits into
mainfrom
u/dacullop/main/cross_feeding_deadlock_test

Conversation

@dwcullop

Copy link
Copy Markdown
Member

Summary

CacheParentSubscriptionFixture.DeadlockProof_CrossFeedingSubscriptions fails intermittently in CI, unrelated to the PRs it fails on (#1207, #1103 and several Renovate runs). It is a flaky test, not a deadlock in the library.

Root cause

  • The test's two writers were Task.Run threads meeting at a Barrier(2), guarded by a 30 s Task.WhenAny/Task.Delay.
  • The test ran in the parallel phase. On an oversubscribed runner the thread pool is starved, so one writer starts and parks at the barrier while the other is not scheduled before the guard fires.
  • The same failure reproduces locally on unmodified main: run the parallel-phase tests on one core (start /affinity 0x1) with DOTNET_PROCESSOR_COUNT=8. The test passes in about 45 ms on its own.
  • The library cannot deadlock here. Both ObservableCache (DeliveryQueue) and CacheParentSubscription (SharedDeliveryQueue) release their lock before delivering.
  • The old test also did not guard what it claimed. Its writes crossed only the source caches, whose queue lets the second writer enqueue and return, so it still passed when CacheParentSubscription held its gate during delivery.

SharedDeliveryQueueFixture.InlineDeliveryDoesNotLetLaterNotificationsJumpTheQueue fails the same way (5 CI runs, and alongside this test in the local repro). Its drainer is a Task.Run thread, its continuations need the pool, and xUnit v2 only supports Timeout when parallelization is disabled.

Changes

  1. DeadlockProof_CrossFeedingSubscriptions is replaced by CacheParentSubscriptionFixture.IntegrationTests.CrossFeedingSubscriptions_FeedEachOtherDuringDelivery_DoNotDeadlock:
    • Each writer parks inside its own subscription's delivery until the other is there too. It then feeds the other subscription through its child observable and its source cache.
    • Writers run on dedicated threads (TaskCreationOptions.LongRunning).
    • The test runs in the non-parallel IntegrationTests collection. A hang is bounded only by [Fact(Timeout = 10_000)], with no WhenAny, Delay, Sleep or polling.
    • The assertions check the exact contents of both caches and both observers.
    • If either the cache or the subscription holds its lock while delivering, the test deadlocks on every run. I checked both by temporarily changing the library code.
  2. The four parked-delivery SharedDeliveryQueueFixture tests move, unchanged apart from indentation, into a nested IntegrationTests class, and their drainer runs on a dedicated thread.

Test-only change: no library code or public API is affected.

Validation

  • The new test passed 20/20 runs on one oversubscribed core.
  • The parallel-phase suite passed 3/3 runs under the oversubscription that made unmodified main fail 2 of 3.

DeadlockProof_CrossFeedingSubscriptions failed intermittently in CI (PR #1207, PR #1103 and two Renovate runs) by
exceeding its 30 second WhenAny guard. Nothing deadlocked: the writers were Task.Run threads meeting at a Barrier,
so on an oversubscribed runner, during the parallel phase of the test run, a starved thread pool left one writer
unstarted and the other parked at the Barrier. The same failure reproduces locally on unmodified main by running the
parallel tests on one core with DOTNET_PROCESSOR_COUNT=8.

The test also did not guard what it claimed. Its writes crossed only the source caches, whose delivery queue lets a
second writer enqueue and return, so it still passed with CacheParentSubscription holding its gate while delivering.

The replacement parks each writer inside its own subscription's delivery until the other is there too, then feeds
the other subscription through its child observable and its source cache. Holding a lock while delivering, in either
the subscription or the cache, now deadlocks on every run; both were confirmed by mutation. Writers run on dedicated
threads, the test lives in the non-parallel IntegrationTests collection, a Timeout attribute bounds a hang, and the
assertions check the exact contents of both caches and both observers.
…l collection

InlineDeliveryDoesNotLetLaterNotificationsJumpTheQueue failed intermittently in CI (five Renovate runs) and locally
under the same oversubscription that reproduces the cross-feeding CacheParentSubscription failure, by exceeding its
Timeout. The four tests that park a delivery start their drainer with Task.Run and resume through
RunContinuationsAsynchronously, so they wait on the thread pool, and xUnit v2 only supports Timeout in a collection
with parallelization disabled.

Move them, unchanged apart from indentation, into a nested IntegrationTests class, and start the drainer on a thread
of its own.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant