Repository navigation
fix(router): correct diagnostics failure counts and request metrics locking - #7402
Open
lalit-0106 wants to merge 2 commits into
Open
lalit-0106 wants to merge 2 commits into
lalit-0106 wants to merge 2 commits into
Conversation
The guard in commitStatusList checked whether the error response key existed instead of the event key, so every new error response replaced the event's inner map and dropped the counts collected so far. The reported count was always 1. Move the update into trackFailureMetrics, check the event key there, and cover it with TestTrackFailureMetrics.
|
Thank you @lalit-0106 for contributing this PR. |
collectMetrics set requestsMetric to nil while holding only the read lock. takeRequestMetrics now swaps and resets the slice under Lock, and the aggregation runs on the local copy, so Diagnostics.Track is no longer called with a lock held. Both telemetry locks become sync.Mutex since neither is ever read-locked, which turns a reintroduced RLock into a compile error.
lalit-0106
force-pushed
the
fix/router-diagnostics-metrics
branch
2 times, most recently
from
September 16, 2026 14:13
a6be2fb to
2dda5f4
Compare
Contributor
|
This PR is considered to be stale. It has been open 20 days with no further activity thus it is going to be closed in 7 days. To avoid such a case please consider removing the stale label manually or add a comment to the PR. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes two bugs in the router's diagnostics telemetry.
#7260: failure counts are wiped whenever a new error response appears
The guard in
commitStatusListchecked whether the error response key existed instead of the event key. Each new error response replaced the event's whole inner map, sofailuresMetric[event]never held more than one entry and the reportedcountwas always 1.trackFailureMetrics(event, errorResponse), next totrackRequestMetrics, and checks the event key.TestTrackFailureMetricscovers it. It fails on the old guard:router_failedends up as{"timeout": 1}instead of{"connection refused": 1, "timeout": 3}.#7261:
requestsMetricis reset while holding only a read lockcollectMetricssetrequestsMetric = nilunderRLock. Each router runs a singlecollectMetricsgoroutine today, so this doesn't race in practice, but it breaks thesync.RWMutexcontract.takeRequestMetricsnow swaps and resets the slice underLock, andcollectMetricsaggregates the local copy, soDiagnostics.Trackis no longer called with a lock held and workers callingtrackRequestMetricsdon't wait on it.sync.Mutex: neitherrequestsMetricLocknorfailureMetricLockis ever read-locked, so reintroducing the misuse would be a compile error.TestTakeRequestMetricscovers it: the swap hands over everything collected so far and leaves the next batch empty, and concurrent takers and writers neither lose nor double-count a metric. TestingcollectMetricsitself would mean mutating the package-leveldiagnosticsglobals, which no router test does today, so the helper is the seam.This supersedes #7265 and #7266 by @harsh4vardhan, which took the same approach and were closed by the stale bot.
Testing
utils/miscdoesn't build on Windows, so I ran the checks on Linux in GitHub Actions on my fork: run 35045981108. The workflow lives on a throwaway branch and is not part of this PR; it mirrorsverify.ymland thePackage Unit [ router ]job intests.yaml.TestTrackFailureMetricswith the old guard put back: failsgo test -race -run '^(TestTrackFailureMetrics|TestTakeRequestMetrics|TestUpdateRudderSourcesStats_.*)$' ./router/with the fix:ok github.com/rudderlabs/rudder-server/router 1.130sgo mod tidy,make mocks,make fmt: no diffmake test exclude="router/batchrouter/asyncdestinationmanager/bing-ads" package=router: every package passes (1232 tests: 1229 passed, 3 skipped). The skips areTestStreamWriterIntegrationandTestAPIIntegration, which need CI credentials, andTestUploadIntegration, which is skipped in the code.FORCE_RUN_INTEGRATION_TESTSwas left unset since the fork has no CI secrets. Per-test results match the same job run on upstreammaster(80353d18) exactly, apart from the two tests this PR adds.trackFailureMetrics100%,takeRequestMetrics100%,commitStatusList90.3%.collectMetricsstays where it was (15%): it reads the package-leveldiagnosticsglobals, so the helpers are the testable seam.Linear Ticket
N/A (external contribution). Closes #7260, closes #7261.
Security