Repository navigation
Conversation
Add a LeastLoadedPartitioning/LeastLoadedPartitioner pair for links whose partitioning is unconstrained. Instead of dealing tuples across one buffer per receiver (round robin), a sender fills a single batch at a time and sends it to one preferred receiver, so the first batch ships after batchSize tuples rather than senders x receivers x batchSize. Workers also report received_tuple_count, counted when a batch arrives. Received minus processed is a receiver's queued backlog, which later changes use to choose the preferred receiver. Nothing selects the new partitioner yet and RoundRobinPartitioner is untouched, so behaviour is unchanged. Part of apache#8519 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
On each statistics poll the coordinator ranks a least-loaded link's receivers by backlog (received minus processed tuples) and tells every sender which receiver to fill next through a new UpdateRoutingPreference RPC. The RPC only moves an index on the existing partitioner. It is kept separate from AddPartitioning, which rebuilds the partitioner and would discard a partly filled batch on every poll. Adds the enable-least-loaded-routing flag (default false) and a verbose logging flag. No link uses the least-loaded partitioner yet, so behaviour is unchanged. Part of apache#8519 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Contributor
Automated Reviewer SuggestionsBased on the
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
The second of three PRs. Still inert: no link uses the least-loaded partitioner until PR 3, and the flag defaults to off.
LeastLoadedRoutingService: on each statistics poll the coordinator ranks a link's receivers by backlog (received − processed) and gives each sender one, spreading senders across receivers rather than sending them all to the single least-loaded one.UpdateRoutingPreferenceRPC (Scala and Python workers): tells each sender which receiver to fill next. It only moves an index on the existing partitioner. It is kept separate fromAddPartitioning, which rebuilds the partitioner and would throw away a partly filled batch on every poll.network-buffering.enable-least-loaded-routing(defaultfalse) andleast-loaded-routing-verbose.Any related issues, documentation, discussions?
Part of: #8858
Depends on: #8862
How was this PR tested?
sbt "testOnly *LeastLoadedRoutingServiceSpec": the least-backlogged receiver goes to the first sender, senders with equal counts get different receivers, a receiver with no reading counts as idle, an empty link yields no assignments, and every sender is assigned when senders outnumber receivers.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Sonnet 5.5, Claude Opus 5.5)
🤖 Generated with Claude Code