Repository navigation
HBASE-30411 Support regex based RS membership assignment to RS group - #8733
sanjeet006py wants to merge 12 commits into
Conversation
Port branch-2's hbase.rsgroup.regex.<groupname> config-driven automatic RSGroup membership onto master's RSGroupInfoManagerImpl, adapting it to master's compute-on-demand default-group pattern instead of branch-2's persist-on-every-change background thread. Also fix ServerEventsListenerThread's change detection: it previously diffed newly computed auto-managed membership against a private in-memory snapshot instead of the live holder state. An out-of-band admin mutation (e.g. moveServersToRSGroup) could desync that snapshot from reality, and if a later recompute coincidentally matched the stale snapshot, the thread would skip persisting -- permanently freezing a phantom server in the cached/persisted group. Comparing against holder.groupName2Group directly removes the stale baseline.
No functional change; reflows comment line breaks to match the project's formatter output.
…ng the way RSGroupInfoManagerImpl: add handleServerEvent(), called synchronously from serverAdded/serverRemoved. A recompute that only affects 'default' membership now applies immediately in-memory instead of waking the background ServerEventsListenerThread; only a regex-governed group's membership change (which must be persisted to hbase:rsgroup/ZK) hands off to that thread. Also fix the thread's run loop to wait for a serverChanged() signal before recomputing, instead of recomputing once unconditionally on start and only waiting after. TestRSGroupsBase: strengthen addGroup/removeGroup assertions (ported from branch-2) to verify servers actually land where expected, and switch the table-membership check to ADMIN.getRSGroup(TableName) instead of defaultInfo.getTables() -- MasterRpcServices.getRSGroupInfo (the live RPC backing ADMIN.getRSGroup(String)) never resolves table membership from TableDescriptor attributes, unlike listRSGroups() or getRSGroup(TableName). TestRegexBasedRSGroupMembership: drop two triggerListenerCycle() calls (in afterMethod()'s dead hadLeftoverRegex guard, and in clearRegex()) that forced a synthetic server event to mask a regex-governed group's membership from going stale after a config change. That staleness is expected -- recompute is event-driven by design, matching branch-2's semantics -- and removeGroup()/VerifyingRSGroupAdmin#verify() already tolerate it correctly without help. Also fix several comments left over from the branch-2 port referencing RSGroupAdminServer and moveTableRegionsToGroup, neither of which exist on master; point them at the real master-side methods (RSGroupInfoManagerImpl#setRSGroup / moveTablesAndWait / moveServers, MasterRpcServices).
Ported INFO-level log lines from branch-2's original regex-based RSGroup membership feature commits that were missing from master's adapted implementation: config resolution, hostname matching, server-to-group resolution, refresh entry/exit, and validation entry/exit. Also added a server count to the existing "Updated auto-managed RSGroup servers" log, matching branch-2's equivalent message.
RSGroupInfoManagerImpl#updateAutoManagedRSGroupServers() used to take a precomputed assignments map as a parameter. ServerEventsListenerThread#run() computed that map -- and the comparison against live group state used to decide whether an update was even needed -- before acquiring the enclosing instance's lock, the same lock the method itself (and every other mutator) relies on to serialize access to holder.groupName2Group. A concurrent admin-driven change (moveServers, flushConfig, etc.) landing in that window would be silently clobbered by the stale assignments the background thread had already computed. Make the method self-contained and parameterless: it now reads holder.groupName2Group, recomputes the auto-managed assignment, compares against current state, and applies the update, all under its own synchronized block, so the whole decide-and-apply sequence is atomic. Separately, ServerEventsListenerThread's own wait/notify signal used a boolean changed flag that was only cleared after calling updateAutoManagedRSGroupServers() -- which can block on I/O (hbase:rsgroup table and ZK writes via flushConfig). A new server event arriving while that call was in flight would set changed = true, but the subsequent unconditional changed = false once the call returned would wipe that signal with nothing left to wake the thread for it. Replace the boolean with an int eventCount: serverChanged() increments it, and the run loop decrements it by exactly one per completed pass, so a surplus signal forces an extra (idempotent) pass instead of being lost. Also add INFO-level logging for serverAdded/serverRemoved callbacks, matching the logging already present on the other paths through this class. TestRegexBasedRSGroupMembership: fix a ~3%-flaky crash-target assumption in testUngracefulCrashOfRegexMemberReassignsRegionsOnlyToOtherGroupMembers -- retainAssignment's random fallback (BaseLoadBalancer#randomAssignment) places each region independently, so it's possible for all 6 of this test's regions to land on sn2 alone instead of being split with sn1. Crash whichever server actually ended up holding regions, instead of hardcoding sn1, so ServerCrashProcedure always has something to reassign and roundRobinAssignment is guaranteed to fire.
…ckground thread Defensive guard around the eventCount-- in ServerEventsListenerThread.run(); the decrement already only runs once the while-loop confirms eventCount > 0, so this is a no-op in practice but makes that invariant explicit.
…sting it Servers of regex-governed RSGroups (hbase.rsgroup.regex.<group>) are no longer stored in hbase:rsgroup or ZooKeeper. Membership is recomputed from live servers on every server event and on master start; stale copies stored in other groups are moved to default and persisted once. Server events are handled synchronously, so the background event thread and the flushConfig membership check are removed. - moveServers rejects moves that contradict a regex - offline mode allows changes to regex-governed group servers - admin-managed and regex-matched server sets are kept disjoint - renaming a regex-governed group is allowed - updateRSGroupConfig updates a copy and flushes the new group map - docs: log levels and strong advice to enable hbase.rsgroup.fallback.enable - tests: strengthened regex membership tests and VerifyingRSGroupAdmin checks
|
There are some refguide and checkstyle warnings. |
- updateRSGroupConfig: iterate over a snapshot of the keys when clearing configuration on the copy; removing while iterating threw ConcurrentModificationException - getRegexGroupMap: skip entries whose value is null (a concurrent configuration reload can remove a key between name and value reads) - checkMoveAgreesWithRegex: no longer prints hbase.rsgroup.regex.null for a server that matches no regex - Stale-copy WARN no longer advises a move that would be rejected; stale copies are dropped and the server lands in its regex group - Drop the per-server "does not match any regex" INFO line - Docs: regex removal and enable behavior, dynamic RSGroups, Dead Nodes, disable runbook, same config on all masters, ReDoS note, dynamic.mdx note
…ity.mdx The refguide build runs prettier --check on the docs and failed on the two callouts added for regex-based RSGroup membership.
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
9 open findings
These test-only helpers arepublicon a production class, which increases the public API surface… · New These test-only helpers arepublicon a production class, which increases the public API surface… · New A number of logs here are emitted at INFO on every RS join/leave (and potentially on every move via… · New A number of logs here are emitted at INFO on every RS join/leave (and potentially on every move via… · New A number of logs here are emitted at INFO on every RS join/leave (and potentially on every move via… · New A number of logs here are emitted at INFO on every RS join/leave (and potentially on every move via… · New A number of logs here are emitted at INFO on every RS join/leave (and potentially on every move via… · New The validation pattern allows underscores ([a-zA-Z0-9_]+), but the exception message says 'only… · New LogCapturer#stopCapturing removes the appender from the logger but does not stop the appender… · New
What changed in this PR
Adds regex-driven automatic RegionServer membership assignment to RSGroups (HBASE-30411), including docs and tests to validate behavior and operational guidance.
Changes:
- Implement regex-based auto-managed RSGroup server membership (computed from live servers; not persisted) and enforce “regex wins” against conflicting manual moves.
- Add extensive black-box tests for regex membership, persistence semantics, crash/fallback behavior, and admin operations.
- Update website documentation to describe regex-based membership, caveats, and recommended fallback settings.
| File | Description |
|---|---|
| hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/operational-management/region-and-capacity.mdx | Documents regex-governed RSGroups, dynamics vs persisted membership, and operational caveats. |
| hbase-website/app/pages/_docs/docs/_mdx/(multi-page)/configuration/dynamic.mdx | Clarifies when regex config reloads take effect (next server event / master restart). |
| hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java | Core implementation: regex config parsing, auto-managed membership recomputation, persistence stripping, and move constraints. |
| hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java | Adds test-only toggles and assignment-path call tracking for assertions. |
| hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRegexBasedRSGroupMembership.java | New comprehensive test suite for regex-based membership behavior. |
| hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsBase.java | Strengthens base test helpers with additional assertions around group membership cleanup. |
| hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdmin.java | Updates verification logic to account for non-persisted regex-governed membership. |
| hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestRegionMoverFilterRSGroupServers.java | Updates comment reference for default/auto-managed membership computation. |
🧠 Review effort: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| @RestrictedApi(explanation = "Should only be called in tests", link = "", | ||
| allowedOnPath = ".*/src/test/.*") | ||
| public void setFallbackEnabledForTest(boolean enabled) { | ||
| this.fallbackEnabled = enabled; | ||
| } | ||
|
|
||
| // Test-only call-tracking flags, set true whenever the corresponding assignment method below is | ||
| // invoked. Reset via resetAssignmentCallFlagsForTest() before the operation under test. | ||
| private volatile boolean roundRobinAssignmentInvoked = false; | ||
| private volatile boolean retainAssignmentInvoked = false; | ||
| private volatile boolean randomAssignmentInvoked = false; | ||
|
|
||
| @RestrictedApi(explanation = "Should only be called in tests", link = "", | ||
| allowedOnPath = ".*/src/test/.*") | ||
| public boolean isRoundRobinAssignmentInvoked() { | ||
| return roundRobinAssignmentInvoked; | ||
| } |
| @RestrictedApi(explanation = "Should only be called in tests", link = "", | ||
| allowedOnPath = ".*/src/test/.*") | ||
| public void resetAssignmentCallFlagsForTest() { | ||
| roundRobinAssignmentInvoked = false; | ||
| retainAssignmentInvoked = false; | ||
| randomAssignmentInvoked = false; | ||
| } |
| return; | ||
| } | ||
| } | ||
| LOG.info("No changes in auto-managed RSGroup server membership."); |
There was a problem hiding this comment.
Removed this log line.
| static Map<String, Pattern> getRegexGroupMap(Configuration conf) { | ||
| Map<String, String> rsGroupNameToRegexMap = conf.getPropsWithPrefix(RS_GROUP_REGEX_PREFIX); | ||
| Map<String, Pattern> rsGroupNameToPatternMap = new HashMap<>(); | ||
| for (Map.Entry<String, String> e : rsGroupNameToRegexMap.entrySet()) { |
There was a problem hiding this comment.
Changed to debug log level.
| LOG.info("Resolved regex-based RSGroup membership config: {}", rsGroupNameToPatternMap); | ||
| return rsGroupNameToPatternMap; | ||
| } | ||
|
|
||
| // Called by ServerEventsListenerThread. Presume it has lock on this manager when it runs. | ||
| private SortedSet<Address> getDefaultServers(List<RSGroupInfo> rsGroupInfoList) { | ||
| // Build a list of servers in other groups than default group, from rsGroupMap | ||
| Set<Address> serversInOtherGroup = new HashSet<>(); | ||
| for (RSGroupInfo group : rsGroupInfoList) { | ||
| if (!RSGroupInfo.DEFAULT_GROUP.equals(group.getName())) { // not default group | ||
| serversInOtherGroup.addAll(group.getServers()); | ||
| private static List<String> getMatchingRSGroupNames(String hostname, | ||
| Map<String, Pattern> rsGroupNameToPatternMap) { | ||
| List<String> rsGroupNames = new ArrayList<>(); | ||
| for (Map.Entry<String, Pattern> e : rsGroupNameToPatternMap.entrySet()) { | ||
| if (e.getValue().matcher(hostname).matches()) { | ||
| rsGroupNames.add(e.getKey()); | ||
| } | ||
| } | ||
| LOG.info("Hostname '{}' matches RSGroup name(s) {}", hostname, rsGroupNames); | ||
| return rsGroupNames; | ||
| } |
There was a problem hiding this comment.
Removed this log line.
|
|
||
| private Map<String, SortedSet<Address>> | ||
| computeAutoManagedRSGroupServers(Collection<RSGroupInfo> existingGroups) { | ||
| LOG.info("Computing auto-managed RSGroup server membership."); |
There was a problem hiding this comment.
Removed this log line.
| entry.getKey(), RS_GROUP_REGEX_PREFIX, entry.getKey()); | ||
| } | ||
| } | ||
| LOG.info("Computed auto-managed RSGroup server membership: {}", result); |
There was a problem hiding this comment.
Changed to debug log level.
| private void checkGroupName(String groupName) throws ConstraintException { | ||
| if (!groupName.matches("[a-zA-Z0-9_]+")) { | ||
| if (!GROUP_NAME_PATTERN.matcher(groupName).matches()) { | ||
| throw new ConstraintException("RSGroup name should only contain alphanumeric characters"); |
| } | ||
|
|
||
| void stopCapturing() { | ||
| this.logger.removeAppender(this.appender); |
…isted for table readers Backport of the on-the-fly regex RSGroup membership from apache#8733 to branch-2. Membership of regex-governed RSGroups is always recomputed from the live servers and the configured regexes. Servers stored for such a group are ignored on refresh, and stale copies in other groups are dropped. Unlike master, the recomputed membership is still persisted to hbase:rsgroup and ZooKeeper, because RSGroupTableAccessor readers (RegionMover and the master web UI) read the table directly. The server-event thread flushes after every recompute; if the flush fails and the default group is unchanged, the change is applied in memory only. Also validate that explicit server moves agree with the configured regexes, and update the RSGroup config on a copy of the group. Tests: extend TestRegexBasedRSGroupMembership, including checks that stored membership matches live membership and that a stale stored copy is replaced after a master restart.
The invalid-name WARN for hbase.rsgroup.regex.<group> and the ConstraintException from checkGroupName now print GROUP_NAME_PATTERN instead of a hardcoded description, so the messages follow the pattern.
- Say that hbase.rsgroup.regex.* changes are applied by restarting all HMasters (recommended) or, after a reload, only at the next server event - Update the regex group removal and RegionServer Grouping removal steps to restart the HMasters instead of reloading - Document that adding or changing a regex drops unmatched servers later - Note that add_rsgroup and rename do not apply a regex
|
@sanjeet006py just add information related to auto-managed vs admin-managed groups and the purpose behind introducing auto-managed regex-based rsgroup in Javadoc. We are good to go. |
…Impl Add a class-level javadoc section that defines admin-managed and auto-managed groups, and says why auto-managed groups are needed for auto-scaling in cloud deployments.


JIRA: HBASE-30411