Visitar URL original
HBASE-30411 Support regex based RS membership assignment to RS group by sanjeet006py · Pull Request #8733 · apache/hbase · GitHub
Skip to content

HBASE-30411 Support regex based RS membership assignment to RS group - #8733

Open
sanjeet006py wants to merge 12 commits into
apache:masterfrom
sanjeet006py:rsgroup-membership-regex-master
Open

sanjeet006py wants to merge 12 commits into
apache:masterfrom
sanjeet006py:rsgroup-membership-regex-master

Conversation

@sanjeet006py

Copy link
Copy Markdown
Contributor

JIRA: HBASE-30411

Sanjeet Malhotra added 6 commits October 1, 2026 02:09
Port branch-2's hbase.rsgroup.regex.<groupname> config-driven automatic
RSGroup membership onto master's RSGroupInfoManagerImpl, adapting it to
master's compute-on-demand default-group pattern instead of branch-2's
persist-on-every-change background thread.

Also fix ServerEventsListenerThread's change detection: it previously
diffed newly computed auto-managed membership against a private
in-memory snapshot instead of the live holder state. An out-of-band
admin mutation (e.g. moveServersToRSGroup) could desync that snapshot
from reality, and if a later recompute coincidentally matched the stale
snapshot, the thread would skip persisting -- permanently freezing a
phantom server in the cached/persisted group. Comparing against
holder.groupName2Group directly removes the stale baseline.
No functional change; reflows comment line breaks to match the
project's formatter output.
…ng the way

RSGroupInfoManagerImpl: add handleServerEvent(), called synchronously from
serverAdded/serverRemoved. A recompute that only affects 'default'
membership now applies immediately in-memory instead of waking the
background ServerEventsListenerThread; only a regex-governed group's
membership change (which must be persisted to hbase:rsgroup/ZK) hands
off to that thread. Also fix the thread's run loop to wait for a
serverChanged() signal before recomputing, instead of recomputing once
unconditionally on start and only waiting after.

TestRSGroupsBase: strengthen addGroup/removeGroup assertions (ported
from branch-2) to verify servers actually land where expected, and
switch the table-membership check to ADMIN.getRSGroup(TableName)
instead of defaultInfo.getTables() -- MasterRpcServices.getRSGroupInfo
(the live RPC backing ADMIN.getRSGroup(String)) never resolves table
membership from TableDescriptor attributes, unlike listRSGroups() or
getRSGroup(TableName).

TestRegexBasedRSGroupMembership: drop two triggerListenerCycle() calls
(in afterMethod()'s dead hadLeftoverRegex guard, and in clearRegex())
that forced a synthetic server event to mask a regex-governed group's
membership from going stale after a config change. That staleness is
expected -- recompute is event-driven by design, matching branch-2's
semantics -- and removeGroup()/VerifyingRSGroupAdmin#verify() already
tolerate it correctly without help. Also fix several comments left
over from the branch-2 port referencing RSGroupAdminServer and
moveTableRegionsToGroup, neither of which exist on master; point them
at the real master-side methods (RSGroupInfoManagerImpl#setRSGroup /
moveTablesAndWait / moveServers, MasterRpcServices).
Ported INFO-level log lines from branch-2's original regex-based
RSGroup membership feature commits that were missing from master's
adapted implementation: config resolution, hostname matching,
server-to-group resolution, refresh entry/exit, and
validation entry/exit. Also added a server count to the existing
"Updated auto-managed RSGroup servers" log, matching branch-2's
equivalent message.
RSGroupInfoManagerImpl#updateAutoManagedRSGroupServers() used to take a
precomputed assignments map as a parameter. ServerEventsListenerThread#run()
computed that map -- and the comparison against live group state used to
decide whether an update was even needed -- before acquiring the enclosing
instance's lock, the same lock the method itself (and every other mutator)
relies on to serialize access to holder.groupName2Group. A concurrent
admin-driven change (moveServers, flushConfig, etc.) landing in that window
would be silently clobbered by the stale assignments the background thread
had already computed. Make the method self-contained and parameterless: it
now reads holder.groupName2Group, recomputes the auto-managed assignment,
compares against current state, and applies the update, all under its own
synchronized block, so the whole decide-and-apply sequence is atomic.

Separately, ServerEventsListenerThread's own wait/notify signal used a
boolean changed flag that was only cleared after calling
updateAutoManagedRSGroupServers() -- which can block on I/O (hbase:rsgroup
table and ZK writes via flushConfig). A new server event arriving while that
call was in flight would set changed = true, but the subsequent
unconditional changed = false once the call returned would wipe that signal
with nothing left to wake the thread for it. Replace the boolean with an int
eventCount: serverChanged() increments it, and the run loop decrements it by
exactly one per completed pass, so a surplus signal forces an extra
(idempotent) pass instead of being lost.

Also add INFO-level logging for serverAdded/serverRemoved callbacks,
matching the logging already present on the other paths through this class.

TestRegexBasedRSGroupMembership: fix a ~3%-flaky crash-target assumption in
testUngracefulCrashOfRegexMemberReassignsRegionsOnlyToOtherGroupMembers --
retainAssignment's random fallback (BaseLoadBalancer#randomAssignment)
places each region independently, so it's possible for all 6 of this test's
regions to land on sn2 alone instead of being split with sn1. Crash whichever
server actually ended up holding regions, instead of hardcoding sn1, so
ServerCrashProcedure always has something to reassign and
roundRobinAssignment is guaranteed to fire.
…ckground thread

Defensive guard around the eventCount-- in ServerEventsListenerThread.run();
the decrement already only runs once the while-loop confirms eventCount > 0,
so this is a no-op in practice but makes that invariant explicit.
@virajjasani
virajjasani self-requested a review October 2, 2026 19:19
…sting it

Servers of regex-governed RSGroups (hbase.rsgroup.regex.<group>) are no longer
stored in hbase:rsgroup or ZooKeeper. Membership is recomputed from live
servers on every server event and on master start; stale copies stored in
other groups are moved to default and persisted once. Server events are
handled synchronously, so the background event thread and the flushConfig
membership check are removed.

- moveServers rejects moves that contradict a regex
- offline mode allows changes to regex-governed group servers
- admin-managed and regex-matched server sets are kept disjoint
- renaming a regex-governed group is allowed
- updateRSGroupConfig updates a copy and flushes the new group map
- docs: log levels and strong advice to enable hbase.rsgroup.fallback.enable
- tests: strengthened regex membership tests and VerifyingRSGroupAdmin checks
@virajjasani

Copy link
Copy Markdown
Contributor

There are some refguide and checkstyle warnings.

Sanjeet Malhotra added 2 commits October 8, 2026 14:17
- updateRSGroupConfig: iterate over a snapshot of the keys when clearing
  configuration on the copy; removing while iterating threw
  ConcurrentModificationException
- getRegexGroupMap: skip entries whose value is null (a concurrent
  configuration reload can remove a key between name and value reads)
- checkMoveAgreesWithRegex: no longer prints hbase.rsgroup.regex.null for a
  server that matches no regex
- Stale-copy WARN no longer advises a move that would be rejected; stale
  copies are dropped and the server lands in its regex group
- Drop the per-server "does not match any regex" INFO line
- Docs: regex removal and enable behavior, dynamic RSGroups, Dead Nodes,
  disable runbook, same config on all masters, ReDoS note, dynamic.mdx note
…ity.mdx

The refguide build runs prettier --check on the docs and failed on the two
callouts added for regex-based RSGroup membership.
@apurtell
apurtell requested a balanced review from Copilot October 8, 2026 16:54

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

9 open findings
What changed in this PR

Adds regex-driven automatic RegionServer membership assignment to RSGroups (HBASE-30411), including docs and tests to validate behavior and operational guidance.

Changes:

  • Implement regex-based auto-managed RSGroup server membership (computed from live servers; not persisted) and enforce “regex wins” against conflicting manual moves.
  • Add extensive black-box tests for regex membership, persistence semantics, crash/fallback behavior, and admin operations.
  • Update website documentation to describe regex-based membership, caveats, and recommended fallback settings.
File Description
hbase-website/​app/​pages/​_docs/​docs/​_mdx/​(multi-page)/​operational-management/​region-and-capacity.mdx Documents regex-governed RSGroups, dynamics vs persisted membership, and operational caveats.
hbase-website/​app/​pages/​_docs/​docs/​_mdx/​(multi-page)/​configuration/​dynamic.mdx Clarifies when regex config reloads take effect (next server event / master restart).
hbase-server/​src/​main/​java/​org/​apache/​hadoop/​hbase/​rsgroup/​RSGroupInfoManagerImpl.java Core implementation: regex config parsing, auto-managed membership recomputation, persistence stripping, and move constraints.
hbase-server/​src/​main/​java/​org/​apache/​hadoop/​hbase/​rsgroup/​RSGroupBasedLoadBalancer.java Adds test-only toggles and assignment-path call tracking for assertions.
hbase-server/​src/​test/​java/​org/​apache/​hadoop/​hbase/​rsgroup/​TestRegexBasedRSGroupMembership.java New comprehensive test suite for regex-based membership behavior.
hbase-server/​src/​test/​java/​org/​apache/​hadoop/​hbase/​rsgroup/​TestRSGroupsBase.java Strengthens base test helpers with additional assertions around group membership cleanup.
hbase-server/​src/​test/​java/​org/​apache/​hadoop/​hbase/​rsgroup/​VerifyingRSGroupAdmin.java Updates verification logic to account for non-persisted regex-governed membership.
hbase-server/​src/​test/​java/​org/​apache/​hadoop/​hbase/​util/​TestRegionMoverFilterRSGroupServers.java Updates comment reference for default/auto-managed membership computation.

🧠 Review effort: Lite


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +103 to +119
@RestrictedApi(explanation = "Should only be called in tests", link = "",
allowedOnPath = ".*/src/test/.*")
public void setFallbackEnabledForTest(boolean enabled) {
this.fallbackEnabled = enabled;
}

// Test-only call-tracking flags, set true whenever the corresponding assignment method below is
// invoked. Reset via resetAssignmentCallFlagsForTest() before the operation under test.
private volatile boolean roundRobinAssignmentInvoked = false;
private volatile boolean retainAssignmentInvoked = false;
private volatile boolean randomAssignmentInvoked = false;

@RestrictedApi(explanation = "Should only be called in tests", link = "",
allowedOnPath = ".*/src/test/.*")
public boolean isRoundRobinAssignmentInvoked() {
return roundRobinAssignmentInvoked;
}
Comment on lines +133 to +139
@RestrictedApi(explanation = "Should only be called in tests", link = "",
allowedOnPath = ".*/src/test/.*")
public void resetAssignmentCallFlagsForTest() {
roundRobinAssignmentInvoked = false;
retainAssignmentInvoked = false;
randomAssignmentInvoked = false;
}
return;
}
}
LOG.info("No changes in auto-managed RSGroup server membership.");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah this has to go!

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed this log line.

Comment on lines +839 to +842
static Map<String, Pattern> getRegexGroupMap(Configuration conf) {
Map<String, String> rsGroupNameToRegexMap = conf.getPropsWithPrefix(RS_GROUP_REGEX_PREFIX);
Map<String, Pattern> rsGroupNameToPatternMap = new HashMap<>();
for (Map.Entry<String, String> e : rsGroupNameToRegexMap.entrySet()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same!

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to debug log level.

Comment on lines +865 to +879
LOG.info("Resolved regex-based RSGroup membership config: {}", rsGroupNameToPatternMap);
return rsGroupNameToPatternMap;
}

// Called by ServerEventsListenerThread. Presume it has lock on this manager when it runs.
private SortedSet<Address> getDefaultServers(List<RSGroupInfo> rsGroupInfoList) {
// Build a list of servers in other groups than default group, from rsGroupMap
Set<Address> serversInOtherGroup = new HashSet<>();
for (RSGroupInfo group : rsGroupInfoList) {
if (!RSGroupInfo.DEFAULT_GROUP.equals(group.getName())) { // not default group
serversInOtherGroup.addAll(group.getServers());
private static List<String> getMatchingRSGroupNames(String hostname,
Map<String, Pattern> rsGroupNameToPatternMap) {
List<String> rsGroupNames = new ArrayList<>();
for (Map.Entry<String, Pattern> e : rsGroupNameToPatternMap.entrySet()) {
if (e.getValue().matcher(hostname).matches()) {
rsGroupNames.add(e.getKey());
}
}
LOG.info("Hostname '{}' matches RSGroup name(s) {}", hostname, rsGroupNames);
return rsGroupNames;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same!!

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed this log line.


private Map<String, SortedSet<Address>>
computeAutoManagedRSGroupServers(Collection<RSGroupInfo> existingGroups) {
LOG.info("Computing auto-managed RSGroup server membership.");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same!!!

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed this log line.

entry.getKey(), RS_GROUP_REGEX_PREFIX, entry.getKey());
}
}
LOG.info("Computed auto-managed RSGroup server membership: {}", result);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same!!!!

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to debug log level.

private void checkGroupName(String groupName) throws ConstraintException {
if (!groupName.matches("[a-zA-Z0-9_]+")) {
if (!GROUP_NAME_PATTERN.matcher(groupName).matches()) {
throw new ConstraintException("RSGroup name should only contain alphanumeric characters");
}

void stopCapturing() {
this.logger.removeAppender(this.appender);
sanjeet006py pushed a commit to sanjeet006py/hbase that referenced this pull request Oct 8, 2026
…isted for table readers

Backport of the on-the-fly regex RSGroup membership from apache#8733 to branch-2.

Membership of regex-governed RSGroups is always recomputed from the live servers and the
configured regexes. Servers stored for such a group are ignored on refresh, and stale copies
in other groups are dropped. Unlike master, the recomputed membership is still persisted to
hbase:rsgroup and ZooKeeper, because RSGroupTableAccessor readers (RegionMover and the master
web UI) read the table directly. The server-event thread flushes after every recompute; if the
flush fails and the default group is unchanged, the change is applied in memory only.

Also validate that explicit server moves agree with the configured regexes, and update the
RSGroup config on a copy of the group.

Tests: extend TestRegexBasedRSGroupMembership, including checks that stored membership matches
live membership and that a stale stored copy is replaced after a master restart.
Sanjeet Malhotra added 2 commits October 9, 2026 16:36
The invalid-name WARN for hbase.rsgroup.regex.<group> and the
ConstraintException from checkGroupName now print GROUP_NAME_PATTERN
instead of a hardcoded description, so the messages follow the pattern.
- Say that hbase.rsgroup.regex.* changes are applied by restarting all
  HMasters (recommended) or, after a reload, only at the next server event
- Update the regex group removal and RegionServer Grouping removal steps to
  restart the HMasters instead of reloading
- Document that adding or changing a regex drops unmatched servers later
- Note that add_rsgroup and rename do not apply a regex
@virajjasani

Copy link
Copy Markdown
Contributor

@sanjeet006py just add information related to auto-managed vs admin-managed groups and the purpose behind introducing auto-managed regex-based rsgroup in Javadoc. We are good to go.

…Impl

Add a class-level javadoc section that defines admin-managed and
auto-managed groups, and says why auto-managed groups are needed for
auto-scaling in cloud deployments.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants