Visitar URL original
fix(tcp): fix ignored proxy selector port and property updates by lymerin · Pull Request #7497 · apache/shenyu · GitHub
Skip to content

fix(tcp): fix ignored proxy selector port and property updates - #7497

Merged
Aias00 merged 1 commit into
apache:masterfrom
lymerin:7471
Oct 10, 2026
Merged

Aias00 merged 1 commit into
apache:masterfrom
lymerin:7471

Conversation

@lymerin

@lymerin lymerin commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #7471

Problem

  • Editing an existing TCP proxy selector's forwardPort or props updates the database, but Admin only publishes upstream changes.
  • The TCP handler and factory also skip existing selector names without comparing the listener configuration.

Impact

The gateway can continue listening on the old port with the old properties after an Admin update, while upstream changes are still applied independently.

Changes

  • Publish PROXY_SELECTOR/UPDATE after the Admin transaction commits so listener configuration changes reach the gateway.
  • Cache each listener together with a defensive snapshot of its effective port and properties, and skip an update only when that configuration is unchanged.
  • Preserve the existing fix: prevent duplicate TCP bootstrap server creation #7012 CompletableFuture<BootstrapServer> single-flight behavior for initial listener creation, including shared creation failures.
  • Use a per-selector ReentrantLock to serialize listener replacement and removal, with configuration rechecks after waiting for creation and after acquiring the replacement lock.
  • Recreate changed listeners without resetting existing upstream data:
    • for a different port, start the replacement before shutting down the previous listener;
    • for the same port, release the existing listener first and restore its previous configuration if the replacement fails to start.
  • Simplify clearCache() so it no longer participates in a separate operation registry.
  • Keep DataChangedEvent.getEventType() package-private and document the intentional replacement ordering.
  • Add focused regression coverage for Admin commit/rollback propagation, subscriber-driven port changes, same-port property changes, replacement failure recovery, initial-creation single-flight behavior, creation/update handoff, and replacement/removal synchronization.

Validation

  • TCP targeted tests: 21 passed, with zero failures or errors.
  • Checkstyle passed with zero violations.

Checklist

Make sure that:

@wy471x wy471x left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, @lymerin, thanks for your PR. I do not see a blocking issue; a few notes below on the new per-selector operation lock.

Verification

On 5915dc0b5:

  • ./mvnw -pl shenyu-plugin/shenyu-plugin-proxy/shenyu-plugin-tcp -am test -Dtest='TcpBootstrapFactoryTest,TcpProxySelectorDataHandlerTest,TcpUpstreamDataHandlerTest' -> 18 tests, 0 failures.
  • ./mvnw -pl shenyu-admin -am test -Dtest=ProxySelectorServiceTest -> 19 tests, 0 failures.
  • checkstyle:check passes for shenyu-admin and shenyu-plugin-tcp.

Notes

  • Both halves of the bug are addressed: admin now publishes a PROXY_SELECTOR UPDATE event (registered on the transaction and published only for STATUS_COMMITTED), and the gateway factory replaces the cached listener when the port or props change instead of returning early. The payload uses the same mapToData shape as the existing discovery-processor events, so subscriber lookup and namespace routing reuse the established path.
  • Comparing the listener configuration against a defensive snapshot (snapshot() plus Properties.equals) is what makes an unconditional handlerProxySelector safe: unrelated selector syncs do not restart the listener, and props are compared by content, not by object identity. That matters because a restart drops live TCP connections.
  • The two replacement branches are ordered correctly: same port -> shutdown then start (the port must be free), different port -> start then shutdown (no gap on the old port). The recovery path that re-creates the previous configuration, and the removal of the cache entry when even that fails, keep the failure modes honest - shouldRetainPreviousConfigurationAfterUpdateFailure covers both branches.
  • createServer initialising UpstreamProvider only when it has no entry is the subtle half of the fix; without it a selector update could wipe upstreams that arrived before the listener.

Possible Improvements

  • acquireOperation is a spin over putIfAbsent + join@ and the futures double as per-selector locks. Correctness looks fine (a finallyalways completes the future, and the removal is compare-and-remove), but waiters block uninterruptibly, so a critical section that never returns - for exampleshutdown()on a hung listener - would block every later update for that selector indefinitely. A per-selectorReentrantLock` (or a bounded wait with an explicit failure) would be easier to reason about and to observe in production.
  • clearCache() now takes the union of cache and operations keys and calls removeAndShutdown per selector. It is not called from inside an operation today, but if it ever were, it would self-deadlock on that selector's own entry; a comment would help.
  • DataChangedEvent.getEventType() was widened to public only so the new test can assert the event type. Since ProxySelectorServiceTest lives in another package, keeping the accessor package-private and asserting through getGroupKey() and the payload would avoid widening the API.
  • The asymmetry between the two replacement branches (shutdown-before-start vs start-before-shutdown) is deliberate but subtle; one comment per branch would stop a future reader from "simplifying" it into a port conflict.

@lymerin

lymerin commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

Hi, @lymerin, thanks for your PR. I do not see a blocking issue; a few notes below on the new per-selector operation lock.

Verification

On 5915dc0b5:

  • ./mvnw -pl shenyu-plugin/shenyu-plugin-proxy/shenyu-plugin-tcp -am test -Dtest='TcpBootstrapFactoryTest,TcpProxySelectorDataHandlerTest,TcpUpstreamDataHandlerTest' -> 18 tests, 0 failures.
  • ./mvnw -pl shenyu-admin -am test -Dtest=ProxySelectorServiceTest -> 19 tests, 0 failures.
  • checkstyle:check passes for shenyu-admin and shenyu-plugin-tcp.

Notes

  • Both halves of the bug are addressed: admin now publishes a PROXY_SELECTOR UPDATE event (registered on the transaction and published only for STATUS_COMMITTED), and the gateway factory replaces the cached listener when the port or props change instead of returning early. The payload uses the same mapToData shape as the existing discovery-processor events, so subscriber lookup and namespace routing reuse the established path.
  • Comparing the listener configuration against a defensive snapshot (snapshot() plus Properties.equals) is what makes an unconditional handlerProxySelector safe: unrelated selector syncs do not restart the listener, and props are compared by content, not by object identity. That matters because a restart drops live TCP connections.
  • The two replacement branches are ordered correctly: same port -> shutdown then start (the port must be free), different port -> start then shutdown (no gap on the old port). The recovery path that re-creates the previous configuration, and the removal of the cache entry when even that fails, keep the failure modes honest - shouldRetainPreviousConfigurationAfterUpdateFailure covers both branches.
  • createServer initialising UpstreamProvider only when it has no entry is the subtle half of the fix; without it a selector update could wipe upstreams that arrived before the listener.

Possible Improvements

  • acquireOperation is a spin over putIfAbsent + join@ and the futures double as per-selector locks. Correctness looks fine (a finallyalways completes the future, and the removal is compare-and-remove), but waiters block uninterruptibly, so a critical section that never returns - for exampleshutdown()on a hung listener - would block every later update for that selector indefinitely. A per-selectorReentrantLock` (or a bounded wait with an explicit failure) would be easier to reason about and to observe in production.
  • clearCache() now takes the union of cache and operations keys and calls removeAndShutdown per selector. It is not called from inside an operation today, but if it ever were, it would self-deadlock on that selector's own entry; a comment would help.
  • DataChangedEvent.getEventType() was widened to public only so the new test can assert the event type. Since ProxySelectorServiceTest lives in another package, keeping the accessor package-private and asserting through getGroupKey() and the payload would avoid widening the API.
  • The asymmetry between the two replacement branches (shutdown-before-start vs start-before-shutdown) is deliberate but subtle; one comment per branch would stop a future reader from "simplifying" it into a port conflict.

Thank you for the detailed review and suggestions. I’ll take a closer look at the locking implementation, API exposure, and comments, and follow up here.

@lymerin
lymerin marked this pull request as draft October 9, 2026 00:15
@lymerin
lymerin marked this pull request as ready for review October 9, 2026 10:12
@lymerin

lymerin commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the detailed review. I revisited the synchronization design after your comments.

The original CompletableFuture<Void> operation map came from trying to extend the single-flight idea introduced in #7012 beyond initial creation, so that creation, replacement, and removal could use a similar synchronization model.

After reviewing #7471 again, I think that extends the abstraction further than this issue requires. I’ve therefore narrowed the design.

1. Per-selector operation lock

CompletableFuture<BootstrapServer> is restored to the original #7012 role: initial creation single-flight, where concurrent callers participate in the same creation and share its result or failure.

Existing listener replacement and removal now use a per-selector ReentrantLock, since they only need mutual exclusion while modifying the same cached listener.

After waiting for an in-flight creation, the update path re-reads the cache; after acquiring the replacement lock, it re-reads and compares the configuration again. This avoids applying an update based on stale state.

I did not add a bounded wait or timeout/cancellation policy. A ReentrantLock still cannot make a hung shutdown() bounded, but handling that correctly would require defining what happens to the pending configuration update after timeout, including retry or reconciliation semantics. Since #7471 is specifically about ignored forwardPort / props updates, I’d prefer not to expand this fix into a broader lifecycle recovery design.

2. clearCache()

I removed the previous operation-registry handling from clearCache(). It now works only from the cached selector set, so the self-deadlock scenario you pointed out for the operation registry no longer exists.

3. DataChangedEvent.getEventType()

I restored getEventType() to package-private and removed the test-only API widening.

4. Replacement ordering

I added comments explaining the intentional asymmetry:

  • same port: shut down the current listener first because the replacement cannot bind to the same port;
  • different port: start the replacement first so a startup failure does not take the current listener offline.

There is one API point where I’d like your guidance. TcpBootstrapFactory.cache() previously accepted:

cache(String selectorName, BootstrapServer server)

but #7471 requires the cache to retain the listener’s effective forwardPort and props, so the current version uses:

cache(TcpServerConfiguration configuration, BootstrapServer server)

I found only test callers of this method in the repository, but changing the signature is still technically a public API change.

My preference is to keep the new signature. The old API does not provide enough information to build a correct configuration snapshot, and keeping a compatibility overload would require inventing an unknown/default port and properties, which could make later configuration comparisons incorrect and trigger unnecessary replacements.

However, whether this method needs source compatibility is a project-level API decision, so I’d like your opinion: is changing this public method acceptable here, or would you prefer that we preserve the old signature in some other way?

@Aias00 Aias00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: fix(tcp) — fix ignored proxy selector port and property updates

Reviewed the current revision (head a577cf9). Verified DataChangedEvent(ConfigGroupEnum, DataEventTypeEnum, List) ctor + getGroupKey() exist, and ApplicationEventPublisher is injected via the ProxySelectorServiceImpl constructor.

Why approve

Two solid, related fixes:

  1. Event propagation: ProxySelectorServiceImpl.update now publishes DataChangedEvent(PROXY_SELECTOR, UPDATE, …) only after the transaction commits (via TransactionSynchronization.afterCompletion gated on STATUS_COMMITTED). This is the actual fix for TCP proxy selector config (port/property) changes not reaching gateway nodes. The test testUpdatePublishesListenerConfigurationAfterCommit covers both the committed (event published, payload carries name/port/props) and rolled-back (no event) paths.

  2. Config re-apply: TcpBootstrapFactory now caches CachedServer(server + configuration snapshot) and adds createOrUpdateBootstrapServer that compares a defensive snapshot and replaces the listener when the port or properties change. Same-port updates release the current listener before rebinding; different-port updates keep the old listener available until the replacement starts. It preserves the existing upstream cache (only initializes upstreams if not already present), serializes replacement/removal via per-selector ReentrantLocks, and recovers the previous config on failure. TcpProxySelectorDataHandler.handlerProxySelector now calls createOrUpdateBootstrapServer instead of early-returning when already cached — this was precisely why port/property updates were previously ignored.

Extensive concurrency tests added (shouldApplyDifferentConfigurationAfterWaitingForCreation, shouldSerializeReplacementAndRemovalForSameSelector, shouldReplacePropertiesOnSamePort, shouldRetainPreviousConfigurationAfterUpdateFailure), plus subscriber/upstream tests. build + integrated tests pass.

Non-blocking

None. Clean, thread-safe refactor with strong test coverage.

LGTM — approving.

@Aias00
Aias00 merged commit 76122a4 into apache:master Oct 10, 2026
51 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] TCP proxy selector updates ignore changed forwardPort and properties

3 participants