[improve][pip] PIP-502: Improve replication observability in the Admin API - #27472

Rewind reason and readException cannot show the publish/read failures they document

On current master: - A replicated send that completes exceptionally goes through ProducerSendCallback.sendComplete. It sets skipReadResultDueToCursorRewind, cancelReadRequested and rewindRequested directly and never calls beforeTerminateOrCursorRewinding, so no reason is recorded. - FAILED_PUBLISHING and DISCONNECTING are set and then cleared by back-to-back calls. - A failed cursor read gets an empty result. Its task is marked submissionComplete and becomes isDone(), so it drops out of the proposed 'unfinished tasks' list. readException (line 113) is then reset when the task is recycled. An operator polling replicatorInternalStats during a publish-failure or read-failure loop sees reasonOfWaitForCursorRewinding = null and no readException. The PIP defines null as 'no such event occurred', so the response points away from the real cause. In practice only FETCHING_SCHEMA, and briefly TERMINATING, is ever observable. Two motivating causes, interrupted sends/dedup retries and failed storage reads, stay invisible.

• Send failure: PersistentReplicator.java:649-668 has no beforeTerminateOrCursorRewinding call; its comment says 'this recovery has no outstanding stage to wait for'.
• Immediate clear: GeoPersistentReplicator.java:236-237 and 299-300, and ShadowReplicator.java:115-116 and 144-145, call doRewindCursor right after setting Disconnecting or Failed_Publishing.
• Lifecycle: PersistentReplicator.java:1183-1202 sets the reason in beforeTerminateOrCursorRewinding and sets it to null in doRewindCursor. Only Fetching_Schema (GeoPersistentReplicator.java:256-272) waits on an async stage.
• Read failure: readEntriesFailed sets readException and an empty entries list (PersistentReplicator.java:763-771). processReadResult's finally block sets submissionComplete (595-600). isDone() is true for an empty result (1096-1104). recycle() clears readException (1080-1088).

Replace these with state that persists until superseded and carries a time:

• lastReadFailure and lastPublishFailure ({timestamp, exception class or short message});
• waitingForSchemaFetch (boolean) or pendingCursorRewinds (count), instead of a single overwritable reason.
  Alternatively, specify the broker changes that would record and retain FAILED_PUBLISHING in the send-failure path, and define readException on the most recent read rather than on unfinished tasks.

Updated PIP-502 in 9ae75fd. I removed reasonOfWaitForCursorRewinding and added lastReadFailure and lastPublishFailure. Each record carries the source broker timestamp and a sanitized, length-limited exception message. A failed read stays visible after its task is recycled, and a failed send is recorded even when its callback triggers a rewind directly. Each replicator keeps the latest read and publish failures independently across successful retries, rewinds and producer reconnects. inFlightTasks[].readException remains scoped to its batch.

One correction to my earlier reply about the record format. In 6afef32, lastReadFailure.message preserves the read exception message verbatim, and lastPublishFailure.message contains only the exception class name. Each record still retains its timestamp after the batch is recycled or the cursor rewinds.

createAt has no reliable cheap source and its guidance contradicts the default replicationStartAt

Case 1: ZooKeeper metadata store with replication newly enabled. ManagedCursorImpl.initialize persists the new cursor with a null Stat, so MetaStoreImpl creates the znode, and ZKMetadataStore returns new Stat(name, 0, 0, 0, ...) for creates. The in-memory cursorLedgerStat keeps creationTimestamp 0 until the first cursor-ledger switch or a topic reload, which is exactly while a newly enabled replicator is stuck. Case 2: with the default replicationStartAt=latest, the new replication cursor opens at the end of the topic. - A cheap implementation that reads the in-memory Stat reports 1970-01-01 for a stuck, newly created replicator. An accurate one needs a metadata-store read on every stats call. - Monitoring #1 (line 126) says a recent createAt plus a large backlog means historical catch-up. Under the default config that is false: the backlog has built up since the cursor was created, meaning replication is not keeping up. - The correct value is already available from info-internal.

• ManagedCursorImpl.java:922-942 (initialize calls persistPositionMetaStore), 3447-3487 (asyncUpdateCursorInfo with cursorLedgerStat, then updateCursorLedgerStat), 4126-4128 (switchToNewLedger re-persists).
• MetaStoreImpl.java:272-287: a null Stat means expectedVersion -1 (create).
• ZKMetadataStore.java:375 and 519: a create returns a Stat with creationTimestamp 0. setData returns the real ctime (529-533, 578-582).
• ServiceConfiguration.java:1946-1950: replicationStartAt defaults to latest. PersistentTopic.java:2791-2802 opens the cursor at Latest unless earliest is configured.
• ManagedLedgerFactoryImpl.java:1013-1021: info-internal reads the cursor znode and returns creationDate.

Drop createAt from topic stats and point operators to 'pulsar-admin topics info-internal' (cursors["pulsar.repl."].creationDate). If it stays, define the source (for example a creation time persisted in the cursor properties) and return an unknown value (0) as null. Either way, qualify Monitoring #1 with replicationStartAt=earliest.

Agreed. I removed createAt from the proposed topic stats in a954067. The monitoring section now points to the existing cursors["pulsar.repl.<target-cluster>"].creationDate in pulsar-admin topics info-internal. It also clarifies that with the default replicationStartAt=latest, the backlog accumulated after cursor creation. With earliest, the backlog can include retained messages published before replication was enabled.

replicatorInternalStats is under-specified and duplicates the internalStats endpoint

Voting on or implementing the endpoint as written. Implementers must guess the Java admin API, the error contract and the response shape. Internal enums copied verbatim become a public contract, so later replicator or client refactors would break the API. The 503 behavior diverges from every other stats endpoint without achieving its stated goal.

1. No Java admin API (Topics method, async variant, return type or data classes), although pip/README.md treats admin/client APIs as public API.
2. Only 503 is documented. 307/401/403/404/412 are missing, and so is the response for a partitioned-topic name, which line 69 says is unsupported.
3. stats and internalStats load an unloaded topic: getTopicReferenceAsync → BrokerService.getTopicIfExists → getTopic(topic, false) (PersistentTopicsBase.java:1556-1577, 4957-4961; BrokerService.java:1386-1387). The PIP's own Monitoring steps start from stats, so returning 503 here does not prevent loading.
4. The response is wrapped in an array, [ {"cluster-b": {...}} ] (lines 72-91), while stats.replication is a plain map.
5. Lines 62 and 101 copy the package-private client HandlerState.State (HandlerState.java:27-48; getState() is protected) and the protected PersistentReplicator.ReasonOfWaitForCursorRewinding (PersistentReplicator.java:121-127).
6. Line 100 mentions an 'inconclusive remote check' without saying whether a GET triggers a cross-cluster call. The replication producer already tracks incompatibility locally (SchemaState.Broken at ProducerImpl.java:3076-3099; pause mode enabled for replication producers at ProducerImpl.java:233).
7. Line 102 says inFlightTasks is null when the replicator 'has not started', but the list always exists and the replicator returns to DISCONNECTED after an idle disconnect.

Prefer a 'replicators' map on PersistentTopicInternalStats, served by the existing internalStats and partitioned-internalStats endpoints and 'pulsar-admin topics stats-internal'. That gives the same GET_STATS authorization and loading semantics, and the fields can be documented API values rather than Java enum names.
If a separate endpoint is kept, specify:

• the Java admin methods and classes;
• every response code, including the partitioned-name case;
• a map-shaped response;
• how each value is derived (local only, no remote call);
• why a separate endpoint was chosen, in Alternatives.

I adopted the replicators map on PersistentTopicInternalStats in 8569257. Both stats-internal and partitioned-stats-internal expose it through the existing Admin methods. The schema check contacts the target only when the last send started more than a minute ago and a pending send has a schema to check. The inFlightTasks wording for an idle, disconnected instance is still open.

6afef32 clarifies the remaining inFlightTasks case. An existing replicator with no unfinished reads returns [], including after an idle disconnect. A target with no replicator instance returns null.

Required PIP sections are missing, including Security for a new endpoint that exposes error text

The PIP goes from Monitoring straight to Links. Goals has no In Scope/Out of Scope, and there is no Design & Implementation Details, Security Considerations, Backward & Forward Compatibility or Alternatives section. pip/TEMPLATE.md requires all of them. - Voters cannot see who may call the new endpoint or what it reveals. If it reuses TopicOperation.GET_STATS, as stats, internalStats and info-internal do, PulsarAuthorizationProvider grants it to any role that can produce or consume on the topic. Non-admin client roles would then see broker-side read-error text and internal read positions. 'Sanitized' is not defined. - Compatibility is not stated: the new getters on the public ReplicatorStats interface are also inherited by NonPersistentReplicatorStats, and older admin clients, mixed-version brokers and rollback are not described.

• pip/TEMPLATE.md:44-58, 76-84, 131-164; pip/README.md step 3 ('Write the proposal following the section outlined by the template').
• GET_STATS is used at PersistentTopicsBase.java:1558, 1567 and 1581.
• PulsarAuthorizationProvider.java:720-735: for non-admin roles, GET_STATS → canLookupAsync.
• NonPersistentReplicatorStats.java:24 extends ReplicatorStats.
• ObjectMapperFactory.java:169: FAIL_ON_UNKNOWN_PROPERTIES=false, so old admin clients ignore the new fields. This is worth stating.

Add the missing sections:

• Security: name the TopicOperation; say whether tenant-admin is required for the diagnostics; define what readException exposes (for example exception class plus a bounded message with no hosts).
• Compatibility: the new ReplicatorStats getters (default methods or not), mixed-version brokers during a rolling upgrade, old admin clients and rollback.
• Alternatives: extending internalStats, OpenTelemetry metrics for alertable signals, logs.
• Design & Implementation Details: where each timestamp is written and how producerState is read across the client package boundary.
• Goals: In Scope and Out of Scope.

96a04c1 adds the mixed-version and rollback behavior. An older Java Admin client ignores unknown fields from a newer broker. With an older broker, the newer CLI omits the absent fields. This addresses the client-output part of your comment.

6afef32 defines what callers see. lastReadFailure.message and the batch’s readException expose the read exception message verbatim. Producer startup and publish failure records expose only the exception class name. Operators can use the timestamp to find details in the source broker logs.

These fields inherit the GET_STATS permission on internalStats. With the built-in authorization provider, a role with produce or consume permission on the topic can therefore see the full read error.

00c1b65 specifies that each new ReplicatorStats getter has a default implementation returning null. NonPersistentReplicatorStats inherits those defaults. The implementation tests will cover recompiling unchanged implementation source and calling every new getter on a class file compiled against the old API.

I also checked getLastSendStartedAt() in isolation on JDK 21 with implementations of both interfaces. Both recompiled unchanged, and the old class files returned null when a caller compiled against the updated interface invoked the new getter.

I intentionally omitted the standalone Security Considerations, Alternatives, In Scope / Out of Scope, and Design & Implementation Details sections. The scope of this PIP is the replication fields added to stats and internalStats. The Public API section describes their behavior, including which exception details callers can see and when a stats request triggers a remote schema check. I’d prefer to keep clarifications about those behaviors with the relevant fields.

Timestamp type and zone differ from existing stats conventions

A broker whose JVM time zone is not UTC returns stats for a replicated topic. In the same replication[cluster] object, outboundConnectedSince and inboundConnectedSince are local-offset strings (e.g. +08:00), while the new fields would be UTC 'Z' strings. Elsewhere in topic, subscription, consumer and compaction stats, last-activity times are epoch-millis longs, so tooling that computes ages arithmetically has to special-case these fields. The types cannot be changed after release.

• DateFormatter.java:31-32: ISO_OFFSET_DATE_TIME.withZone(ZoneId.systemDefault()).
• ProducerImpl.java:2283: connectedSince = DateFormatter.now().
• Epoch-millis precedents: SubscriptionStatsImpl.java:112-124 (lastAckedTimestamp, lastMarkDeleteAdvancedTimestamp); ConsumerStatsImpl.java:115-122; TopicStatsImpl.java:131-187 (lastOffloadSuccessTimeStamp, topicCreationTimeStamp, lastPublishTimeStamp); CompactionStatsImpl.java:33-36.

Use long epoch-millis fields named like the existing ones, e.g. lastCursorReadStartedTimestamp, lastSendStartedTimestamp and lastReplicationSucceededTimestamp, with 0 meaning never. If strings are kept, produce them with DateFormatter so all timestamps in the object share one format.

90bc517 changes all proposed timestamps to signed 64-bit Unix epoch milliseconds, including the timestamps in failure records. I kept the ...At names and used null for an event the replicator has never observed or a value the broker cannot supply.

Partitioned-stats aggregation and non-persistent behavior of the new stats fields are undefined

Calling partitioned-stats on a replicated partitioned topic, or stats on a replicated non-persistent topic. TopicStatsImpl.add merges per-partition replication entries through ReplicatorStatsImpl.add, which only sums rates and backlog, ANDs connected and takes the maximum delay. Unless specified, the six new fields would be null in the aggregate, indistinguishable from 'unavailable'. NonPersistentReplicatorStatsImpl extends ReplicatorStatsImpl, so non-persistent stats would also carry cursor-based fields that mean nothing there.

TopicStatsImpl.java:353-361; ReplicatorStatsImpl.java:83-94; NonPersistentReplicatorStatsImpl.java:34.

State the aggregate rule for each field (for example states null, timestamps min/max, or 'available per partition only, via perPartition=true'), and which fields non-persistent replicators populate.

2bed17a specifies that the partitioned-stats aggregate omits the new replication fields. With perPartition=true, they appear in each persistent partition’s replication entry. Likewise, partitioned-internalStats puts replicators under each returned partition and omits it at the top level. Non-persistent topics omit all the additions.

Several motivating causes get no corresponding signal

Replication slowed by the replicator dispatch-rate limit, or a producer that cannot start because of authentication, authorization, network or partition-count mismatch. Throttling leaves every proposed field looking healthy (no pending read, queue not full) while the backlog grows. Producer-start failures show only as STARTING/DISCONNECTED flapping, with the cause still only in broker logs, which is the gap the PIP sets out to close.

• PersistentReplicator.java:263-277: with no rate-limit permits, getReadLimits returns ReadLimits(-1,-1) with only a DEBUG log. 470-483: the read is deferred and retried.
• AbstractReplicator.java:223-233: logs 'Failed to create remote producer, retrying'; estimatedTimeStampProducerConnected holds the next attempt time.
• Nothing in lines 58-65 or 96-113 reflects either case.

Either add minimal signals, such as lastProducerStartFailure {timestamp, exception class}, nextProducerStartAttemptTimestamp (from estimatedTimeStampProducerConnected), and a readThrottled flag or last throttled time; or list rate limiting, producer-start failures and deduplication under Out of Scope.

67d748a adds lastProducerStartFailure to internal stats with the time and cause, including failures during target-topic preparation and producer creation. In ordinary stats, replicatorHealth.message points to replicatorDispatchRate when progress has stalled for a minute and a read is waiting for dispatch permits. Slow replication that still makes progress and target-side deduplication have no dedicated signal in this revision.

lastPublishFailure: the example cannot occur, and skipped invalid messages are unspecified

(1) The target stops acknowledging replicated sends. (2) A replicated message is rejected on the client side with InvalidMessageException, for example because it exceeds the target broker's maxMessageSize. (1) Replication producers are built with sendTimeout(0), so a pending send never times out and no failure record is created. The example at line 82 ('Timed out waiting for the publish acknowledgement') suggests the opposite for the PR's headline case, a target that stops acknowledging. (2) The replicator does not count InvalidMessageException as a failure. It logs 'Message persisted on remote broker' at DEBUG and acks the entry, so the message is skipped and never replicated. An implementation built on the existing failed/success branches would record no lastPublishFailure and would advance lastReplicationSucceededAt. Monitoring #6 (line 139) would then report progress after a message was silently dropped.

• AbstractReplicator.java:156-157: the replication producer uses enableBatching(false) and sendTimeout(0).

• ProducerImpl.java:289-290: the send-timeout timer is scheduled only when sendTimeoutMs > 0.

• PersistentReplicator.java:650: failed excludes InvalidMessageException. Lines 669-675: the success branch logs at DEBUG and calls cursor.asyncDelete on the entry.

• ProducerImpl.java:2927-2931 and 3217-3228: an oversized message fails through failSendOp with InvalidMessageException. ClientCnx.java:629-634: the size limit comes from the target broker's CommandConnected.

• ProducerImpl.java:2933-2936: terminal producer states (closed, terminated, fenced) fail sends immediately, which is a reachable publish failure.

• Replace the example with a reachable failure, such as a message over the target's maxMessageSize or a fenced or terminated producer.

• State that replication sends have no send timeout, so a missing acknowledgement shows up only as a stalled lastReplicationSucceededAt and a growing replicationDelayInSeconds.

• Specify that an InvalidMessageException is recorded in lastPublishFailure, ideally marked as skipped because the entry is acknowledged and not retried, and that it does not advance lastReplicationSucceededAt.

I replaced the acknowledgement-timeout example with TopicTerminatedException in 6afef32. Replication sends use sendTimeout(0), so a missing acknowledgement can leave sends pending and replication delay growing without producing a failure record.

For InvalidMessageException, the proposal follows the existing skip path. The callback calls cursor.asyncDelete() and preserves the previous lastPublishFailure because it skips the entry without retrying it. lastReplicationSucceededAt advances on that delete request. That means it can advance for an entry the target never stored. Monitoring now calls out that distinction explicitly.

sendingMessageSchemaCompatible turns internalStats into an uncached cross-cluster admin call

A replicator stalls with schema-bearing messages pending and no new send for more than a minute: the target is hung or unreachable, it stops acknowledging, or schema registration is stuck. Any caller with GET_STATS then polls stats-internal or partitioned-stats-internal. - Every internalStats call makes the source broker POST to the target's /compatibility endpoint for each qualifying target. With a hung target this adds up to the replication admin client's 30 s read timeout (10 s connect timeout if the target is unreachable) to an endpoint that today returns local state. - partitioned-internalStats fans out one internal admin call per partition with the same 30 s read timeout and silently drops partitions whose call fails. It can return incomplete data during the very outage it is used to diagnose. - Nothing caches or rate-limits the check, so each poll by any produce/consume role (GET_STATS) sends authenticated requests to the target with the broker's replication credentials. - The target endpoint requires tenant-admin. A least-privilege replication role always gets null after a wasted round trip, and making the field work means granting tenant-admin to the replication identity. - The verdict can contradict what the producer actually sees. If the target namespace disallows schema auto-update for replicators, registration fails with IncompatibleSchemaException while /compatibility reports true, so Monitoring #4 and #8 would wrongly clear the schema.

• Target endpoint: SchemasResource.java:317-334.
• It requires tenant-admin: SchemasResourceBase.java:235-237 (validateAdminAccessForTenantAsync).
• It checks only the compatibility strategy (SchemasResourceBase.java:151-170). Real registration also rejects when auto-update is disallowed (AbstractTopic.java:872-884). The replicator override is_allow_auto_update_schema_with_replicator defaults to true but can be disabled (Policies.java:124; AbstractTopic.java:888-900).
• Replication admin: PersistentTopic.java:2832 → BrokerService.getClusterPulsarAdmin. readTimeout = metadataStoreOperationTimeoutSeconds (BrokerService.java:2326; default 30, ServiceConfiguration.java:566); connect timeout default 10 s (ClientConfigurationData.java:322).
• partitioned-internalStats fans out through the internal admin client (PersistentTopicsBase.java:1831-1835; read timeout 30 s, PulsarService.java:2198) and keeps only partitions that completed successfully (PersistentTopicsBase.java:1842-1856).
• The replication producer already tracks its own registration outcome: pause mode for replication producers (ProducerImpl.java:233), IncompatibleSchemaException handling (ProducerImpl.java:1030-1059) and the Broken schema state (ProducerImpl.java:3076-3099).

Derive the value locally from the replication producer's own latest schema-registration result for the pending message: registered → true, IncompatibleSchemaException → false, otherwise null. Optionally add a lastSchemaRegistrationFailure {timestamp, exception class}. Drop the remote call.
If a remote check is kept:

• cache the verdict per replicator (for example at most once a minute);
• bound the call with a short timeout;
• in a Security section, document the tenant-admin requirement and that GET_STATS callers can trigger cross-cluster requests.

In f5a92a4, the broker checks the target only when replicationBacklog > 0, the last send started more than a minute ago, lastReplicationSucceededAt < lastSendStartedAt, and a pending send has a schema to check. For example, if another send completes after the latest send started, the next poll skips the check. A missing timestamp also skips it.

This narrows when the request runs. Repeated polls can still contact the target while all four conditions hold. The timeout and partition fan-out concerns remain, as does the target tenant-admin permission requirement. Also, a true compatibility verdict can coexist with a producer registration failure when schema auto-update is disabled. Those points remain open.

Messages dropped on InvalidMessageException are reported as healthy progress

Replicated messages are rejected on the client side with InvalidMessageException. For example, the target's maxMessageSize is smaller than the source's, and producers publish messages between the two limits. The replicator acks and skips each such entry. Under this revision every new signal then reports success while the messages never reach the target: - cursor movement counts as progress, so replicatorHealth.healthy stays true (line 75); - lastReplicationSucceededAt advances (line 70); - lastPublishFailure is left unchanged (line 126). The only trace is a DEBUG broker log. The note in Monitoring #6 (line 161) documents the behavior but gives operators no runtime signal to notice a replication drop.

• PersistentReplicator.java:650: failed excludes InvalidMessageException.

• PersistentReplicator.java:669-675: the success branch logs 'Message persisted on remote broker' at DEBUG and calls cursor.asyncDelete.

• ProducerImpl.java:2927-2931 and 3217-3228: an oversized send fails with InvalidMessageException.

• ClientCnx.java:629-634: the size limit comes from the target broker's CommandConnected.

• AbstractReplicator.java:156: replication producers have batching disabled, and chunking is off by default.

• Record skipped entries, for example a lastSkippedMessage {timestamp, exception class} record in replicators[cluster], or record them in lastPublishFailure.

• Do not count InvalidMessageException skips as progress for replicatorHealth; when a skip occurs in the window, report healthy=false with a message pointing to that record.

• Exclude skips from lastReplicationSucceededAt, or rename the field to match its cursor-delete semantics.

I agree that the target may never receive the message in this case. The existing replicator acknowledges the source entry after InvalidMessageException. I consider that a replication bug to address in a separate fix PR.

For this PIP, I would keep the documented update point for lastReplicationSucceededAt: the send callback requesting deletion of the cursor entry. Today a skipped entry can therefore advance the timestamp and keep the replicator healthy even though the target never stored the message. If the replication fix prevents that deletion for a rejected entry, the timestamp will naturally stop advancing for that entry.

Narrowed conditions still run the remote schema check on every poll during a target outage

The target cluster hangs or becomes unreachable for a replicated topic that uses a schema. The backlog grows. No send starts for over a minute because reads stop while the producer is not writable. The last send stays unacknowledged, and schema-bearing messages are pending. All four conditions hold, and an operator or monitoring job polls stats-internal or partitioned-stats-internal. - Each poll still makes the source broker POST to the target's /compatibility endpoint, with no caching and no specified timeout. Each call can wait up to the replication admin client's connect or read timeout (10 s / 30 s by default). - partitioned-internalStats fans out one call per partition under a 30 s internal read timeout and silently drops partitions that fail. It can return incomplete data during the outage it is meant to diagnose. - As the author confirmed on this thread, two more problems remain: the target endpoint needs tenant-admin, and a 'true' verdict can coexist with a real registration failure when schema auto-update is disabled for replicators.

• Why the four conditions persist: PersistentReplicator.java:588-593 keeps requesting reads after a send only while the producer is writable. Otherwise only send callbacks request reads (684-703), so lastSendStartedAt stops advancing once the target stops responding.
• Unchanged from the previous review: BrokerService.java:2326 and ClientConfigurationData.java:322 (admin timeouts); PulsarService.java:2198 and PersistentTopicsBase.java:1842-1856 (fan-out and partitions dropped on failure); SchemasResourceBase.java:235-237 (tenant-admin) and 151-170 vs AbstractTopic.java:872-884 (verdict vs registration).
• The author's reply on this thread: 'The timeout and partition fan-out concerns remain, as does the target tenant-admin permission requirement… Those points remain open.'

Derive sendingMessageSchemaCompatible locally from the replication producer's latest schema-registration result for the head pending message: registered → true, IncompatibleSchemaException → false, otherwise null. Drop the remote call. This needs no target permission and reflects what the target actually did.
If the community still wants a remote pre-check, make it explicit and bounded:

• cache one verdict per replicator for at least a minute;
• set a short timeout;
• run it off the request path;
• state the tenant-admin requirement in the PIP.

Updated in 997aa69. The proposed read-only method on GeoReplicationProducerImpl reads schemaState directly from the message at the head of pendingMessages: Ready maps to true, Broken to false, and None to null. An absent producer or head message also yields null. The producer marks replicated messages that need no schema registration as Ready, so those yield true as well. Both stats endpoints read this field locally without a cross-cluster request.