Repository navigation
PIP-183: Reduce unnecessary REST call in broker #16422
Description
Motivation
The design of admin API now is such that: when handle a partitioned topic request, the broker will query the topic's partition meta, and then use the internal admin client to query all the non-partitioned topics (I.e. the suffix of the topic name is -partition-),
even if the non-partitioned topic is owned by the broker, which will cause unnecessary REST call in the broker.
we can call the methods directlly, who handle the non-partitioned topic, to reduce the unnecessary REST call.
Goal
- Try to call the methods directlly if the non-partitioned topic is owned by the broker
Implementation
-
We need to check all the place where
org.apache.pulsar.broker.PulsarService#getAdminClientis invoked inorg.apache.pulsar.broker.admin.impl.PersistentTopicsBase -
take
internalGetPartitionedStatsfor example:- Original:
for (int i = 0; i < partitionMetadata.partitions; i++) {
try {
topicStatsFutureList
.add(pulsar().getAdminClient().topics().getStatsAsync(
(topicName.getPartition(i).toString()), getPreciseBacklog, subscriptionBacklogSize,
getEarliestTimeInBacklog));
} catch (PulsarServerException e) {
asyncResponse.resume(new RestException(e));
return;
}
}
- Suggest to do like this:
for (int i = 0; i < partitionMetadata.partitions; i++) {
TopicName topicNamePartition = topicName.getPartition(i);
topicStatsFutureList.add(
pulsar().getNamespaceService().isServiceUnitOwnedAsync(topicName)
.thenCompose(owned -> {
if (owned) {
// local call
return getTopicReferenceAsync(topicNamePartition)
.thenCompose(topic ->
topic.asyncGetStats(getPreciseBacklog, subscriptionBacklogSize,
getEarliestTimeInBacklog));
} else {
// call from admin client
try {
pulsar().getAdminClient().topics().getStatsAsync(topicNamePartition.toString()),
getPreciseBacklog, subscriptionBacklogSize, getEarliestTimeInBacklog)
} catch (PulsarServerException e) {
throw new RestException(e);
}
}
})
);
Activity
[-]PIP-181: Reduce unnecessary REST call between brokers[/-][+]PIP-181: Reduce unnecessary REST call in broker[/+]HQebupt commentedon Jul 7, 2022
@AnonHxy We have a conflict in the PIP number here. Could you please change the number ?
#16274
#16250
[-]PIP-181: Reduce unnecessary REST call in broker[/-][+]PIP-183: Reduce unnecessary REST call in broker[/+]Jason918 commentedon Jul 7, 2022
@AnonHxy changed to 183
AnonHxy commentedon Jul 7, 2022
Thanks @Jason918 @HQebupt
github-actions commentedon Aug 7, 2022
The issue had no activity for 30 days, mark with Stale label.
merlimat commentedon Oct 24, 2026
Thanks @AnonHxy for the proposal. PIP-183 was discussed on dev@ (https://lists.apache.org/thread/4p8wh57c8tlx7gn95z30nyvf9d3dg56v) but never went to a vote. Part of the idea landed as regular improvements: #16916 (partitioned stats) and #16946 (replicated subscription status) call the topic directly when the broker owns the partition. Since there has been no activity on the broader change since 2022, I'm closing this as not planned. Anyone is welcome to restart it as a pip/ document following pip/README.md, or to send further improvements of this kind as regular PRs.