Repository navigation
Admin tool: purge old metrics reports from the relational JDBC metrics tables - #6014
Draft
sririshindra wants to merge 1 commit into
Draft
sririshindra wants to merge 1 commit into
sririshindra wants to merge 1 commit into
Conversation
…s tables The scan_metrics_report and commit_metrics_report tables, written by the persisting Iceberg metrics reporter, grow without bound. Add a purge-metrics admin tool command that deletes the reports of the given realms older than a given age from both tables, in batches of one transaction each. The delete logic is a JDBC-only bean in polaris-relational-jdbc, which owns the DDL of these tables. No schema change is needed: both tables already have a (realm_id, timestamp_ms) index.
Contributor
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
With the relational JDBC persistence, the
persistingIceberg metrics reporter writes every scan and commit report to thescan_metrics_reportandcommit_metrics_reporttables. Nothing in Polaris deletes these rows, so the tables grow without bound. This is the same problem as theeventstable, which #6013 addresses.This adds a
purge-metricsadmin tool command that deletes the scan and commit metrics reports of the given realms that are older than a given age:Operators run it periodically (for example from a cron job) to enforce a retention period. It has the same shape and options as
purge-eventsin #6013.How it works:
DELETE ... WHERE realm_id = ? AND report_id IN (SELECT report_id ... WHERE realm_id = ? AND timestamp_ms < ? LIMIT ?). The subquery form is used because PostgreSQL has noDELETE ... LIMIT. It works on PostgreSQL, CockroachDB and H2. The loop stops when a batch deletes fewer rows than the batch size, so an interrupted run can be repeated. Both tables have the primary key(realm_id, report_id)and every delete is scoped to one realm.(realm_id, timestamp_ms)index (idx_scan_report_timestamp,idx_commit_report_timestamp) in every schema version that has them (v4 and later), so each batch is an index range scan.JdbcMetricsReportsPurger) inpolaris-relational-jdbc, which owns the DDL of these tables. It lives there rather than in the metrics reports extension, so the admin tool does not depend on the extension (and through it onpolaris-runtime-service). The admin tool now has a compile dependency onpolaris-relational-jdbc(it wasruntimeOnly). The table names are repeated in the purger rather than taken from the extension'sModel*MetricsReportclasses, for the same reason.Relation to other PRs:
runtimeOnly→implementationchange inruntime/admin/build.gradle.ktsand adjacent lines inPolarisAdminTool,admin-tool.mdandCHANGELOG.md, which I'll rebase once one merges. If reviewers prefer a single generic command (for examplepurge-time-series --kind events|scan-metrics|commit-metrics) over one command per table, I'd reshape both PRs together. Once both are in, the batching loop shared by the two purgers could be factored out.Things I'd like feedback on:
purge-events,purge-metrics) vs. one generic command.Related to #2573
Checklist
CHANGELOG.md(if needed)site/content/in-dev/unreleased(if needed)Tests:
JdbcMetricsReportsPurgerTest(H2, parameterized over batch sizes, both tables, realm and table isolation, cutoff boundary) and admin tool Quarkus tests on PostgreSQL and CockroachDB (purge-metricsdeletes exactly the old reports of each realm from both tables; invalid duration), plus NoSQL in-memory (unsupported persistence).