Visitar URL original
HBASE-30407 Add persistence support for cache components by VladRodionov · Pull Request #8739 · apache/hbase · GitHub
Skip to content

HBASE-30407 Add persistence support for cache components - #8739

Open
VladRodionov wants to merge 2 commits into
apache:HBASE-30018from
VladRodionov:HBASE-30407-cache-component-persistence
Open

VladRodionov wants to merge 2 commits into
apache:HBASE-30018from
VladRodionov:HBASE-30407-cache-component-persistence

Conversation

@VladRodionov

Copy link
Copy Markdown
Contributor

Summary

Adds a persistence framework for stateful components in the pluggable HBase block cache
architecture.

Persistence is optional and component-local. A cache component participates by implementing
PersistentCacheComponent; components that do not support persistence are unaffected.

The change also introduces persistence orchestration and a Hadoop FileSystem-backed storage
implementation so persistent topology, policy, and cache-engine state can be saved and restored
independently.

Changes

  • Add PersistentCacheComponent for cache components that support persistence.
    • getPersistenceId() provides a stable identifier for the component persistence format.
    • save(OutputStream) saves component-owned runtime state.
    • restore(InputStream) restores state into an already constructed component.
  • Persistence remains optional:
    • CacheEngine, CacheTopology, and CachePlacementAdmissionPolicy do not automatically
      implement persistence.
    • Concrete implementations opt in by implementing PersistentCacheComponent.
  • Add persistence infrastructure under the cache persistence package:
    • CachePersistenceStorage
    • CachePersistenceOutput
    • CachePersistenceCoordinator
    • HadoopFsCachePersistenceStorage
  • Add persistence orchestration through CacheAccessService.
  • TopologyBackedCacheAccessService delegates persistence to
    CachePersistenceCoordinator.
  • The coordinator discovers and persists topology, placement/admission policy, and individual
    cache engines independently.
  • Persistence storage keys include both component role and persistence identifier, for example:
    • topology/<id>
    • policy/<id>
    • engine/l1/<id>
    • engine/l2/<id>
  • Add a Hadoop FileSystem storage implementation supporting local FS, HDFS, and other Hadoop
    filesystem implementations.
  • Hadoop FS writes are staged through temporary files and published only after an explicit
    successful commit.
  • Failed or abandoned writes are aborted without replacing previously committed state.
  • Add unit tests covering persistence coordination, save/restore behavior, missing state,
    persistent and non-persistent components, staged publication, abort behavior, and preservation
    of previously committed state after save failures.

Persistence Model

Cache components are constructed normally from the current HBase configuration before persisted
state is restored.

Persistence therefore restores runtime state into existing objects; it does not construct or
replace cache components.

Current configuration remains authoritative. Restored state must be accepted subject to the
current component configuration and runtime constraints.

Persistence is local to each component. For example, a persistent topology saves only
topology-owned state and does not recursively persist its cache engines. Engines and
placement/admission policies are persisted independently by the coordinator.

Stream and Storage Ownership

PersistentCacheComponent operates only on caller-provided streams and must not close them.

The persistence coordinator owns the persistence streams and is responsible for closing them.

For writes, CachePersistenceOutput provides explicit commit/abort semantics:

component.save(...)
      |
      v
temporary state
      |
      +-- success -> commit -> publish
      |
      `-- failure -> abort  -> discard temporary state

This prevents a partial component save from replacing previously valid persisted state.

HadoopFsCachePersistenceStorage uses Hadoop FileSystem, keeping the component persistence API
independent of local files, HDFS, object stores, or other Hadoop-compatible storage systems.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Checkpoint recovery, commit failure reporting, and storage-key validation need correction before approval.

Review effort: Balanced
Findings: 2 High severity · 1 Medium severity

Open (3)
What changed in this PR

Adds optional, component-local persistence to HBase’s pluggable block cache, restoring runtime state into existing configured components.

Changes:

  • Defines persistence contracts and coordinates topology, policy, and engine state.
  • Adds Hadoop filesystem storage with staged writes and commit/abort handling.
  • Tests service delegation, persistence coordination, and storage behavior.
File Description
hbase-server/​src/​test/​java/​org/​apache/​hadoop/​hbase/​io/​hfile/​cache/​TestTopologyBackedCacheAccessService.java Tests persistence delegation.
hbase-server/​src/​test/​java/​org/​apache/​hadoop/​hbase/​io/​hfile/​cache/​persistence/​TestHadoopFsCachePersistenceStorage.java Tests publication, aborts, and state preservation.
hbase-server/​src/​test/​java/​org/​apache/​hadoop/​hbase/​io/​hfile/​cache/​persistence/​TestCachePersistenceCoordinator.java Tests component discovery and save/restore coordination.
hbase-server/​src/​main/​java/​org/​apache/​hadoop/​hbase/​io/​hfile/​cache/​TopologyBackedCacheAccessService.java Delegates persistence to the coordinator.
hbase-server/​src/​main/​java/​org/​apache/​hadoop/​hbase/​io/​hfile/​cache/​PersistentCacheComponent.java Defines opt-in component persistence.
hbase-server/​src/​main/​java/​org/​apache/​hadoop/​hbase/​io/​hfile/​cache/​persistence/​HadoopFsCachePersistenceStorage.java Implements filesystem-backed staged persistence.
hbase-server/​src/​main/​java/​org/​apache/​hadoop/​hbase/​io/​hfile/​cache/​persistence/​CachePersistenceStorage.java Defines the storage abstraction.
hbase-server/​src/​main/​java/​org/​apache/​hadoop/​hbase/​io/​hfile/​cache/​persistence/​CachePersistenceOutput.java Defines write commit/abort semantics.
hbase-server/​src/​main/​java/​org/​apache/​hadoop/​hbase/​io/​hfile/​cache/​persistence/​CachePersistenceCoordinator.java Saves and restores components independently.
hbase-server/​src/​main/​java/​org/​apache/​hadoop/​hbase/​io/​hfile/​cache/​CacheAccessService.java Adds default persistence hooks.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@VladRodionov
VladRodionov force-pushed the HBASE-30407-cache-component-persistence branch from 3053298 to a9037e7 Compare October 7, 2026 01:01
@VladRodionov

Copy link
Copy Markdown
Contributor Author

I addressed the review comments. The failed job is not related to this PR. cc: @taklwu

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants