Repository navigation
postgres backup: BackupJob re-applies its ObjectStore every 5 seconds, so concurrent jobs overwrite each other and a failed job leaves its destination #5202
Description
Describe the bug
reconcileCNPG calls applyClusterPluginBackup on every pass (every 5 seconds while a BackupJob is Running), and it server-side applies the strategy ObjectStore and the Cluster spec.plugins with ForceOwnership (internal/backupcontroller/cnpgstrategy_controller.go, lines 604 to 666). Two things follow.
Two BackupJobs for the same Postgres app with different strategies overwrite each other. A custom BackupClass next to cozy-default is the normal way to give a tenant its own destination, so no broken strategy is needed to hit it. And after a BackupJob with a wrong destination fails, the Cluster keeps that destination until another BackupJob applies the good one.
Second thing, every error from applyClusterPluginBackup other than NotFound goes straight to markBackupJobFailed (line 345), so a webhook or apiserver blip fails the BackupJob after seconds, while a permanent backup failure waits 30 minutes.
Environment
Cozystack v1.7.0 (backupstrategy-controller v1.7.0, CloudNativePG 1.30.0, plugin-barman-cloud v0.13.0). The controller code is the same on main (4087218).
To reproduce
Overwrite: copy cozy-default-cnpg with a destination bucket that does not exist, bind it from a second BackupClass, start a BackupJob with that class and a BackupJob with cozy-default for the same Postgres app, and sample .spec.configuration.destinationPath of the live ObjectStore.
Transient error: start a BackupJob while the CloudNativePG operator pod restarts.
Actual behaviour
Overwrite: live destinationPath flipped between the two destinations (40 samples, 14 broken and 26 healthy), and the cnpg.io/Backup of the healthy BackupJob failed, it ended Failed at the 30 minute mark. After the broken BackupJobs failed the ObjectStore kept the missing bucket for 38 minutes, ContinuousArchiving=False and WAL piled up on the PVC (barman-cloud-wal-archive exit status 4), until a healthy BackupJob re-applied the good destination.
Transient error: BackupJob went Failed after 41 seconds with failed to attach barman-cloud plugin to Cluster: Internal error occurred: failed calling webhook "mcluster.cnpg.io": ... dial tcp ...:443: connect: connection refused. The operator pod was restarting at that moment.
Expected behaviour
ObjectStore is applied once when the cnpg.io/Backup is created, or only when no other BackupJob of the app is running, and a failed job does not leave its destination on the live Cluster. A transient apply error is retried and does not fail the BackupJob at once.
The 30 minute wait on a failed Backup is a separate problem, see the issue about BackupJob staying Running.
Activity
dosubot commentedon Nov 4, 2026
There is no chart-driven recovery for the reported
useSystemBucket: truecase: the chart deliberately omits bothspec.pluginsand the backup ObjectStore, leaving the driver's configuration intact across Helm/Flux reconciliations. The ObjectStore is also owned by the Cluster, not the BackupJob, so job deletion does not garbage-collect it. That explains why waiting for chart reconciliation would not restore the healthy destination. [1] [2]The transient-error case can leave the same residue:
applyClusterPluginBackuppatches the ObjectStore before patching the Cluster. If the Cluster webhook fails at that second step, the destination has already changed, and the caller marks the BackupJob failed without rolling it back. [3] [4]Related: #4559 tracks the backup-versus-restore version of this shared-state problem. Its September 29 report describes a running CNPG backup re-pointing an in-place restored Cluster and proposes an atomic per-application claim shared by both controllers. That is relevant context for any serialization approach here, rather than checking only for another running BackupJob. [5]
To reply, just mention @dosu.
Docs are dead. Just use Dosu.