Repository navigation
backups: changing publishing.host leaves the old S3 endpoint in the COSI secret, the Velero BSL and the default Strategies #5205
Description
Describe the bug
After publishing.host is changed, backups keep the old S3 endpoint. The platform builds the endpoint as https://s3.<publishing.host>, and two copies of it never refresh.
First, the COSI Secret bucket-cozy-backups-system (BucketInfo) keeps secretS3.endpoint from the moment the BucketAccess was created. bucket-cozy-backups-system-credentials and the projected cozy-backups-creds.endpoint are derived from it (packages/system/bucket/templates/user-credentials.yaml, line 53).
Second, the Velero BackupStorageLocation and every default Strategy CR bake the endpoint at the render of the backupstrategy-controller release and keep it until that release is forced again.
Nothing re-renders them. The default objects gate acts only when an object is missing.
Environment
Cozystack v1.7.0, variant isp-full, platform Package with publishing.host changed from example.org to a real domain after install.
To reproduce
- Install with
publishing.host: example.org, enableseaweedfsandingresson the root Tenant, wait for the bucketcozy-backups. - Change
publishing.hostin theplatformcomponent values of the Package to a real domain and wait for everything to settle (certificate fors3.<new host>is issued). kubectl -n cozy-velero get bsl cozy-default -o jsonpath='{.spec.config.s3Url}'andkubectl get cnpgs.strategy.backups.cozystack.io cozy-default-cnpg -o jsonpath='{.spec.template.barmanObjectStore.endpointURL}'.
Actual behaviour
Both still show https://s3.example.org, the BSL stays Unavailable (lookup s3.example.org ... no such host) and the BucketInfo and cozy-backups-creds carry the old host, so every driver gets the old endpoint.
Recovery that worked: delete the BucketAccess, force bucket-cozy-backups (it owns the BucketAccess, forcing only bucket-cozy-backups-system removes the credentials Secret), then force bucket-cozy-backups-system, then force backupstrategy-controller (reconcile.fluxcd.io/forceAt and requestedAt). After that BSL went Available and the strategies had the new endpoint.
Expected behaviour
Endpoint follows the host, or the docs list what to force after a host change.
Activity
dosubot commentedon Nov 4, 2026
This is a separate gap from the missing-object recovery in #3247/#3524: that fix forces renders when credentials or default objects are absent, not when their endpoint is stale. It also means
cozystack_backup_default_objects_missingcan stay at zero during this failure, so that metric alone cannot detect it. [1] [2] [3]One extra recovery caveat for ClickHouse: with
backup.useSystemBucket: true, the sidecar reads the projected endpoint and keys through environment variables. After refreshingcozy-backups-creds, existing ClickHouse pods may still need a restart to pick them up. #3986 reports the same stale-environment behavior on v1.7.0, including after credential rotation, so a host-change runbook should include that check rather than stop at BSL availability. [4] [5]To reply, just mention @dosu.
Share context across your team and agents. Try Dosu.
fix(backups): re-render the default backup objects when the S3 endpoi…
fix(backups): re-render the default backup objects when the S3 endpoi…