fix(backup): only list and restore backups complete on their tier - #495

An incomplete backup was reported as a NoBackupFoundError carrying the
missing parts. Report it as an IncompleteBackupError instead, which
unwraps to NoBackupFoundError so MultiConnection still falls back to
tier2.

Without tier2, MultiConnection now returns the tier1 error rather than a
fresh not-found one, so the missing parts reach the user. It also no
longer shadows the tier1 error, which made a tier1 failure read as
`backup "" for host "" not found`.

Assisted-By: Claude
Signed-off-by: Francesco Canovai <francesco.canovai@enterprisedb.com>

ListBackupsByCompleteness already returns the same backups, split into
complete and incomplete ones. WAL retention now joins the two lists
instead of using a dedicated listing method.

Assisted-By: Claude
Signed-off-by: Francesco Canovai <francesco.canovai@enterprisedb.com>

The listBackups function took a flag to return only the complete backups and, in
that mode, to skip reading the metadata of backups missing their pgdata
or control data. It now always returns both lists, and ListBackups drops
the incomplete one. This costs one extra metadata read per incomplete
backup, which is rare.

Assisted-By: Claude
Signed-off-by: Francesco Canovai <francesco.canovai@enterprisedb.com>

The backup metrics now walk each cluster's metadata snapshots in end
time order to find the latest and oldest complete backup, so the latest
and oldest IDs tracked per group were no longer read. Group the
snapshots by source, sorted by end time, and count them with len.

Assisted-By: Claude
Signed-off-by: Francesco Canovai <francesco.canovai@enterprisedb.com>

The backups metric checked only the pgdata and control data of each
backup, because the tablespaces are listed in the metadata and only the
latest and oldest metadata snapshots were restored. A backup missing a
tablespace was counted, while ListBackups does not list it.

Restore the metadata of every backup and check it like ListBackups does.
The collector keeps the metadata by snapshot ID, as a snapshot never
changes, so each backup is restored once rather than on every pass. A
backup whose metadata cannot be read is skipped and retried on the next
pass, instead of dropping the series of its whole cluster.

Assisted-By: Claude
Signed-off-by: Francesco Canovai <francesco.canovai@enterprisedb.com>

The upload tags a tablespace snapshot with the base name of the
tablespace, and the completeness check expects the same, but the
restore looked it up with the full name. A tablespace whose name
contains a slash could then be listed as complete but not restored.
Compute the tag value with one helper in all three places.

Assisted-By: Claude
Signed-off-by: Francesco Canovai <francesco.canovai@enterprisedb.com>

Assisted-By: Claude
Signed-off-by: Francesco Canovai <francesco.canovai@enterprisedb.com>

After deleting the pgdata snapshot, check that the metadata snapshot is
still there on both tiers. The backup is then incomplete rather than
gone, and the final check that retention deleted its metadata cannot
pass for a lookup that never matched.

Assisted-By: Claude
Signed-off-by: Francesco Canovai <francesco.canovai@enterprisedb.com>

The metrics collector had its own copy of the metadata reader and of the
interface it needs. Export them from the Kopia backup client and reuse
them.

Assisted-By: Claude
Signed-off-by: Francesco Canovai <francesco.canovai@enterprisedb.com>

Assisted-By: Claude
Signed-off-by: Francesco Canovai <francesco.canovai@enterprisedb.com>

The retention sweeper deleted backup attempts that never produced a
metadata snapshot, once a later backup of the same cluster completed.
It now leaves every incomplete backup in place and reports it, with a
status telling whether it can still complete: PENDING (the latest
tier1 attempt, possibly still uploading), RECOVERABLE (on tier2,
complete on tier1, rebuilt by the next relay) or UNRECOVERABLE (to be
deleted by hand).

The incomplete backups are logged at each sweep, exported as the
klio.server.backup.incomplete_backups gauge, shown on a new Grafana
dashboard panel, and listed by `klio admin list-backups --incomplete`.

Assisted-by: Claude
Signed-off-by: Francesco Canovai <francesco.canovai@enterprisedb.com>

Manifest.StartTime is a UTCTimestamp, but the backup grouping code
still treated it as a string, so the package did not compile. Keep
the start time as a UTCTimestamp while grouping and format it as
RFC3339 for the reported StartedAt.

Assisted-By: Claude
Signed-off-by: Francesco Canovai <francesco.canovai@enterprisedb.com>

The wait for no incomplete backups compared the empty list the server
returns with a nil slice using reflect.DeepEqual, which tells them
apart, so it could never succeed. Treat two empty lists as equal.

Assisted-By: Claude
Signed-off-by: Francesco Canovai <francesco.canovai@enterprisedb.com>

GetMetadata restored Kopia checkpoints of an interrupted metadata upload as
if they were finished, while the backup scan already ignored them. Share
one definition of a finished metadata snapshot between the two.

Assisted-by: Claude
Signed-off-by: Gabriele Quaresima <gabriele.quaresima@enterprisedb.com>

Assisted-by: Claude
Signed-off-by: Gabriele Quaresima <gabriele.quaresima@enterprisedb.com>

Retention no longer deletes incomplete backups. Fix the retention section
of the plugin configuration page and add an upgrade note.

Assisted-by: Claude
Signed-off-by: Gabriele Quaresima <gabriele.quaresima@enterprisedb.com>

Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>