pruning: grid: fix in 0.3.1 has unexpected destructive consequences for multi job setups #400

Activity

1. problame commented on Nov 30, 2020

   

   problame

   on Nov 30, 2020

   Member

   More actions

   And that seemingly innocent "[bug]" tag changelog managed to clear up my entire snapshots upside_down_face

   First of all, I'm sorry for the trouble :/
   The issue in question was #292 , where multiple grid rules in the same keep section caused no snapshots to be ever pruned at all.

   The config you posted above should probably be rejected by zrepl because the filesystems overlap. See https://zrepl.github.io/configuration/overview.html#multiple-jobs-more-than-2-machines :

   As a general rule, multiple jobs configured on one machine must operate on disjoint sets of filesystems. Otherwise, concurrently running jobs might interfere when operating on the same filesystem.

   That being said, I see your use case of different 'trains', 'tiers', or 'frequencies' that are pruned independently.
   Such a feature has been in the back of my mind for quite a while but I was never able to figure out a proper UX for it.
   How would the ideal config syntax for your use case look like?
2. problame

   added

   docs

   usability

   on Nov 30, 2020
3. problame

   added this to the 0.3.2This milestone has been deleted milestone on Nov 30, 2020
4. problame

   changed the title [-]A more prominent documentation for prune change in 0.3.1[/-] [+]pruning: grid: fix in 0.3.1 has unexpected destructive consequences for multi job setups[/+] on Nov 30, 2020
5. nanaya commented on Nov 30, 2020

   

   nanaya

   on Nov 30, 2020

   Last edited by nanaya

   Author

   More actions

   How would the ideal config syntax for your use case look like?

   I'm not sure about overall job syntax but for pruning itself maybe a flag (type?) to keep snapshots not matched by any other prune policies? (- type: no_match?)

   Or simple pruning syntax in snapshotting rules itself? Something like this maybe (not sure if makes sense in other contexts):

   - name: snap_hourly
     type: snap
     filesystems: {
       "<": true,
     }
     snapshotting:
       type: periodic
       prefix: zrepl_hourly_
       interval: 10m
       keep:
         type: grid
         grid: 1x1h(keep=all) | 72x1h

   The prefix option and timestamp format in snapshotting should already provide enough hints to construct the full prune rule.

   I mainly find the regex option in pruning policy weird because I kind of expect the section to be for pruning the snapshots generated in the same job so it shouldn't be needed (but instead "pruning" and "snapshotting" are actually two completely separate thing).
6. problame commented on Dec 5, 2020

   

   problame

   on Dec 5, 2020

   Member

   More actions

   but instead "pruning" and "snapshotting" are actually two completely separate thing

   Yeah, replication, pruning and snaphotting are only very loosely coupled in the code base.
   There is no concept of 'ownership' of snapshots, i.e., even if a job creates snapshots with a certain prefix, it doesn't assume anything beyond that.

   I'm not sure about overall job syntax but for pruning itself maybe a flag (type?) to keep snapshots not matched by any other prune policies? (- type: no_match?)

   The idea behind the pruning system is that you define 'keep' rules, i.e., you specify what snapshots should be kept and all other snapshots are destroyed. If I understand correctly, what you are requesting is a kind of flip: keep everything by default, only destroy those snapshots with matching prune rules.
7. problame

   modified the milestones: 0.3.2This milestone has been deleted, 0.4 on Feb 20, 2021
8. problame

   added a commit that references this issue on Mar 14, 2021

   docs: 0.4 changelog: mention the 0.3.1 fix to prune:grid

   ...

   This issue will close once commit 299808a is merged into the 'master' branch.

   299808a
9. problame

   closed this as completedin 299808aon Mar 14, 2021