Receive side pruning appears to mismatch jobs

I am using FreeBSD 12 with zrepl 0.1.0rc3. I have a config file with 3 jobs. The first job is called "nextcloud" and backs up a dataset called "nextcloud". The third job is called "documents" and backs up a number of things including a dataset called "development". The receive side pruner fails because it appears to get these mixed up. I am using ssh+stdinserver transport, with a Ubuntu machine on the other side, same zrepl 0.1.0rc3.

Mar 12 07:06:23 fileserver zrepl[38476]: 2019-03-12T07:06:23-04:00 [ERRO][nextcloud][pruning]: cannot get replication cursor fs="zfspool/development" invocation="1" err="endpoint does not allow access to filesystem zfspool/development" prune_side="receiver"

    Type: push                                                                                                                                                                                
    Replication:                                                                                                                                                                              
        Status: Completed                                                                                                                                                                     
        Progress: [===========================>] 0 B / 0 B @ 0 B/s                                                                                                     
          zfspool/development Completed (step 0/0, 0 B/0 B)                                                                                                                                   
          zfspool/shareddocs  Completed (step 0/0, 0 B/0 B)                                                                                                                                   
    Pruning Sender:                                                                                                                                                                           
        Status: Done                                                                                                                                                                          
        Progress: [>] 0/0 snapshots                                                                                                                                                           
        zfspool/development Completed  (destroy 0 of 44 snapshots)                                                                                                                            
        zfspool/shareddocs  Completed  (destroy 0 of 149 snapshots)                                                                                                                           
    Pruning Receiver:                                                                                                                                                                         
        Status: ErrPerm                                                                                                                                                                       
        Error: endpoint does not allow access to filesystem zfspool/nextcloud                                                                                                                 
                                                                                                            
Job: nextcloud                                                                                                                                                                                
    Type: push                                                                                                                                                                                
    Replication:                                                                                                                                                                              
        Status: Completed                                                                                                                                                                     
        Progress: [============================>] 624 B / 624 B @ 0 B/s                                                                                                 
          zfspool/nextcloud Completed (step 1/1, 624 B/624 B)                                                                                                                                 
    Pruning Sender:                                                                                                                                                                           
        Status: Done                                                                                                                                                                          
        Progress: [>] 0/0 snapshots                                                                                                                                                           
        zfspool/nextcloud Completed  (destroy 0 of 4 snapshots)                                                                                                                               
    Pruning Receiver:                                                                                                                                                                         
        Status: ErrPerm                                                                                                                                                                       
        Error: endpoint does not allow access to filesystem zfspool/development```

Could you post the zrepl.yml for both the sending and receiving side?
Note that the line prefix contains the job name, not the dataset name: [LOG_LEVEL][JOB_NAME][SUBSYSTEM]. In zrepl 0.0.*, the line prefix would also contain the dataset name. This will be gone in 0.1.

Could that be another instance of the filesystem moving/renaming/zrepl:placeholder problem mentioned in #126 ?

global:
  logging:
    - type: "stdout"
      level:  "error"
      format: "human"
    - type: "syslog"
      level:  "debug"
      format: "logfmt"

jobs:
 - type: push
   name: "nextcloud"
   connect:
      type: ssh+stdinserver
      host: backup
      user: root
      port: 22
      rpc:
         timeout: 60s
      identity_file: /root/.ssh/backup.pem
   filesystems: {
      "zfspool/nextcloud": true
   }
   snapshotting:
      type: periodic
      prefix: zrepl_
      interval: 4h
   pruning:
      keep_sender:
       - type: not_replicated
       - type: grid
         grid: 1x1d(keep=all) | 7x1d | 4x7d | 999999x180d
         regex: "^zrepl_.*"
      keep_receiver:
       - type: grid
         grid: 1x1d(keep=all) | 7x1d | 4x7d | 999999x180d
         regex: "^zrepl_.*"

 - type: push
   name: "machines"
   connect:
      type: ssh+stdinserver
      host: backup
      user: root
      port: 22
      identity_file: /root/.ssh/backup.pem
      rpc:
         timeout: 60s
   filesystems: {
      "zfspool/proxmox": true,
      "zfspool/oldmachines": true,
      "zfspool/vms": true
   }
   snapshotting:
      type: periodic
      prefix: zrepl_
      interval: 24h
   pruning:
      keep_sender:
       - type: not_replicated
       - type: grid
         grid: 5x1d | 3x30d | 999999x360d
         regex: "^zrepl_.*"
      keep_receiver:
       - type: grid
         grid: 5x1d | 3x30d | 999999x360d
         regex: "^zrepl_.*"

 - type: push
   name: "documents"
   connect:
      type: ssh+stdinserver
      host: backup
      user: root
      port: 22
      identity_file: /root/.ssh/backup.pem
      rpc:
         timeout: 60s
   filesystems: {
      "zfspool/development": true,
      "zfspool/shareddocs": true
   }
   snapshotting:
      type: periodic
      prefix: zrepl_
      interval: 1h
   pruning:
      keep_sender:
       - type: not_replicated
       - type: regex
         regex: "^auto-.*"
       - type: grid
         grid: 1x8h(keep=all) | 30x1d | 999999x7d
         regex: "^zrepl_.*"
      keep_receiver:
       - type: regex
         regex: "^auto-*"
       - type: grid
         grid: 1x8h(keep=all) | 30x1d | 999999x7d
         regex: "^zrepl_.*"

jobs:
 - type: sink
   name: sinkjob
   root_fs: zfsbackup
   serve:
      type: stdinserver
      client_identities:
       - "fileserver"

It gets mixed up because all jobs push into the same sink and with the same identity. You need to give each push job a different client identity or define 3 different sinks.

Do the names of the sink jobs need to correspond to the push jobs?

No.

I defined three separate sink jobs and it still gives the same error.

Please post the sink config again.

 - type: sink
   name: nextcloud
   root_fs: zfsbackup
   serve:
      type: stdinserver
      client_identities:
       - "fileserver"
 - type: sink
   name: machines
   root_fs: zfsbackup
   serve:
      type: stdinserver
      client_identities:
       - "fileserver"
 - type: sink
   name: documents
   root_fs: zfsbackup
   serve:
      type: stdinserver
      client_identities:
       - "fileserver"

Yeah, that won't help if the client identities are still the same: see https://zrepl.github.io/configuration/jobs.html#how-the-passive-side-works

I would recommend to give separate client identities on the source per job.

The alternative is to have different root_fs per sink job

Could the "separate client identities" or the "multiple sinks with different root_fs" solve your issue?
Where would you have expected to see this in the documentation?

I did not try either because I don't want to set up multiple client identities (which would require multiple ssh keys for the same computer) or use a different root fs. I would recommend making the setup I tried possible. Other than that, this is a good piece of software.

Specifically where in the documentation I am not sure but it seemed logical to me that the sink would not need more than one job as the job itself looked like it was dataset-agnostic and wouldn't be subject to concurrency issues.

I would recommend making the setup I tried possible.

Given that the push jobs pushed disjoint filesystems, you are right in that your original setup should just work.

Would you mind trying out the improvements to the pruner made in #139 ?
Binary builds can be retrieved from CircleCI https://circleci.com/gh/zrepl/zrepl/107#artifacts/containers/0 /go/src/github.com/zrepl/zrepl/artifacts/release.

You might encounter one-time race conditions in placeholder filesystem creation during initial replication.
But the pruners should no longer interfere on the receiving side: the zrepl status output should contain something like filesystem has no correspondence on sender in each push job's prune_receiver report.

Update: I replicated your setup on my test installation using the CircleCI build and confirmed the following:

You might encounter one-time race conditions in placeholder filesystem creation during initial replication.
But the pruners should no longer interfere on the receiving side: the zrepl status output should contain something like filesystem has no correspondence on sender in each push job's prune_receiver report.

I'll try to figure out a solution to the placeholder races (which do indeed happen on the first replication), but as long as all the filesystems are disjoint, it works afterwards.

#126 (comment) reports that a very similar problem has been fixed in current master . Closing this issue until further notice.

Navigation Menu

Repository navigation

Receive side pruning appears to mismatch jobs #135

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Receive side pruning appears to mismatch jobs #135

Description

Activity

problame commented on Mar 12, 2019

jbliesener commented on Mar 12, 2019

stingray-11 commented on Mar 12, 2019

stingray-11 commented on Mar 12, 2019

problame commented on Mar 13, 2019

stingray-11 commented on Mar 13, 2019

problame commented on Mar 13, 2019

stingray-11 commented on Mar 13, 2019

problame commented on Mar 13, 2019

stingray-11 commented on Mar 13, 2019

problame commented on Mar 14, 2019

problame commented on Mar 18, 2019

stingray-11 commented on Mar 19, 2019

problame commented on Mar 20, 2019

problame commented on Mar 20, 2019

problame commented on Mar 21, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions