Chained replication jobs conflicting

I have a pretty unusual setup with three servers doing "chained replication" of zfs filesystems.

By "chained replication" I mean the following scenario:
Consider servers A,B and C. The zfs filesystem from A is replicated using zrepl to B using a snap job on A and a pull job on B. The zfs filesystem on B is replicated using zrepl to C using a snap job on B and a pull job on C.

The conflict arises because the pull job on B finds the bookmark created on the filesystem by the pull job running on C. This bookmark obviously cannot be found on A so the job considers the two filesystems diverged and the conflict cannot be resolved.

Jun 7 12:02:36 backup01 zrepl[9348]: [nxtmail01_sync][repl][Uv38$Ny6V$7Bc5$7Bc5.6hqC]: conflict conflict="the receiver's latest snapshot is not present on sender:\n last common: @zrepl_nxtm01-b01_20210606_140303_000\n sender-only:\n receiver-only:\n @zrepl_b01-b03_20210607_090523_000\n" filesystem="mail/vmail"

I have drilled this problem down to the endpoint.go implementation and have come up with a very dirty fix by filtering out the listing of bookmarks in the Receiver's ListFileSystemVersions method. I realize this is not a usable fix for any other use case, it could in fact cause problems, this is the reason I didn't create a PR. I'm attaching a patch.

zrepl_conflict_dirty_fix.patch.txt

The way I see it, the bookmark causing the problems does contain the name of the job that created it. Maybe a good solution would be to somehow process only bookmarks created by the job currently running and disregarding the rest.

I do not understand the inner working of zrepl enough to be able to go further fixing this, but if I could get some pointers on how to proceed further I'm willing to help fix the issue.

The conflict arises because the pull job on B finds the bookmark created on the filesystem by the pull job running on C. This bookmark obviously cannot be found on A so the job considers the two filesystems diverged and the conflict cannot be resolved.

Great analysis!

I have drilled this problem down to the endpoint.go implementation and have come up with a very dirty fix by filtering out the listing of bookmarks in the Receiver's ListFileSystemVersions method. I realize this is not a usable fix for any other use case, it could in fact cause problems, this is the reason I didn't create a PR. I'm attaching a patch.

👍

The way I see it, the bookmark causing the problems does contain the name of the job that created it. Maybe a good solution would be to somehow process only bookmarks created by the job currently running and disregarding the rest.

I think the proper solution is that the replication algorithm shouldn't consider those bookmarks when determining divergence.

I do not understand the inner working of zrepl enough to be able to go further fixing this, but if I could get some pointers on how to proceed further I'm willing to help fix the issue.

The code that computes the incremental path is this function (use VS Code or similar to find its callers)

zrepl/replication/logic/diff/diff.go

Lines 75 to 131 in 3f8fe3a

    
           func IncrementalPath(receiver, sender []*FilesystemVersion) (incPath []*FilesystemVersion, conflict error) { 
        
           	receiver = SortVersionListByCreateTXGThenBookmarkLTSnapshot(receiver) 
        
           	sender = SortVersionListByCreateTXGThenBookmarkLTSnapshot(sender) 
        
           	var mrcaCandidate struct { 
        
           		found bool 
        
           		guid  uint64 
        
           		r, s  int 
        
           	} 
        
           findCandidate: 
        
           	for r := len(receiver) - 1; r >= 0; r-- { 
        
           		for s := len(sender) - 1; s >= 0; s-- { 
        
           			if sender[s].GetGuid() == receiver[r].GetGuid() { 
        
           				mrcaCandidate.guid = sender[s].GetGuid() 
        
           				mrcaCandidate.s = s 
        
           				mrcaCandidate.r = r 
        
           				mrcaCandidate.found = true 
        
           				break findCandidate 
        
           			} 
        
           		} 
        
           	} 
        
           	// handle failure cases 
        
           	if !mrcaCandidate.found { 
        
           		return nil, &ConflictNoCommonAncestor{ 
        
           			SortedSenderVersions:   sender, 
        
           			SortedReceiverVersions: receiver, 
        
           		} 
        
           	} else if mrcaCandidate.r != len(receiver)-1 { 
        
           		return nil, &ConflictDiverged{ 
        
           			SortedSenderVersions:   sender, 
        
           			SortedReceiverVersions: receiver, 
        
           			CommonAncestor:         sender[mrcaCandidate.s], 
        
           			SenderOnly:             sender[mrcaCandidate.s+1:], 
        
           			ReceiverOnly:           receiver[mrcaCandidate.r+1:], 
        
           		} 
        
           	} 
        
           	// incPath is possible 
        
           	// incPath must not contain bookmarks except initial one, 
        
           	incPath = make([]*FilesystemVersion, 0, len(sender)) 
        
           	incPath = append(incPath, sender[mrcaCandidate.s]) 
        
           	// it's ok if incPath[0] is a bookmark, but not the subsequent ones in the incPath 
        
           	for i := mrcaCandidate.s + 1; i < len(sender); i++ { 
        
           		if sender[i].Type == FilesystemVersion_Snapshot && incPath[len(incPath)-1].Guid != sender[i].Guid { 
        
           			incPath = append(incPath, sender[i]) 
        
           		} 
        
           	} 
        
           	if len(incPath) == 1 { 
        
           		// nothing to do 
        
           		incPath = incPath[1:] 
        
           	} 
        
           	return incPath, nil 
        
           }

We have some unit tests for the diff algorithm in replication/logic/diff/diff_test.go.

Feel free to start hacking on this. I won't have too much time for at least another week.
Please start a WIP PR early so that you get notified by CI.

I'm considering adopting zrepl to replace my current solution (which is a combination of a hacked version of zxfer and a hacked version of zfSnap), since this is more modern, maintained, and will allow me to use ZFS-native encryption in the way that I want to do so.

However, I'm in this exact scenario: multiple source machines (laptops and servers) which replicate to a primary backup server; that machine then replicates to removable storage disks and a secondary backup server at a different physical location. For bonus points, the secondary server (C, in this situation) also hosts some of its own content, and replicates that content the other direction to B (but not to any of the A machines), and B replicates it onto the removable storage disks.

I'm more than willing to help out here to try to resolve this bug, although I've only written a small amount of Go code I can read it reasonably well. Is the proposed solution in this issue still the preferred way to approach it?

I'm more than willing to help out here to try to resolve this bug, although I've only written a small amount of Go code I can read it reasonably well. Is the proposed solution in this issue still the preferred way to approach it?

Yes, it is.

Excellent. I won't have my systems fully setup with zrepl until this weekend, but at that point I should be able to replicate this problem and then begin work on solving it.

Ok. For dev setup, I'd recommend two Vagrant boxes, or two local VMs. Don't mess with your prod system!

Filter bookmarks when searching filesystem versions.

I've published my fix for this issue, please feel free to test and/or review.

For what it's worth, my fleet of machines don't suffer from this problem.

I've got:

(A1 .. A7) -> B -> (C & D1..D3)

and for completeness I've also got:

C -> B -> D1..D3

where:

A1..A7 are 'client' machines (snap and push jobs only)
B is my NAS, which has snap/push for sending to C and D, and sink for receiving from A1..A7 and C
C is my remote server, which has snap/push for sending to B and sink for receiving from B
D1..D3 is a set of removable hard drives which have 'local' push/sink jobs on B

The jobs that move content between B and C cover distinct (non-overlapping) sets of filesystems.

All of this is working fine, with no cursor/bookmark conflicts at all.

replication: ignore bookmarks when computing incremental path

@kpfleming

@kpfleming how do you handle snapshot pruning on the A>B>C chain?

Oh, 0.6.0 released today. What timing. Maybe that'll fix my problems.

Navigation Menu

Repository navigation

Chained replication jobs conflicting #490

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	func IncrementalPath(receiver, sender []FilesystemVersion) (incPath []FilesystemVersion, conflict error) {

	receiver = SortVersionListByCreateTXGThenBookmarkLTSnapshot(receiver)
	sender = SortVersionListByCreateTXGThenBookmarkLTSnapshot(sender)

	var mrcaCandidate struct {
	found bool
	guid uint64
	r, s int
	}

	findCandidate:
	for r := len(receiver) - 1; r >= 0; r-- {
	for s := len(sender) - 1; s >= 0; s-- {
	if sender[s].GetGuid() == receiver[r].GetGuid() {
	mrcaCandidate.guid = sender[s].GetGuid()
	mrcaCandidate.s = s
	mrcaCandidate.r = r
	mrcaCandidate.found = true
	break findCandidate
	}
	}
	}

	// handle failure cases
	if !mrcaCandidate.found {
	return nil, &ConflictNoCommonAncestor{
	SortedSenderVersions: sender,
	SortedReceiverVersions: receiver,
	}
	} else if mrcaCandidate.r != len(receiver)-1 {
	return nil, &ConflictDiverged{
	SortedSenderVersions: sender,
	SortedReceiverVersions: receiver,
	CommonAncestor: sender[mrcaCandidate.s],
	SenderOnly: sender[mrcaCandidate.s+1:],
	ReceiverOnly: receiver[mrcaCandidate.r+1:],
	}
	}

	// incPath is possible

	// incPath must not contain bookmarks except initial one,
	incPath = make([]*FilesystemVersion, 0, len(sender))
	incPath = append(incPath, sender[mrcaCandidate.s])
	// it's ok if incPath[0] is a bookmark, but not the subsequent ones in the incPath
	for i := mrcaCandidate.s + 1; i < len(sender); i++ {
	if sender[i].Type == FilesystemVersion_Snapshot && incPath[len(incPath)-1].Guid != sender[i].Guid {
	incPath = append(incPath, sender[i])
	}
	}
	if len(incPath) == 1 {
	// nothing to do
	incPath = incPath[1:]
	}
	return incPath, nil
	}

Chained replication jobs conflicting #490

Description

Activity

problame commented on Jun 15, 2021

kpfleming commented on Jun 22, 2022

problame commented on Jun 27, 2022

kpfleming commented on Jun 29, 2022

problame commented on Jun 30, 2022

kresike commented on Jul 18, 2022

kpfleming commented on Jul 22, 2022

lpulley commented on Nov 1, 2022

lpulley commented on Nov 1, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions