Visitar URL original
address new rpc in package replication & pruner by problame · Pull Request #139 · zrepl/zrepl · GitHub
Skip to content

address new rpc in package replication & pruner - #139

Merged
problame merged 24 commits into
masterfrom
problame/replication_refactor
Mar 20, 2019
Merged

address new rpc in package replication & pruner#139
problame merged 24 commits into
masterfrom
problame/replication_refactor

Conversation

@problame

@problame problame commented Mar 13, 2019 •
edited
Loading

Copy link
Copy Markdown
Member

This refactoring is mandated and enabled by #111
Development happened in #125

Overview

Todo

@problame
problame force-pushed the problame/replication_refactor branch from 4bded91 to c87759a Compare March 13, 2019 14:02
@problame problame mentioned this pull request Mar 13, 2019
Closed
problame added 7 commits March 13, 2019 16:39
@problame
zfs + replication: migrate dead zfs/diff_test.go to replication/logic…
8129ed9 
…/diff

(and remove the dead code from package zfs)
@problame
replication: add diff test case
1eb0f12
@problame
handle changes to placeholder state correctly
d50e553 
We assumed that `zfs recv -F FS` would basically replace FS inplace, leaving its children untouched.
That is in fact not the case, it only works if `zfs send -R` is set, which we don't do.

Thus, implement the required functionality manually.

This solves a `zfs recv` error that would occur when a filesystem previously created as placeholder on the receiving side becomes a non-placeholder filesystem (likely due to config change on the sending side):

  zfs send pool1/foo@1 | zfs recv -F pool1/bar
  cannot receive new filesystem stream:
  destination has snapshots (eg. pool1/bar)
  must destroy them to overwrite it
@problame
client/status: fix wrap for multiline strings with leading space
d5250bb
@problame
replication: more elaborate messages for Conflict errors
edcd258
@problame
rpc/ctrl: nicer perr info debug log messages
b85ec52
@problame
pruner: skip placeholders + FSes without correspondents on source
d78d20e 
fixes #126
@problame problame changed the title replication refactoring address new rpc in package replication & pruner Mar 13, 2019
@problame
pruner: remove retry handling + fix early give-up
7584c66 
Retry handling is broken since the gRPC changes (wrong error classification).
Will come back at some point, hopefully by merging the replication
driver retry infrastructure.

However, the simpler architecture allows an easy fix for the problem
that the pruner practically gave up on the first error it encountered.

fixes #123
@problame
problame force-pushed the problame/replication_refactor branch from bf61c5d to 7584c66 Compare March 13, 2019 20:04
@problame

problame commented Mar 13, 2019 •
edited
Loading

Copy link
Copy Markdown
Member Author

@jbliesener would you mind trying this branch against the problems described in this comment?

Same question to @JMoVS with regards to #126

Build & GPG sigs: https://nextcloud.cschwarz.com/s/KYCxdSnbYHJCrnR
(Or use the Circle CI build artifacts (click the status indicator next to the commit, then pick a go version and navigate to /go/src/github.com/zrepl/zrepl/artifacts/release on the 'Artifacts' tab on CircleCI)

@jbliesener

jbliesener commented Mar 13, 2019

Copy link
Copy Markdown

Big patch. OK, I'll give it a try when the travis job terminates. Will there be any x86_64 binary that I can run directly or will I have to build?

@problame

problame commented Mar 13, 2019

Copy link
Copy Markdown
Member Author

Big patch. OK, I'll give it a try when the travis job terminates. Will there be any x86_64 binary that I can run directly or will I have to build?

See link above (edited).

@problame problame mentioned this pull request Mar 13, 2019
Closed
@JMoVS

JMoVS commented Mar 14, 2019

Copy link
Copy Markdown
Contributor

I will try to do some stuff on the weekend. Thx for following up!

@problame problame mentioned this pull request Mar 14, 2019
Closed
@problame
problame force-pushed the problame/replication_refactor branch 2 times, most recently from d0685a8 to f26b2e2 Compare March 15, 2019 15:08
problame added 2 commits March 15, 2019 16:18
@problame
rpc: treat protocol handshake errors as permanent
ab3e783 
treat handshake errors as permanent on the client

The issue was observed by 100% CPU usage due to lack ofrate-limiting in
dataconn.ReqPing retries=> safeguard that
@problame
rpc: do not leak grpc state change logger goroutine
2c3b3c0
@problame
problame force-pushed the problame/replication_refactor branch from f26b2e2 to 2c3b3c0 Compare March 15, 2019 15:18
@problame
problame force-pushed the problame/replication_refactor branch from 944cfa9 to c655622 Compare March 15, 2019 21:43
@problame problame mentioned this pull request Mar 16, 2019
Closed
This was referenced Mar 17, 2019
Closed
Closed
@problame problame mentioned this pull request Mar 19, 2019
Open
@problame problame mentioned this pull request Mar 20, 2019
Closed
@problame
problame merged commit 86fdcfc into master Mar 20, 2019
@problame
problame deleted the problame/replication_refactor branch March 21, 2019 16:22
@Debilski

Debilski commented Mar 25, 2019

Copy link
Copy Markdown

Hi, I was testing the above snapshot last week (freebsd) and it indeed solves a lot of problems that we had with a previous version (no more checksum errors after a few GB). But after a manual zrepl signal wakeup of a push job, I saw dozens (?) of zfs send jobs showing up in htop at the same time driving up the load and eventually crashing the server. Is this a deliberate behaviour that it starts so many sends at once?

@problame problame mentioned this pull request Mar 26, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

awaiting_feedback

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

pruner exits on filesystem-local errors

4 participants

@problame @jbliesener @JMoVS @Debilski