Repository navigation
transport type local panics on Illumos #598
Description
I'm running zrepl-0.5.0 on OI, an illumos distro. The daemon terminates with this error message:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xaea33e]
goroutine 614 [running]:
github.com/zrepl/zrepl/rpc/grpcclientidentity.NewInterceptors.func1({0xf1e158, 0xc000631200}, {0xd514c0, 0xc000631230}, 0xc000032740, 0xc000524348)
/dpool/export/home/mills/Downloads/code/oi-userland-gh/components/sysutils/zrepl/build/amd64/rpc/grpcclientidentity/authlistener_grpc_adaptor.go:121 +0x13e
github.com/zrepl/zrepl/replication/logic/pdu._Replication_ListFilesystems_Handler({0xdb30c0, 0xc00001a630}, {0xf1e158, 0xc000631200}, 0xc00052b7a0, 0xc000522000)
/dpool/export/home/mills/Downloads/code/oi-userland-gh/components/sysutils/zrepl/build/amd64/replication/logic/pdu/pdu_grpc.pb.go:186 +0x16a
google.golang.org/grpc.(*Server).processUnaryRPC(0xc00016e700, {0xf2bc00, 0xc0000f2780}, 0xc00011c200, 0xc000522150, 0x1497c78, 0x0)
/dpool/export/home/mills/Downloads/code/oi-userland-gh/components/sysutils/zrepl/zrepl-0.5.0/gopath/pkg/mod/google.golang.org/grpc@v1.35.0/server.go:1217 +0xe28
google.golang.org/grpc.(*Server).handleStream(0xc00016e700, {0xf2bc00, 0xc0000f2780}, 0xc00011c200, 0x0)
/dpool/export/home/mills/Downloads/code/oi-userland-gh/components/sysutils/zrepl/zrepl-0.5.0/gopath/pkg/mod/google.golang.org/grpc@v1.35.0/server.go:1540 +0xcb3
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc000373b70, 0xc00016e700, {0xf2bc00, 0xc0000f2780}, 0xc00011c200)
/dpool/export/home/mills/Downloads/code/oi-userland-gh/components/sysutils/zrepl/zrepl-0.5.0/gopath/pkg/mod/google.golang.org/grpc@v1.35.0/server.go:878 +0xad
created by google.golang.org/grpc.(*Server).serveStreams.func1
/dpool/export/home/mills/Downloads/code/oi-userland-gh/components/sysutils/zrepl/zrepl-0.5.0/gopath/pkg/mod/google.golang.org/grpc@v1.35.0/server.go:876 +0x1ec
I was testing a replication config when this happened. Zrepl had not even created the intermediate filesystems when this happened. What causes this error? How do I prevent it?
Activity
problame commentedon May 7, 2022
So, the offending line is
zrepl/rpc/grpcclientidentity/authlistener_grpc_adaptor.go
Line 121 in efe7b17
I'm pretty sure
logger != nil, but not so sure aboutpwhich is returned from gRPC'speerpackage.That code returns
!= nilon other platforms, but this seems like the kind of thing that could be broken on Illumos.Can you reproduce this?
If so, please enable debug logs and provide them:
https://zrepl.github.io/configuration/logging.html?highlight=logging
jgmills commentedon May 8, 2022
I will do the debug logging, but first I should report that I'm testing zrepl on a zpool that does not support feature flags of any sort. I created it this way:
# zpool create -o version=15 dpool mirror ... ...Here's what "zpool status" says about it:
# zpool status pool: dpool state: ONLINE status: The pool is formatted using a legacy on-disk format. The pool can still be used, but some features are unavailable. action: Upgrade the pool using 'zpool upgrade'. Once this is done, the pool will no longer be accessible on software that does not support feature flags. scan: none requested config: ...Nevertheless, zrepl should report the problem instead of dereferencing a null pointer.
jgmills commentedon May 14, 2022
Okay, this is part of the debugging log output. It's slightly reformatted to fit my screen, with empty lines between messages. I have the whole thing in raw format, if that helps.
problame commentedon May 15, 2022
These log messages don't provide enough context. I'd need the last 10 seconds or so.
Also, please generate a coredump by running the zrepl daemon with
GOTRACEBACK=crash.Learn more here: https://medium.com/a-journey-with-go/go-debugging-with-delve-core-dumps-384145b2e8d9
(Don't post the coredump publicly, it contains sensitive information. Either use the
dlvdebugger to explore it yourself, or share it with me via e-mail)FYI, you can use three backticks to insert preformatted text. I edited your last message to make it look nice. Learn more at https://github.github.com/gfm/
jgmills commentedon May 15, 2022
Something peculiar happened when I prefixed the command with "env GOTRACEBACK=crash". It did dump core, as I expected, but the output ran off the screen. I tried it again, under "script" so I could capture all of the output. That time, it did not terminate with a core dump, but kept on going. The same thing happened when I tried it again without script. Both times I had to kill the daemon to terminate it. So, I have the full output from a previous run, without a corresponding core file, and the core file from the next run, without the output.
By the way, illumos and OI do not have the "dlv" debugger. I can use "mdb" on the core file, but it's not particularly useful. I could send you the core file if I knew your e-mail address. Here is the full output from a previous run:
problame commentedon Jun 19, 2022
My conclusion from this output is still the same as above.
We're using
socketpairto implement thelocaltransport type. Maybe gRPC is a little special about it on Illumos?You can send the coredumps to me, using the e-mail address on my website, or the one that shows in my GitHub profile.
jgmills commentedon Jun 20, 2022
I will try sending the core file by e-mail, but I may run up against the size limit enforced by my e-mail service. I'll let you know what happens. In the meantime, here is an mdb session against the core file:
As you can see, it's not very useful for this core dump.
jgmills commentedon Jun 24, 2022
Yes, I was unable to send the core file to you through e-mail. My e-mail service did block it because of its size. Is there some other way I could get the core file to you?
problame commentedon Jun 25, 2022
Sent an upload link to you via email.
jgmills commentedon Jun 25, 2022
Thanks. I uploaded zrepl.core to that location.
problame commentedon Jun 27, 2022
Would you please upload the zrepl binary to that location as well?
jgmills commentedon Jun 27, 2022
I just uploaded this file:
problame commentedon Jun 27, 2022
Quick glimpse at the core dump confirms that
pisnil:So, why did
p, ok := peer.FromContext(ctx)returnnil, ok?Let's inspect the local
ctx, which for some reason doesn't show up in dlv'slocalsoutput, but is still accessible:There we go. The underlying
Addrfield is indeed nil.So who is supposed to set the
peer.Peervalue?Good thing we just need to look for callers of
peer.NewContextwithin the gRPC source code.(zrepl is using
google.colang.org/grpc@1.35.0)A quick [SourceGraph search(https://sourcegraph.com/search?q=context:global+repo:%5Egithub%5C.com/grpc/grpc-go%24%40v1.35.0+peer.NewContext&patternType=literal) reveals three places that are particularly relevant:
My gut feeling is that, for our code path, the first one is most relevant.
It calls
.RemoteAddr()to get thenet.Addrwhich isnilin your case, but usually (Linux, FreeBSD) it's!= nil.Where to go from here?
Here are a few experiments to run.
Please report if the daemon crashes or not, and if it crashes, please upload the binary & core files into correspondingly named directories on the web share that I sent you.
transport_tcp:Use a 127.0.0.1 TCP listener (zrepl transport type
tcp) instead of thelocaltransport.go117:Use Go 1.17 to compile the package.
zrepl_0.4:Use Go 1.18 with zrepl 0.4.
If you can't do some of the experiments, just do the ones you can and report back. The most important one is
transport_tcp.6 remaining items
jgmills commentedon Jul 5, 2022
I can give you access to one of my servers, although it may take me some time to set this up. It's a bit complicated because it's a two-step process. First you would ssh to an IP address. That would mean that you login to my main server. Then you connect to one of the servers on my private network, where you can do your debugging. I may be able to automate the two-step part. How much disk space do you need?
Alternatively, you can download an image from https://www.openindiana.org/download/. OI is an illumos distro. I'd recommend the live USB image. These will boot and install to a virtual disk from a variety of virtualization platforms. The version available from that web page is 2021-10 . I'm running 2022-05 on my development server, making it only 7 months newer. That should be fine. Will that way be suitable for you?
jgmills commentedon Jul 6, 2022
The download link above should be https://www.openindiana.org/download/ .
problame commentedon Jul 6, 2022
I created a VM and am able to reproduce the crash.
jgmills commentedon Jul 6, 2022
That's excellent news. illumos does have a socketpair function, so I'd guess that the zrepl source is doing something wrong. Here's the man page:
problame commentedon Jul 7, 2022
I have opened the following bug against the Go stdlib: golang/go#53739
tl;dr:
getpeernamedoesn't work from Go on socketpair sockets. That's why.RemoteAddr()returns nil. And that's what leads to the panic, because gRPC / zrepl expect it to be not nil.I have no idea who exactly is at fault here, but it seems like Illumos is quite alone in not returning a RemoteAddr in this case.
Anyways, I guess the workaround will be to check for != nil first, and hope we only look at it in this specific place where it crashes.
rpc/grpcclientidentity/authlistener_grpc_adaptor: don't assume peer.A…
problame commentedon Jul 7, 2022
Validated the commit on my OpenIndiana VM. @jgmills please try it out and report back.
jgmills commentedon Jul 8, 2022
I'm curious what the return value of getpeername is, and if it is non-zero, what the value of errno is. Typically, you should check for successful invocation of the function, before you examine any of the function parameters. On illumos, I'm guessing that a socket pair is actually a pipe, and the peer may not have a name. The man page for getpeername on illumos is shown below:
problame commentedon Jul 8, 2022
Well, that's the thing. From C,
getpeernamereturns 0, and everything is all right.But from Go,
syscall.Getpeernamereturns an error.My gut feeling is that something in Go is wrong, though.
However, for all practical purposes, we're going to have to work around this in zrepl.
Were you able to test the patch above?
jgmills commentedon Jul 9, 2022
I'm pleased to report that, with the patch, the local RPC transport worked: the daemon did not crash. As far as I could tell, the daemon operated normally.
problame commentedon Jul 10, 2022
Cool, I'll merge it then.
rpc/grpcclientidentity/authlistener_grpc_adaptor: don't assume peer.A…