Repository navigation
ZFS send/recv with ashift 9->12 leads to data corruption #12762
Description
System information
| Type | Version/Name |
|---|---|
| Distribution Name | Ubuntu |
| Distribution Version | 20.04 |
| Kernel Version | 5.4.0-90-generic |
| Architecture | x86_64 |
| OpenZFS Version | zfs-2.1.99-528_g8ac58c3f5 |
Describe the problem you're observing
ZFS send/recv from pool with ashift=9 to pool with ashift=12 silently damages files on disk after a subsequent send/recv. No error is reported anywhere, zpool scrub reports no errors, but files are corrupted.
Describe how to reproduce the problem
zdb -C tank | grep ashift
ashift: 9
zdb -C dozer | grep ashift
ashift: 12
zfs set compression=on tank/test
zfs set compression=on dozer
... put some data to /tank/test, I'm using a container image ...
zfs snapshot tank/test@snapshot
zfs send -c -L tank/test@snapshot | zfs recv dozer/test-first
zfs send -c -L dozer/test-first@snapshot | zfs recv dozer/test-second
Now data in dozer/test-second is corrupted. I'm just looking at text files. All files are there, metadata like timestamps and file size seem to match, but the contents are garbled. Please note that the data is apparently ok after the first send/recv, but after the second it is not.
If I repeat the above steps, but remove the -c switch, the data survives.
We have tracked it to commit c634320, If I revert it, I can't reproduce the problem anymore.
Activity
[-]ZFS send/recv with with ashift 9->12 leads to data corruption[/-][+]ZFS send/recv with ashift 9->12 leads to data corruption[/+]pcd1193182 commentedon Nov 15, 2021
I'm looking into this, and the error is odd... when you examine the zdb outputs for a file in
test-firstandtest-second, they appear identical. The l0 blkptrs have the same lsize, same psize, same asize, and same checksum. The datasets themselves also have the same properties, aside from the necessarily different ones like guid and objsetid. So why is the file so messed up?pcd1193182 commentedon Nov 15, 2021
Ah, if you pass the proper flags to zdb to get the full blkpr information, it looks like in
test-second, the blkptr is listed as not compressed. It looks like we're dropping the compression information when we send it on, presumably because the lsize and the psize are the same?pcd1193182 commentedon Nov 15, 2021
The issue here is that the way we do compression detection in the send path is to compare the psize to the lsize. The fix is to have a direct check of the blockpointer and the zio flags to see if the buffer is compressed or not.
marker5a commentedon Jan 5, 2022
Long shot here, but I've been spending time over in #12594 and in the process of debugging, stumbled into this issue which now seems like a candidate for a dataset that is not mountable. It's an encrypted dataset that was migrated from one pool to another using raw sends and there is a non-zero chance I went from a lower shift to a larger shift during the migration, thus causing corruption... at least from the looks of this issue.
Question: If I suffered from this, is it safe to assume that my data is totally garbage now? Or is there a hope of getting some/all of it back?
The odd thing when reading this issue is that the symptom is garbled data. My symptom is that I cannot even mount the dataset (I get input/output errors) to begin with, so it's not clear to me if the blocks themselves are garbled and zfs is throwing errors because it detected that, or if the i/o error stems from some other issue.
rincebrain commentedon Jan 5, 2022
If I read the error condition right, you could try to cut a patch that would try uncompressing unconditionally, even if the blkptr doesn't think it's marked compressed...
psy0rz commentedon Jan 7, 2022
Because we cant always 100% trust zfs, I'm trying to create an intelligent zfs-compare tool that will compare the latest common snapshots in two pools. It will shasum the actual zvols and files, instead of relying on zfs. It will also transfer a remote dataset thats encrypted and only has a local key, so that the encryption key isnt needed remotely. Does anyone want this tool as well?
gdevenyi commentedon Jan 18, 2022
Any idea here of what versions this affects? I have a 140TB migration to do from a 9->12 pool and not super happy having to upgrade to master to get the send working properly...
rincebrain commentedon Jan 18, 2022
2.0.7 and 2.1.2 both include ports of 3760273, so you could go either route.
gdevenyi commentedon Jan 18, 2022
Thanks, that's helpful. Any idea of 0.8.6 is affected?
I see the issue is
-cso I guess I could also avoid that.rincebrain commentedon Jan 18, 2022
Well, the commit the original reporter is pointing to was from September 2021, and 0.8.6 was December 2020, so one assumes not.
Amusingly, 2.0.x does not appear to have gotten the original commit, just the fix...2.1.1 had it, though, so if you were running 2.1.1 specifically, it could happen.
lapo-luchini commentedon Feb 2, 2022
Can this be detected on the
zfs recvside, after the fact?Example given: would a
zpool scrubdetect this?psy0rz commentedon Feb 3, 2022
i think not, which is the biggest ans scariest issue.
the data is corrupt before the checksums etc are calculatuted
38 remaining items
Add a "zstream decompress" subcommand
Add a "zstream decompress" subcommand
Add a "zstream decompress" subcommand
Add a "zstream decompress" subcommand
Add a "zstream decompress" subcommand
Add a "zstream decompress" subcommand
Add a "zstream decompress" subcommand
Add a "zstream decompress" subcommand
Add a "zstream decompress" subcommand
Add a "zstream decompress" subcommand
Add a "zstream decompress" subcommand
Add a "zstream decompress" subcommand
Add a "zstream decompress" subcommand
Add a "zstream decompress" subcommand
Add a "zstream decompress" subcommand