Repository navigation
Use BTRFS reflink for safer deduplication #201
Description
tldr;
BTRFS can copy a file without using additional space on disk, and it's safe to delete a file that it was copied from, basically symlinks but show up as a file, and source file can be deleted safely.
https://youtu.be/15_-hgsX2V0?t=409 I've noticed this video in my recommendations and decided to watch it, but on timecode the guy was talking about deduplication being dangerous, as external modification of actual files can break the symlink. Then I immediatly thought about reflinks in BTRFS
BTRFS supports a feature named "reflinks", it's basically like a symlink, but to a user it shows up as a completely regular file, even if you delete the source file that it was copied from, the copy of the file will still be valid, as the source file is like a "link" to the actual data of file, and only the link gets deleted, the actual data is deleted only if there are no "links" to the data.
So this can be used for safe deduplication, maybe a check for this, or an additional flag for deduplication config.
Activity
9001 commentedon Jul 27, 2025
yep, this is an excellent idea -- someone mentioned it in #197 too :>
I plan to add this 👍
DADA30000 commentedon Jul 27, 2025
oh wow, an instant response :D
thx for working on this :)
I will then just leave the issue there to let people see if the feature is implemented
surfaceflinger commentedon Jul 27, 2025
ZFS has/had this quirk when reflinking between datasets, idk if that's still the case tho https://discourse.practicalzfs.com/t/sanity-check-block-cloning/794/2
DADA30000 commentedon Jul 27, 2025
yep, that's why I only mentioned BTRFS, AFAIK only it has completely working reflink implementation without any problems for few years already
9001 commentedon Jul 28, 2025
it appears python 3.14 will have a battery for reflinking; python/cpython#93152
compiling 3.14rc1 to give it a workout (primarily on btrfs) and will add it as an option for the next version if it seems fine
motiejus commentedon Jul 28, 2025
FWIW
copy_file_range(the syscall behind reflink) is also supported on xfs.add reflink-based dedup; closes 9001#201
PotcFdk commentedon Jul 29, 2025
I just wanted to add that
ioctl_fideduperangewould allow to even share partial duplicate ranges within files. Since copyparty already does partial checksums, the easy way to make use of this might be to notice when a range was seen before, assuming the hashes stick around after the upload has finished, which might not be the case (I did not read the source code).9001 commentedon Jul 29, 2025
that's neat!
right now, copyparty is only aware of the individual chunks as the upload is happening - the chunk checksums are discarded when the upload is completed, and only the full combined hash is stored in the db.
if we find other reasons to store the individual checksums at some point, then this would be something to look at for sure.
add reflink-based dedup; closes 9001#201