Repository navigation
Export zlib.{adler,crc}32_combine #134635
Description
Activity
cc @emmatyping (or is zstd not supporting this one? I don't know if you now want to take care of compression issues as well in general)
@picnixz this is zlib related, not zstandard, but I am hoping to take care of more compression issues going forward!
I think this request is reasonable, we probably would want to also add
adler32_combinefor completeness/symmetry.I don't think we should add
crc32_combine_gen/crc32_combine_opwithout additional motivation.this is zlib related
Yeah, I actually misread the issue first :')
I am hoping to take care of more compression issues going forward!
Great to hear!
adler32_combine
Yes, sure. Note that we could also think about whether to add
binascii.crc32_combineas well (we have a minimal support for CRC32 even without zlib in binascii but I don't know if it would also make sense to exposecrc32_combinein this case). I wouldn't recommend it as a first iteration (and we anyway don't havebinascii.adler32)Reacted by Emma SmithI should note that if one is merely iterating over chunks of data, using the
valueargument tozlib.crc32(i.e. setting an initial checksum) should work to compute a running checksum.However, looking at the fastzip code, they are concurrently calculating the checksums and assembling them afterwards, so they cannot chain calls to
zlib.crc32.- marked Expose 'adler32_combine' function from zlib #82868 as a duplicate of this issue
on May 25, 2025 - changed the title
[-]Expose crc32_combine from zlib[/-][+]Export `zlib.{adler,crc}32_combine`[/+]on May 25, 2025 - added a commit that references this issue
on May 27, 2025
Metadata
Metadata
Assignees
Projects
- Status
Feature or enhancement
Proposal:
This is already provided by zlib, and is useful for computing a crc32 in multiple chunks (not linearly).
What I currently do is use ctypes, but that doesn't work on all environments (like Windows). I don't think we'll need a pure python fallback, just a wrapper of what's in zlib, equivalent of https://github.com/fastzip/fastzip/blob/9019107a6732fab9004e625b075faab96f29265a/fastzip/_crc32_combine.py#L63-L67
Has this already been discussed elsewhere?
This is a minor feature, which does not need previous discussion elsewhere
Links to previous discussion of this feature:
No response
Linked PRs
zlib.{adler32,crc32}_combineto combine checksums #134650