Repository navigation
Add ZSTD_decompressMultiThreaded(): frame-level parallel decompression - #4796
ceedot-rock wants to merge 1 commit into
Conversation
|
Hi @ceedot-rock! Thank you for your pull request and welcome to our community. Action RequiredIn order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks! |
|
Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks! |
Decompresses concatenated frames concurrently, one frame per worker thread, addressing facebook#2245. Frame boundaries are scanned up front with ZSTD_findFrameCompressedSize(); each frame is then decompressed by its own job with its own decompression context (blocks within a frame are left serial, since later blocks may reference earlier output). - nbThreads <= 1 and single-frame inputs take the plain serial path. - Frames with stored content sizes decompress straight into disjoint dst slices; each job gets exactly its frame's content size as dstCapacity, since the decoder may use spare dst room as scratch. - Frames without a stored content size use per-job temp buffers that grow until the frame fits, then copy into dst in frame order. - Strict all-or-nothing input validation: bad magic -> prefix_unknown, truncation -> srcSize_wrong, short dst -> dstSize_tooSmall. - Without ZSTD_MULTITHREAD the pool runs jobs synchronously, so the result stays correct (just serial). Tests: tests/mtDecompressTest covers thread counts 1-16, the unknown-content-size path, skippable frames, and exact error codes.
800bb55 to
d38d9bf
Compare
This branch has not been deployed
1 participant
Addresses #2245 (related: #4775).
Adds a one-shot ZSTD_decompressMultiThreaded(dst, dstCapacity, src, srcSize, nbThreads). It scans the frame boundaries up front, then decompresses frames in parallel on the existing thread pool from lib/common/pool.h. One frame, or nbThreads <= 1, goes through plain ZSTD_decompress() with no thread overhead.
When every frame's content size is known (the normal case), each frame decodes straight into its slice of dst - no extra copies. Frames without a stored content size fall back to per-frame temp buffers gathered in order. Strict all-or-nothing: bad magic -> prefix_unknown, truncation -> srcSize_wrong, short dst -> dstSize_tooSmall.
One thing worth knowing for review: ZSTD_decompress() parks a block's literals in spare dstCapacity past the frame's true end (ZSTD_allocateLiteralsBuffer). The first cut handed each job the whole dst tail and threads scribbled one byte into the next frame's slice. Each job now gets exactly its frame's content size as its capacity, slices are disjoint, canary tests confirm no cross-talk.
This is the frame-level approach from the #2245 discussion, not single-frame block parallelism - that stays a separate, harder problem.
Tests: new tests/mtDecompressTest.c - 8 mixed frames (incl. skippable) byte-identical to serial across 1/2/4/8/16 threads, unknown-content-size path, single frame, 0/negative thread counts, exact error codes for empty/truncated/corrupt/short-dst. Fuzzer (1083) and zstreamtest (817) pass, ASan clean. Benchmark on 16x1MB frames: 1.7x with 2 threads on a 2-core box (saturates it).
Slid Phi Labs accepts donations to keep the lab independent.