Visitar URL original
Add a pure-Rust _zstd module by youknowone · Pull Request #8917 · RustPython/RustPython · GitHub
Skip to content
Draft
Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
4d68d0e
Add _zstd stdlib module backing compression.zstd
JamesClarke7283 May 23, 2026
9b517da
Fix iOS build by removing static PyMutex<PyTypeRef>
JamesClarke7283 May 23, 2026
5720b10
Fix clippy errors in zstd module
JamesClarke7283 May 23, 2026
d5037be
Retrigger CI
JamesClarke7283 May 23, 2026
6a67eaf
chore: format zstd module
JamesClarke7283 May 23, 2026
0ceea4a
Update crates/stdlib/src/zstd.rs
JamesClarke7283 May 24, 2026
e3b6888
chore: remove rustdoc comments from zstd module
JamesClarke7283 May 24, 2026
72b718c
commit message
JamesClarke7283 May 25, 2026
2575363
commit message
JamesClarke7283 May 25, 2026
c812dd8
Identify zstd parameter enums by type identity, not name
JamesClarke7283 Jun 3, 2026
4fa3ca9
Match CPython error behavior in _zstd ZstdCompressor
JamesClarke7283 Jun 3, 2026
dd03057
Address review: drop decorative separators and unused zstd crate
JamesClarke7283 Aug 9, 2026
1593954
Switch _zstd backend from zstd-safe to pure-Rust libzstd-rs-sys
JamesClarke7283 Aug 9, 2026
6d936c6
Fix data loss in ZstdDecompressor.decompress(max_length=0)
JamesClarke7283 Aug 9, 2026
12b96d8
Drop duplicate spell-checker line in zstd.rs
JamesClarke7283 Aug 9, 2026
1f5b219
Address review on _zstd constants, version attrs and dict markers
JamesClarke7283 Aug 19, 2026
116783a
Replace the _zstd flush-mode constants with a CompressMode enum
JamesClarke7283 Aug 19, 2026
692cbe6
Correct the _zstd explanatory comments and raise MemoryError on OOM
JamesClarke7283 Aug 19, 2026
97b63c5
Fix CPython divergences and a crash in the _zstd module
JamesClarke7283 Aug 19, 2026
14aeaee
Restore Lib/test/test_zstd.py to its vendored CPython state
JamesClarke7283 Aug 19, 2026
c5962fb
Rewrite _zstd against rusty_zstd
youknowone Oct 1, 2026
0507916
Use libzstd-rs-sys for _zstd
youknowone Oct 1, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix CPython divergences and a crash in the _zstd module
Found by reviewing the module against CPython's `Modules/_zstd/` and the
pinned libzstd-rs-sys sources; each item below was reproduced before the
fix and re-checked after.

- `train_dict`/`finalize_dict` sized their output buffer with an
  infallible `Vec::with_capacity`/`vec![]` from a caller-supplied
  `dict_size`, so `train_dict([b'x'], 2**62)` aborted the interpreter
  instead of raising. Reserve fallibly and raise `MemoryError`, as CPython
  does.
- A digested dictionary was always built at `ZSTD_CLEVEL_DEFAULT`. Since
  `ZSTD_CCtx_refCDict` takes its parameters from the CDict, that silently
  discarded the requested level: `level=19` with `as_digested_dict`
  compressed a test corpus to 39817 bytes instead of 31951. Build the
  CDict at the compressor's effective level, tracked through `level=` and
  the `options` dict like CPython's `self->compression_level`.
- A bare `ZstdDict` was loaded as digested in both directions; CPython
  compresses with an undigested dictionary by default and only
  decompresses with a digested one. Give `DictLoader` a per-direction
  default.
- Neither compressor nor decompressor reset its session after an error,
  leaving the object permanently broken — every later call failed with
  "Operation not authorized at current processing stage" or re-reported
  the original corruption on valid input. Reset as CPython's error paths
  do, including restoring `last_mode` to `FLUSH_FRAME`.
- `apply_options` range-checked every parameter, rejecting values libzstd
  and CPython accept: 0 ("use the default") for `window_log`, `strategy`,
  `window_log_max` and friends, any non-zero value for the boolean flags,
  and the clamped `nb_workers`/`job_size`/`overlap_log`. Only
  `compression_level` needs the check — libzstd clamps that one silently,
  which is why `test_compress_parameters` requires a `ValueError` — so
  everything else now goes to libzstd, whose error code already maps to
  the same message.
- `ZstdDict` accepted content shorter than 8 bytes; CPython rejects it up
  front regardless of `is_raw`.
- `compress`/`decompress` refused `data` as a keyword and
  `get_param_bounds` required `is_compress` as one, both contrary to
  CPython's clinic signatures.
- The decompressor allocated a full 128 KiB scratch buffer even when
  `max_length` capped output far below that.

`test_zstd` (119 tests) and the snippet test pass; clippy is clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
  • Loading branch information
2 people authored and youknowone committed Oct 1, 2026
commit 97b63c5572e9b086c82004e8761a3c99054fbd3e
Loading