Visitar URL original
Add a pure-Rust _zstd module by youknowone · Pull Request #8917 · RustPython/RustPython · GitHub
Skip to content
Draft
Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
4d68d0e
Add _zstd stdlib module backing compression.zstd
JamesClarke7283 May 23, 2026
9b517da
Fix iOS build by removing static PyMutex<PyTypeRef>
JamesClarke7283 May 23, 2026
5720b10
Fix clippy errors in zstd module
JamesClarke7283 May 23, 2026
d5037be
Retrigger CI
JamesClarke7283 May 23, 2026
6a67eaf
chore: format zstd module
JamesClarke7283 May 23, 2026
0ceea4a
Update crates/stdlib/src/zstd.rs
JamesClarke7283 May 24, 2026
e3b6888
chore: remove rustdoc comments from zstd module
JamesClarke7283 May 24, 2026
72b718c
commit message
JamesClarke7283 May 25, 2026
2575363
commit message
JamesClarke7283 May 25, 2026
c812dd8
Identify zstd parameter enums by type identity, not name
JamesClarke7283 Jun 3, 2026
4fa3ca9
Match CPython error behavior in _zstd ZstdCompressor
JamesClarke7283 Jun 3, 2026
dd03057
Address review: drop decorative separators and unused zstd crate
JamesClarke7283 Aug 9, 2026
1593954
Switch _zstd backend from zstd-safe to pure-Rust libzstd-rs-sys
JamesClarke7283 Aug 9, 2026
6d936c6
Fix data loss in ZstdDecompressor.decompress(max_length=0)
JamesClarke7283 Aug 9, 2026
12b96d8
Drop duplicate spell-checker line in zstd.rs
JamesClarke7283 Aug 9, 2026
1f5b219
Address review on _zstd constants, version attrs and dict markers
JamesClarke7283 Aug 19, 2026
116783a
Replace the _zstd flush-mode constants with a CompressMode enum
JamesClarke7283 Aug 19, 2026
692cbe6
Correct the _zstd explanatory comments and raise MemoryError on OOM
JamesClarke7283 Aug 19, 2026
97b63c5
Fix CPython divergences and a crash in the _zstd module
JamesClarke7283 Aug 19, 2026
14aeaee
Restore Lib/test/test_zstd.py to its vendored CPython state
JamesClarke7283 Aug 19, 2026
c5962fb
Rewrite _zstd against rusty_zstd
youknowone Oct 1, 2026
0507916
Use libzstd-rs-sys for _zstd
youknowone Oct 1, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix iOS build by removing static PyMutex<PyTypeRef>
`PyMutex` is built on `RawCellMutex` on single-threaded targets (iOS,
Android, wasm32) and is not `Sync`, so a `static PyMutex<...>` fails to
compile. `PyTypeRef` is also not `Send` by default, which compounds the
issue.

Replace the static parameter-type registry with class-name comparison
(`"CompressionParameter"` / `"DecompressionParameter"`). The two names
originate from the `compression.zstd` module we ship, so identity vs.
name comparison is equivalent in practice. `set_parameter_types` now
just validates that its arguments are type objects.

Verified locally:
- `cargo build -p rustpython-stdlib` (default features): clean.
- `cargo check -p rustpython-stdlib --no-default-features --features host_env`
  (simulates iOS by dropping the `threading` feature): clean.
- `cargo run -- -m test test_zstd`: 119 tests pass, 1 expected failure.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
  • Loading branch information
2 people authored and youknowone committed Oct 1, 2026
commit 9b517da20da1824c835c15b0d27d7defa04899df
81 changes: 37 additions & 44 deletions crates/stdlib/src/zstd.rs
Original file line number Diff line number Diff line change
Expand Up @@ -37,23 +37,15 @@ mod _zstd {
use zstd_safe::zstd_sys;
use zstd_safe::{CCtx, CParameter, DCtx, DParameter, InBuffer, OutBuffer};

/// Stores the `CompressionParameter` and `DecompressionParameter` enum
/// types that the pure-Python `compression.zstd` wrapper passes through
/// `set_parameter_types`. They let us produce CPython-compatible
/// `TypeError`s when callers pass the wrong enum kind to a (de)compressor
/// (a `CompressionParameter` key to a decompressor, or vice versa). Stored
/// in a `PyMutex` so the module init can set them once and constructors
/// can read them later from any thread.
static REGISTERED_PARAM_TYPES: PyMutex<RegisteredParamTypes> =
PyMutex::new(RegisteredParamTypes {
compression: None,
decompression: None,
});

struct RegisteredParamTypes {
compression: Option<PyTypeRef>,
decompression: Option<PyTypeRef>,
}
/// Class names of the `CompressionParameter` and `DecompressionParameter`
/// `IntEnum`s defined by the pure-Python wrapper at
/// `Lib/compression/zstd/__init__.py`. We identify these by name rather
/// than identity because storing `PyTypeRef` in a Rust `static` requires
/// `Sync`, which `PyMutex` does not provide on single-threaded targets
/// (Android, iOS, wasm32). Name comparison is sufficient since both names
/// originate from a module we ship and control.
const C_PARAMETER_TYPE_NAME: &str = "CompressionParameter";
const D_PARAMETER_TYPE_NAME: &str = "DecompressionParameter";

// =========================================================================
// Module-level constants
Expand Down Expand Up @@ -207,29 +199,29 @@ mod _zstd {
arg.into_option().filter(|o| !vm.is_none(o))
}

/// If `key` is a registered `CompressionParameter` or `DecompressionParameter`
/// enum member and its kind does not match the caller's context, return a
/// `TypeError`. Used by both (de)compressor option-dict loops so passing a
/// `CompressionParameter` to a `ZstdDecompressor` (or vice versa) is
/// rejected with the CPython-compatible "must not be a XParameter"
/// message. If `set_parameter_types` has not been called yet, this is a
/// no-op.
/// If `key`'s class is the wrong kind of parameter enum for the caller's
/// context (a `CompressionParameter` passed to a decompressor, or vice
/// versa), return a `TypeError`. Used by both (de)compressor option-dict
/// loops so the resulting message names the actual type rather than the
/// generic "out of range" wording.
///
/// We identify the enums by class name (see [`C_PARAMETER_TYPE_NAME`]),
/// which avoids storing a `PyTypeRef` in a Rust `static` (impossible on
/// single-threaded builds whose `PyMutex` is not `Sync`).
fn check_wrong_param_kind(
key: &PyObjectRef,
is_compress: bool,
vm: &VirtualMachine,
) -> PyResult<()> {
let guard = REGISTERED_PARAM_TYPES.lock();
let (forbidden_type, forbidden_name, kind) = if is_compress {
(&guard.decompression, "DecompressionParameter", "compression")
let class_name = key.class().name();
let (forbidden, kind) = if is_compress {
(D_PARAMETER_TYPE_NAME, "compression")
} else {
(&guard.compression, "CompressionParameter", "decompression")
(C_PARAMETER_TYPE_NAME, "decompression")
};
if let Some(ref t) = *forbidden_type
&& key.fast_isinstance(t)
{
if &*class_name == forbidden {
return Err(vm.new_type_error(format!(
"{kind} options dictionary key must not be a {forbidden_name}"
"{kind} options dictionary key must not be a {forbidden}"
)));
}
Ok(())
Expand Down Expand Up @@ -1684,29 +1676,30 @@ mod _zstd {

/// set_parameter_types(c_parameter_type, d_parameter_type, /)
///
/// Register the `CompressionParameter` and `DecompressionParameter`
/// `IntEnum` types defined in the pure-Python wrapper. Once registered,
/// the (de)compressor constructors validate that callers do not mix the
/// two enum kinds (e.g. passing a `CompressionParameter` key to a
/// `ZstdDecompressor`), raising `TypeError` if they do.
/// Validate that the `CompressionParameter` and `DecompressionParameter`
/// types passed in by the pure-Python wrapper are actual type objects.
/// The (de)compressor constructors use the *names* of these classes
/// (see [`check_wrong_param_kind`]) rather than identity, so this
/// function does not need to retain the type objects beyond the call.
///
/// The Python wrapper at `Lib/compression/zstd/__init__.py:242` calls
/// this exactly once at import time.
/// CPython retains them and uses them for nicer error messages naming
/// the specific enum member; matching that behavior would require a
/// `Sync` cell-of-`PyTypeRef`, which is not available on single-threaded
/// targets. The pure-Python wrapper at
/// `Lib/compression/zstd/__init__.py:242` calls this exactly once at
/// import time.
#[pyfunction]
fn set_parameter_types(
c_parameter_type: PyObjectRef,
d_parameter_type: PyObjectRef,
vm: &VirtualMachine,
) -> PyResult<()> {
let c_type = c_parameter_type
let _ = c_parameter_type
.downcast::<PyType>()
.map_err(|_| vm.new_type_error("c_parameter_type must be a type object"))?;
let d_type = d_parameter_type
let _ = d_parameter_type
.downcast::<PyType>()
.map_err(|_| vm.new_type_error("d_parameter_type must be a type object"))?;
let mut guard = REGISTERED_PARAM_TYPES.lock();
guard.compression = Some(c_type);
guard.decompression = Some(d_type);
Ok(())
}
}