Use zlib-ng (fast!) rather than mainline stale zlib in binary releases

@gpshead

BPO	47193
Nosy	@gpshead, @pfmoore, @tjguk, @zware, @zooba, @corona10, @arhadthedev

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-04-01.19:19:24.173>
labels = ['3.11', 'OS-windows', 'performance']
title = 'Use zlib-ng rather than zlib in binary releases'
updated_at = <Date 2022-04-02.09:48:33.061>
user = 'https://github.com/gpshead'

bugs.python.org fields:

activity = <Date 2022-04-02.09:48:33.061>
actor = 'corona10'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Windows']
creation = <Date 2022-04-01.19:19:24.173>
creator = 'gregory.p.smith'
dependencies = []
files = []
hgrepos = []
issue_num = 47193
keywords = []
message_count = 1.0
messages = ['416508']
nosy_count = 7.0
nosy_names = ['gregory.p.smith', 'paul.moore', 'tim.golden', 'zach.ware', 'steve.dower', 'corona10', 'arhadthedev']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue47193'
versions = ['Python 3.11']

Linked PRs

zlib-ng is an optimized zlib library with better performance on most architectures (with contributions from the likes of Google, Cloudflare, and Intel). It is API compatible with zlib. https://github.com/zlib-ng/zlib-ng

I believe the only platform we don't use the OS's own zlib on is Windows so I'm tagging this issue Windows.

if this hasn't happened in the 3.11 betas this is presumably bumped to 3.12.

I started taking a look at this and it seems like we can build it without having to worry about their build system by renaming the zconf.h.in and zconf-ng.h.in files to remove the .in, and setting ZLIB_COMPAT preprocessor in pythoncore.vcxproj (as well as referencing the files).

Running test_zlib the only failures seem to be testing for certain failures that no longer fail, but I've got no idea how important they are. Also no indication of the performance impact, or anything else that may change (e.g. new DLL exports, etc.), but it certainly does seem like a feasible drop-in replacement.

@zooba

@zooba which failures are these? I have accumulated quite some experience with the zlib/gzip formats due to working on python-isal, bindings for the ISA-L library that speeds up zlib-compatible compression by rewriting the algorithms in x86 Assembly language. It is quite good, but not suitable for a drop-in replacement, hence the python-isal project.

Having said that I'd love to help out with anything zlib related in CPython.

I knew I should've kept better track of the changed error messages 😄

Skimming through the tests, I'm pretty sure test_wbits failed in a few places because we expected some "invalid window size" errors that aren't errors with zlib-ng. I don't think any were critical, but I don't know the usage of the library well enough to be sure (e.g. do people expect these errors and change behaviour? or are they always just developer error and cause changes to just make things work?)

Contributions to the linux build as in configure.ac & Makefile.pre.in are welcome so that when a recent enough version of zlib-ng is found, it is preferred over vanilla zlib.

pythongh-91349: Expose the crc32 function from the lzma library

quick update,

I have attempted to adjust the linux build to detect and use zlib-ng, to no success.
the .configure script eludes me.
there might only be 10 people in the world who understand any of it.
any help would be greatly appreciated.
liblzma just had a new release a few days ago (xz-utils package on linux). https://github.com/tukaani-project/xz/releases
the latest liblzma release 5.8.x is using hardware CRC instructions, it is now as fast as zlib-ng and libdeflate.
I have submitted a PR to expose crc32 from lzma.
after it is merged, I shall do more PR to pick the optimal crc function between zlib/zlibng/liblzma depending on what is available.

It would be great to see what the stats are compared to zlib-ng. It has optimizations for different architectures and even instruction sets. So it will perform well on older hardware if that is necessary for your use case.

pythongh-91349: Adjust default compression level to 6 (down from 9) i…

pythongh-91349: Expose the crc32 function from the lzma library

gh-91349: Adjust default compression level to 6 (down from 9) in gzip…

pythongh-91349: Adjust default compression level to 6 (down from 9) i…

pythongh-91349: Expose the crc32 function from the lzma library

Just a quick update since the Python interpreter transitioned to zlib-ng on Windows, I've performed extensive benchmarks and analysis since we did the transition, you can read here, it's a massive improvement! ->
Faster Python 3.14: faster compression and fixing 28-year-old bugs

@morotti

Thanks @morotti for the write up. Very interesting. It also seems that my side project of python-zlib-ng is now redundant for windows 3.14 and higher. Less maintenance for me in the future. I hope that zlib-ng gets shipped by distro's soon.

Navigation Menu

Repository navigation

Use zlib-ng (fast!) rather than mainline stale zlib in binary releases #91349

Linked PRs

64 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Use zlib-ng (fast!) rather than mainline stale zlib in binary releases #91349

Description

Linked PRs

Activity

gpshead commented on Apr 1, 2022

gpshead commented on Jun 17, 2022

zooba commented on Jul 17, 2022

rhpvorderman commented on Aug 1, 2022

zooba commented on Aug 1, 2022

64 remaining items

gpshead commented on Mar 27, 2025

morotti commented on Apr 17, 2025

nmoinvaz commented on Apr 17, 2025

morotti commented on May 21, 2026

rhpvorderman commented on Jul 21, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions