Repository navigation
Observed memory leak in ssl library: Python 3.14 GC issue #142516
Description
Activity
The details are explained in the urllib3 ticket including outputs from memray and it points to ssl package (precisely ssl.SSLContext.load_verify_locations), see: urllib3/urllib3#3738
Please provide a reproducer, if possible using only the
sslmodule.No reproducer but if it helps: very similar situation here when connecting to valkey with a custom CA cert.
Reacted by IvanI did some more testing to come up with a minimal repro: seems like rss just keeps growing while the heap stays somewhat constant. I was unable to reproduce it using a simple sync case locally and on the cluster and this started to smell an awful lot like #109534 which might be async-related. Like @HenriBlacksmith's case we're also seeing this in an ASGI app on k8s using the debian trixie base image since the 3.14 release.
I'll attempt to redeploy this glibc-less and report back.
Update: the alleged leak is still there even in an alpine container
Reacted by FaolainPotentially related: #84904
I applied a fix that improved the memory situation on my other FastAPI app by reducing the calls generated by MSAL.
I applied a few dependency upgrades on top including a bump of the Python image from 3.14.0-slim to 3.14.2-slim and it caused the memory leak to be worse.
Though I also need to collect detailed figures about this one as urllib3 version is different in this app because of some version constraints on a library.
I will try to get more info from memray on this one.
Reacted by Alex PooneI was able to export new memray profiles from my other application.
This time the leak is coming from
httpxwhich also relies on ssl package under the hood:
Sorry for the cuts on the capture, I am trying to "anonymize" things
Thank you! I'm not 100% sure how to interpret these memray outputs to be honest, because if I understand this correctly this shows us where allocations do happen but not necessarily where the leak is (it may be correlated though). This is useful for finding out 'which objects eat up my memory?'. Technically speaking memray will only show us the allocations/leaks the python allocator knows about, but the leak kind of escapes the realm of the allocator by definition. Another layer which might be adding to this is that this might actually be fragmentation instead of leaks. I hope I ruled this out more or less by checking it's not glibc-related by also observing this behaviour on an alpine container, feel free to double-check.
I'm going to try to find some time to give the minimal (hopefully stdlib only) repro another go and explore analyzing a full heap dump.
I am trying to use
valgrindinstead ofmemrayto get more details (but I am a bit out of my comfort zone to be honest), it does not report much until k8s restarts the container.The command looks like:
valgrind --tool=memcheck --log-file=/tmp/valgrind.log --trace-children=yes --leak-check=yes --verbose python -m uvicorn app.main:shell --host 0.0.0.0 --log-config logging.yamlBut nothing really comes out of it
Reacted by SebastianAnother layer which might be adding to this is that this might actually be fragmentation instead of leaks. I hope I ruled this out more or less by checking it's not glibc-related by also observing this behaviour on an alpine container, feel free to double-check.
I just built an Alpine container and ran it and the behavior is similar (based on
python:3.14.2-alpine3.23)I am running out of ideas, if anyone has guidance to help with investigating this, I would be really happy to test those, if I have extra time, I will try to build a MRO but I am not sure how it can be done
Good news, I might have a MRE, it is using
httpxbut that's the best I was able to do, I will share the whole code in a repo, including commands to export the graphs.Here is the code:
import httpx import asyncio async def _main() -> None: while(1): async with httpx.AsyncClient() as client: await client.request("GET", "https://example.com/") await asyncio.sleep(1) if __name__ == "__main__": asyncio.run(_main())
FROM astral/uv:0.9.17-python3.14-bookworm AS builder ENV UV_COMPILE_BYTECODE=1 UV_LINK_MODE=copy UV_PYTHON_DOWNLOADS=0 # Create and set the working directory WORKDIR /mro-app # Install runtime dependencies RUN --mount=type=cache,target=/root/.cache/uv \ --mount=type=bind,source=uv.lock,target=uv.lock \ --mount=type=bind,source=pyproject.toml,target=pyproject.toml \ uv sync --frozen --no-install-project --no-dev # Copy the application code (excluding files in .dockerignore) COPY . . RUN --mount=type=cache,target=/root/.cache/uv \ uv sync --frozen --no-dev # Use an official lightweight Python image with slim variant FROM python:3.14.2-slim # Set environment variables # See: https://docs.python.org/3/using/cmdline.html#envvar-PYTHONUNBUFFERED ENV PYTHONUNBUFFERED=1 # Create and set the working directory WORKDIR /mro-app # Expose the port the app runs on EXPOSE 8000 # Copy the application from the builder COPY --from=builder /mro-app /mro-app # Place executables in the environment at the front of the path ENV PATH="/mro-app/.venv/bin:$PATH" # Define the default command to run when the container starts CMD ["memray", "run", "--native", "--output", "/tmp/capture.bin", "main.py"]
[project] name = "mro" version = "0.1.0" description = "Add your description here" readme = "README.md" requires-python = ">=3.14" dependencies = [ "httpx>=0.28.1", "memray>=1.19.1", "urllib3>=2.6.2", ]
etc. will build a repo asap with all the other files
Reacted by Sebastian and Max ChernoffHere is the repo, I hope it is an actual valid MRE and not just "normal" usage: https://github.com/HenriBlacksmith/python-ssl-memory-usage-mro/tree/main
Reacted by Sebastian, Faolain and Max ChernoffAnd I was finally able to get the native symbols:
I will try to compare with older Python versions to see if the sensitivity to Python version holds with the MRE.
I was able to reproduce on my Mac in a non-Docker environment.
Reacted by Sebastian47 remaining items
GH-148694 is a backport of this pull request to the 3.14 branch.
It was decided to Reverting the incremental GC in Python 3.14 and 3.15: see PR gh-148720 for the 3.14 branch. So can we close this issue?
Reacted by Neil Schemenauer, HenriBlacksmith, Sergey Miryanov, curekoshimizu, Jalin Wang and Yuki FurukawaReacted by Alex PooneThis has been reverted and released in Python 3.14.5 release candidate, please test.
https://discuss.python.org/t/python-3-14-5-release-candidate/107185/1
We plan a 3.14.5 final for Friday.
Reacted by Alex Poone, Gastón Avila, Nathan Burns, Haowei Wang, Jalin Wang, Yuki Furukawa and Vladyslav BurylovThis has been reverted and released in Python 3.14.5 release candidate, please test.
https://discuss.python.org/t/python-3-14-5-release-candidate/107185/1
We plan a 3.14.5 final for Friday.
I have been testing this on two APIs in our dev environment and it looks memory is stable. Will provide more feedback once it will be in prod.
Reacted by Hugo van Kemenade, Victor Stinner, Sergey Miryanov and Alessio Fachechi- added a commit that references this issue
on May 24, 2026 - added 4 commits that reference this issue
on May 28, 2026 This has been reverted and released in Python 3.14.5 release candidate, please test.
https://discuss.python.org/t/python-3-14-5-release-candidate/107185/1
We plan a 3.14.5 final for Friday.I have been testing this on two APIs in our dev environment and it looks memory is stable. Will provide more feedback once it will be in prod.
It has been running in prod for a week with similar memory profiles as for 3.13.x.
The issue is resolved from my point of view
Reacted by Hugo van Kemenade, Sergey Miryanov, Cristian M, Gregory P. Smith and 93578237I can also confirm that the memory footprint is now "stable" since 3.14.5 - and a side benefit has been that the "floor" is lower from 3.13.x
Reacted by Hugo van Kemenade, Sergey Miryanov, Cody Maloney, HenriBlacksmith, Cristian M, Gregory P. Smith, Felix Fanghaenel, 93578237, Victor Stinner and Kuthair Habboush
Bug report
Bug description:
I already opened a bug on
urllib3for this issue, it looks that may application using MSAL for Python library (which relies onrequests, which relies onurllib3, which relies onssl) is leaking memory when being upgraded from Python 3.13 to Python 3.14 (I tested 3.14.{0, 1, 2})The details are explained in the
urllib3ticket including outputs frommemrayand it points to ssl package (preciselyssl.SSLContext.load_verify_locations), see: urllib3/urllib3#3738CPython versions tested on:
3.14
Operating systems tested on:
Linux
Linked PRs
ssl.SSLContextobjects #143685ssl.SSLContextobjects (GH-143685) #145075ssl.SSLContextobjects (GH-143685) (GH-145075) #148371