Review: PR #14070 (venv-based build environment)

Two correctness regressions versus the legacy build environment, plus one
low-severity test defect. Both regressions are gated behind
--use-feature=venv-isolation. Verified end-to-end against the PR head
(bc9597d9).

Setup

# PR code in a throwaway worktree (keeps your checkout untouched)
git fetch upstream pull/14070/head
git worktree add /tmp/pip-pr14070 bc9597d9
RUNNER=/tmp/pip-pr14070/src/pip/__pip-runner__.py   # runs the PR's pip from source

# a source tree that needs an isolated setuptools build
mkdir -p /tmp/pkg/sampleproj
cat > /tmp/pkg/pyproject.toml <<'EOF'
[build-system]
requires = ["setuptools>=60"]
build-backend = "setuptools.build_meta"
[project]
name = "sampleproj"
version = "0.1"
EOF
printf 'from setuptools import setup\nsetup()\n' > /tmp/pkg/setup.py
echo 'x = 1' > /tmp/pkg/sampleproj/__init__.py

# a host venv to launch pip from for Bug 1 (pip itself comes from $RUNNER)
python3 -m venv --without-pip /tmp/host

--dry-run suffices; the build env and its deps are set up before the dry-run
report.

Bug 1 (high): PIP_PREFIX / pip.conf prefix= redirects build deps out of the venv

src/pip/_internal/build_env/installer.py,
SubprocessBuildEnvironmentInstaller.install: the new if prefix.venv_executable:
branch drops the --prefix <path> that the legacy branch passes. pip derives the
default for --prefix from PIP_PREFIX and from a pip.conf [install] prefix=.
With no --prefix on the command line, that ambient value wins, so build deps
install to the user's prefix instead of the temp venv. The venv is left empty, so
importing the backend with the venv python fails.

PIP_PREFIX=/tmp/leak /tmp/host/bin/python "$RUNNER" install \
    --use-feature=venv-isolation --dry-run /tmp/pkg
#   Getting requirements to build wheel: finished with status 'done'
# ERROR: Exception:
# pip._vendor.pyproject_hooks._impl.BackendUnavailable: Cannot import 'setuptools.build_meta'

Controls, both succeed (Would install sampleproj-0.1):

env -u PIP_PREFIX /tmp/host/bin/python "$RUNNER" install --use-feature=venv-isolation --dry-run /tmp/pkg
PIP_PREFIX=/tmp/leak /tmp/host/bin/python "$RUNNER" install                        --dry-run /tmp/pkg  # legacy is immune

A pip.conf [install] prefix= feeds the same option and behaves identically. This
is the subprocess installer only; the inprocess installer
(--use-feature=inprocess-build-deps) passes prefix.path directly and ignores
PIP_PREFIX (verified). --target "" still neutralizes PIP_TARGET and
--no-user still covers PIP_USER; only --prefix was dropped. PIP_ROOT
affects both the venv and legacy subprocess branches (neither passes --root), so
it is not introduced here. The comment at installer.py ("The prefix specified two
lines above ...") is now stale for this branch.

Fix: pin the install location in the venv branch. Both are immune to PIP_PREFIX
(verified): add --prefix <venv_root> (in a venv this resolves to the venv's own
site-packages), or add --isolated.

Bug 2 (high on standalone interpreters): EnvBuilder() copies the interpreter, the build venv cannot start

src/pip/_internal/build_env/venv.py: venv.EnvBuilder() uses the constructor
default symlinks=False, so on POSIX it copies the interpreter. python -m venv
symlinks on POSIX. For relocatable CPython (python-build-standalone, used by
uv/rye/mise/hatch and most CI) a copied interpreter looks for its stdlib next to
pyvenv.cfg's home. When pip is launched directly by such an interpreter whose
on-PATH name is a shim (for example ~/.local/bin/python3.12), home is the shim
directory, which has no stdlib, so the copied python cannot boot and every
isolated build fails. Running pip from inside an already-created venv is unaffected
(there home points at the real interpreter directory).

# pip launched directly by a uv/standalone shim (not inside a venv)
~/.local/bin/python3.12 "$RUNNER" install --use-feature=venv-isolation \
    --break-system-packages --dry-run /tmp/pkg
#   Installing build dependencies: finished with status 'error'
#   Could not find platform independent libraries <prefix>
#   Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
#   ModuleNotFoundError: No module named 'encodings'

The same command without --use-feature=venv-isolation builds fine
(Would install sampleproj-0.1). Minimal mechanism:

~/.local/bin/python3.12 -c '
import venv, subprocess
b = venv.EnvBuilder(); c = b.ensure_directories("/tmp/bv"); b.create("/tmp/bv")
print("copied :", subprocess.run([c.env_exec_cmd, "-c", "print(1)"]).returncode)'
# copied : 1   (returncode; the boot fails with: No module named encodings)
# with symlinks=True the returncode is 0

Even when the copy boots, it duplicates the interpreter binary (tens of MB) per
build venv; the legacy env copied nothing.

Fix: venv.EnvBuilder(symlinks=(os.name != "nt")), matching the python -m venv
default.

Bug 3 (low): xfail(strict=True) shebang test can XPASS and fail CI

tests/functional/test_build_env.py,
test_venv_build_env_inprocess_scripts_use_venv_python: the only assertion is
guarded by if "/bin/sh" not in shebang:. distlib emits a /bin/sh wrapper when
the interpreter path contains a space or the shebang exceeds 127 chars. Then no
assertion runs, the test passes, and strict=True turns that XPASS into a
failure. It will not fire on pip's hosted runners (short, space-free paths) but is
fragile elsewhere. Assert against the wrapper's exec line instead of skipping, or
drop strict.

Minor

• venv.py __enter__ prepends the venv bin (which holds python/python3) to
  PATH, so a backend that shells out to a bare python gets the pip-less venv
  interpreter. Arguably intended ("as if activated"), and the legacy env already
  hides system site-packages, so this is debatable rather than wrong. Worth a doc
  note.
• test_build_env_can_still_access_python_tools_on_system_path has no skip guard
  and needs a downloadable cmake wheel for the runner arch.
• venv.py calls ensure_directories() then create() (which re-runs it).
  Harmless.

Checked, no issue

configure_backend/load_pyproject_toml split (no path reaches pep517_backend
while it is None); build_isolation == "off" uses NoOp.python_executable; the
abstract __init__ signature mismatch; env_exec_cmd exists on 3.9+; the
lib_path 3.12 guard; venv disables user-site so dropping PYTHONNOUSERSITE is
fine; an empty PYTHONPATH does not inject cwd; --target "" neutralizes
PIP_TARGET.

PR #14070 (venv build environment): issues found on Windows

Tested on Windows 11 / CPython 3.13.5, building sdists from stdlib venv, virtualenv,
uv, and conda base interpreters. Issues below.

1. inprocess-build-deps + venv-isolation: console-script build deps break on Windows

With both features on, a console-script build dep gets a Windows .exe launcher embedding
the parent pip's interpreter instead of the temp venv's. The inprocess installer writes
launchers with the running pip's interpreter (sys.executable); the subprocess installer runs
pip as the venv python, so only it targets the venv. The interpreter is baked into the .exe,
so PATH can't redirect it.

Repro:

• scripttool: a wheel with a console script (scripttool = "scripttool:main") whose main
  imports its own package.
• usestool: build-system.requires = ["setuptools>=61", "scripttool==1.0"]; its setup.py
  runs subprocess.run([shutil.which("scripttool")], check=True) (invoked during
  get_requires_for_build_wheel).
• pip wheel usestool --use-feature=venv-isolation --use-feature=inprocess-build-deps --find-links <dir with scripttool wheel> --no-deps

Result: ModuleNotFoundError: No module named 'scripttool' — the .exe runs the parent
interpreter, which lacks the build deps. Dropping --use-feature=inprocess-build-deps (the
default subprocess installer) builds it. The embedded interpreter is the #!"...python.exe"
line just before the PK\x03\x04 zip header in scripttool.exe: parent python under
inprocess, venv python under subprocess.

The PR description notes this limitation and the xfail covers it, but the xfail is skipped on
Windows as "Unix-only test" (test_build_env.py:414). The defect is present on Windows too;
only the shebang-reading technique doesn't apply to an .exe. The Windows skip should be
removed when the fix lands.

2. A failed python.exe copy during venv creation is not detected (Windows)

EnvBuilder.setup_python copies venvlauncher.exe to Scripts/python.exe inside a
try/except OSError that only logs a warning, so create() returns even when no interpreter
was produced. venv.py:64 then stores context.env_exec_cmd without checking it exists.

Repro (deterministic, no real AV needed):

import os, tempfile, shutil, venv
d = tempfile.mkdtemp()
b = venv.EnvBuilder(symlinks=False)
ctx = b.ensure_directories(d)
_orig = shutil.copy2
def blocked(src, dst, *a, **k):
    if os.path.basename(dst).lower() in ("python.exe", "pythonw.exe"):
        raise OSError(13, "blocked")          # emulate AV / lock on the launcher copy
    return _orig(src, dst, *a, **k)
shutil.copy2 = blocked
b.create(d)                                   # returns without raising
print(os.path.exists(ctx.env_exec_cmd))       # -> False

Real triggers: Defender / controlled-folder-access or a sharing violation on the new exe. The
legacy env has no copy step (uses sys.executable), so this is new here. Through pip it
surfaces as a later "file not found" rather than a build-env error; an os.path.exists check
after create() makes it clear (patch below).

3. A surviving child holding the venv python leaks the temp dir (Windows)

The build env now contains a real python.exe, so a grandchild that outlives the build and
holds it open blocks removal of the pip-build-env-* temp dir (Windows can't delete a running
exe). The legacy env runs sys.executable and leaves only text files, so it never hits this.

Repro:

import os, tempfile, shutil, subprocess, venv, time
d = tempfile.mkdtemp()
venv.EnvBuilder(symlinks=False, with_pip=False).create(d)
vpy = os.path.join(d, "Scripts", "python.exe")
p = subprocess.Popen([vpy, "-c", "import time; time.sleep(20)"])
time.sleep(1)
shutil.rmtree(d)        # PermissionError: [WinError 5] Access is denied

Triggered by backends that leave a daemon or test process running. Minor, but new on Windows.

4. Only import venv is guarded

Only import venv is wrapped (venv.py:27-30 → VenvImportError);
EnvBuilder.ensure_directories() / create() can raise OSError/ValueError that fall
through as a raw ERROR: Exception. Example: a ; in TEMP gives ValueError: Refusing to create a venv in <path> because it contains the PATH separator ;. Not a regression (the
legacy env also fails on ; in TEMP, later and differently), but the venv path exposes more
EnvBuilder failure modes without a diagnostic.

5. context.env_exec_cmd is read unconditionally and is missing on 3.10.0

venv.py:64 reads context.env_exec_cmd. That attribute (bpo-45337) is absent in v3.10.0 and
present from v3.10.1 (per the release tags). On 3.10.0, --use-feature=venv-isolation would
raise AttributeError before any build. getattr(context, "env_exec_cmd", context.env_exe)
covers it (patch below). Narrow, but free to guard.

Patch for issues 2 and 5

One spot in venv.py. Verified on Windows: normal build unchanged; the fallback yields a
working interpreter when env_exec_cmd is deleted; a blocked launcher copy now raises.

-        self.python_executable = context.env_exec_cmd
+        # env_exec_cmd was added in bpo-45337 (Python 3.10.1); fall back to env_exe
+        # on 3.10.0.
+        self.python_executable = getattr(context, "env_exec_cmd", context.env_exe)
+        # On Windows, venv.create() only logs a warning if it fails to copy the
+        # launcher to Scripts/python.exe, so confirm the interpreter exists before
+        # relying on it.
+        if not os.path.exists(self.python_executable):
+            raise VenvCreationError(self.python_executable)

VenvCreationError is a small DiagnosticPipError alongside VenvImportError.

6. lib_dirs is wrong for PyPy on POSIX (not Windows)

For Python < 3.12, venv.py:54-60 builds the site-packages path from python{X.Y}. PyPy's
POSIX layout is lib/pypy{X.Y}/site-packages (confirmed via PyPy 3.11's sysconfig), so
lib_dirs points at a nonexistent dir and check_requirements / --check-build-deps
silently find nothing. Windows PyPy uses Lib/site-packages, so it's unaffected. 3.12+ avoids
this via context.lib_path; deriving the < 3.12 path from sysconfig (as the code comment
suggests) would fix PyPy.

By inspection, not reproduced

The venv path drops --ignore-installed and shares one prefix (installer.py:117-121 adds it
only on the non-venv branch), so a dynamic build req can uninstall or upgrade a static one.
Matches the layering change in the PR description; worth a changelog line.