pnpm 12 cold fetch speed regression on macOS, possibly due to download batching #17271

{
  "name": "cold-fetch-repro",
  "private": true,
  "dependencies": {
    "next": "16.4.0",
    "react": "19.3.0",
    "react-dom": "19.3.0"
  },
  "devDependencies": {
    "typescript": "5.9.3",
    "eslint": "9.39.5",
    "vite": "6.4.4",
    "vitest": "4.1.11",
    "@babel/core": "7.29.6",
    "prisma": "7.9.1",
    "jest": "30.2.0",
    "webpack": "5.102.1",
    "@aws-sdk/client-s3": "3.913.0",
    "@mui/material": "7.3.4",
    "@emotion/react": "11.14.0",
    "@emotion/styled": "11.14.1",
    "firebase": "12.4.0",
    "@storybook/react-vite": "9.1.13",
    "@angular/cli": "20.3.7"
  }
}

strictDepBuilds: false only keeps fetch from exiting non-zero on the unapproved build scripts:

strictDepBuilds: false

• In the original runs above, only pnpm 12 on macOS prints Tarball download average speed … is below 50 KiB/s warnings (16–23 per run, all on tarballs of 5–40 KiB).
• Both v11 (fetch.ts:59–85) and v12 (fetch.rs:25–28) set virtualStoreOnly/virtual_store_only and enableModulesDir/enable_modules_dir to true, so pnpm fetch populates node_modules/.pnpm in addition to the CAS. In those original runs, default macOS fetch on 12 wrote 43,707 files to <store>/v11/files plus 53,895 into <store>/v11/links/…, while Linux writes only the 43,701 files. With packageImportMethod: hardlink in pnpm-workspace.yaml, macOS writes no links/ files, and fetch still takes 25 s.
• pnpm_config_network_concurrency=8 and =256 give the same total time (about 22 s); only the number of slow-download warnings changes.
• On a full install of a larger project, the import step was faster on 12 than on 11 (about 2.3 s vs 4.8 s). Warm-store installs on the same Mac were about 2.2x faster on 12.

• cold.rs:140–160 collects download futures directly into a FuturesUnordered rather than independent tasks, and cold.rs:241–289 (drain_cold_downloads) calls link_cold_chunk synchronously every 32 completions via slot_linking.rs:109–175 and lib.rs:140–148 (tokio::task::block_in_place).
• Tokio's block_in_place lets other tasks run, but cannot poll sibling futures inside the blocked task's FuturesUnordered. In-flight response-body readers (fetch.rs:225–265, body.rs:262–320) sit unpolled during each import chunk, and the speed warning in download.rs:347–350,396–403 does not subtract unpolled time. Meanwhile, Rayon parallel imports across files and package slots (placement.rs:31–68, slot_linking.rs:160–169, vs v11's 4-package worker limit in worker/src/index.ts:267–294) perform substantial filesystem work on APFS (a 3 s sample of the hardlink run recorded 14,698 top-of-stack observations in linkat vs 1,057 in __open and 435 in write; RAYON_NUM_THREADS=4 still took 28.230 s with 17.633 s in imports and 43 warnings). The sample counts are accumulated thread observations, not syscall counts or wall-clock seconds; they do not directly measure kernel lock contention.

Normal fetch: 1,061 packages. In a traced 12.13.0 hardlink fetch (28.532 s total, 12 warnings; default clone 24.154 s with 12.031 s in imports), cold import passes occupied 19.331 s (including a 3,997 ms final 5-package pass). Warning bursts fired immediately after multi-second import passes: 6 warnings within 100 ms (first two at +1 ms) after a 1,585 ms pass, and 3 within 100 ms (first at +0 ms, e.g. 15 KiB @types/babel__traverse at 8 KiB/s and 18 KiB @sigstore/core at 10 KiB/s) after a 1,775 ms pass.

Skipping slot imports (nodeLinker: hoisted fetch, which skips linking in cold.rs:266–268) fetched the same 1,061 packages in 12.876 s and 14.450 s with 0 warnings (install --frozen-lockfile with enableModulesDir: false and enableGlobalVirtualStore: false took 15.630 s on 12.13.0 vs 18.645 s on 11.18.0, also 0 warnings; these install controls recorded 1,068 vs 1,061 fetched events, so they are not an exact package-for-package timing comparison).

In drain_cold_downloads, suppress the mid-drain ready.len() >= COLD_LINK_CHUNK flush while keeping the final link_cold_chunk pass, and compare under TRACE='pacquet::install::phase=info,pacquet::download=info' TRACE_FORMAT=json with --reporter ndjson.

Note that merely replacing block_in_place with spawn_blocking(...).await at the same drain-loop call site still leaves FuturesUnordered unpolled while awaiting the chunk: downloads must be polled concurrently with (or on an independent task from) a sequential stream of import chunks (preserving chunk order because GVS peer variants can share a slot). No patched binary has been tested; these observations support the hypothesis but do not establish how much of the total slowdown a scheduling fix would remove.

#16782 attributed a cold-store install slowdown on macOS with Microsoft Defender to populating the <store>/v11/links/… directory-clone cache and recommended packageImportMethod: hardlink. Here no third-party endpoint security is running, and packageImportMethod: hardlink skips <store>/v11/links/… entirely yet still takes 25 s (28.5 s in a traced run; 29.382 s on 12.14.0): hardlink removes the clone-cache write, but not the node_modules/.pnpm import cost on APFS or its effect on download scheduling.