Visitar URL original
perf: write output files in parallel and create each directory once by Boshen · Pull Request #11680 · rolldown/rolldown · GitHub
Skip to content

perf: write output files in parallel and create each directory once - #11680

Open
Boshen wants to merge 2 commits into
mainfrom
perf/parallel-output-writes
Open

Boshen wants to merge 2 commits into
mainfrom
perf/parallel-output-writes

Conversation

@Boshen

@Boshen Boshen commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

closes #11670

Summary

bundle_write wrote every output file one at a time and probed (exists) the parent directory before each write. On a large app (~28k output files including maps) that loop is serial syscall latency on the critical path between generateBundle and writeBundle.

This PR:

  • validates all filenames for NUL bytes before writing anything (previously an invalid name could leave a partially written outDir)
  • creates each distinct parent directory once, instead of probing per file
  • writes files in parallel with rayon when all destination paths are distinct
  • falls back to sequential writes if two outputs target the same path, preserving last-writer-wins
  • reports the first failure in output order, so errors stay deterministic

writeBundle still runs only after all writes complete.

Measurements

Large synthetic Vite app (13k chunks, 16k assets, client build), M4 Pro, profile build, 2 paired runs, native tracing spans. The write loop plus writeBundle hooks (bundle_write minus generate):

run 1 run 2
before 12.86 s 13.42 s
after* 11.56 s 12.01 s

* measured together with #11681, #11682, #11683. Those don't touch this phase, apart from the lazy metadata change affecting writeBundle hook object construction. Whole-build wall time is too noisy (±5 s) to isolate this change alone.

Test plan

  • cargo test -p rolldown --lib
  • cargo test -p rolldown --test integration: no snapshot changes (only env failures: test262 submodule, missing util-deprecate)
  • cargo clippy -p rolldown --all-targets

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@netlify

netlify Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for rolldown-rs ready!

Name Link
🔨 Latest commit 7f98f92
🔍 Latest deploy log https://app.netlify.com/projects/rolldown-rs/deploys/6af6b4b834543e2154dbc0da
😎 Deploy Preview https://deploy-preview-11680--rolldown-rs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

🚀 Deploying Preview to Cloudflare 🚀

Preview URL: https://perf-parallel-output-writes-rolldown-rs.voidzero-docs.workers.dev (commit 7f98f92)

This URL reflects your latest Preview deployment

Preview Deployments by commit

Status Deployment URL Commit Updated (UTC) See this deployment's details
  • Build: Success ✅
  • Deployment: Success ✅

View logs ↗
https://5d1b3227-rolldown-rs.voidzero-docs.workers.dev 7f98f92 2026-11-13T06:58:45.718Z Visit the dashboard ↗
  • Build: Success ✅
  • Deployment: Success ✅

View logs ↗
https://88a5c0de-rolldown-rs.voidzero-docs.workers.dev 85c3e1c 2026-11-13T02:51:25.766Z Visit the dashboard ↗

@Boshen
Boshen marked this pull request as ready for review November 13, 2026 03:11
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-13T03:13:43.010403Z 85c3e1c Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 85c3e1c6e2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/rolldown/src/bundle/bundle.rs Resolved
@codspeed

codspeed Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

1 warning

⚠️ 4 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

✅ 7 untouched benchmarks
⏩ 10 skipped benchmarks1


Comparing perf/parallel-output-writes (7f98f92) with main (90c79f0)

Open in CodSpeed

Footnotes

  1. 10 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@shulaoda shulaoda self-assigned this Nov 13, 2026
@shulaoda

Copy link
Copy Markdown
Member

Maybe related to #11670, let me take a deep look

@hyfdev hyfdev left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Linux (tmpfs, 13k chunks + 16k assets in one assets/ dir, this PR alone), the write phase is 38 ms (16%) faster and uses 3.3x the CPU of main. Could you share a macOS number for this PR alone, since the one in the description also includes #11681–#11683?

Comment thread crates/rolldown/src/bundle/bundle.rs Outdated
HookBuildEndArgs, HookCloseBundleArgs, HookRenderErrorArgs, SharedPluginDriver,
};
use rolldown_utils::dashmap::FxDashSet;
use rolldown_utils::rayon::{IndexedParallelIterator, IntoParallelRefIterator, ParallelIterator};

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

browser, wasi and Node Validation fail with E0432: unresolved import rolldown_utils::rayon::IndexedParallelIterator, because the wasm shim does not export it (rayon.rs). Gating the import with #[cfg(not(target_family = "wasm"))], as prune_constant_branches.rs does, makes cargo check -p rolldown --target wasm32-wasip1-threads pass.

@hyfdev hyfdev self-assigned this Nov 13, 2026
The wasm rayon shim has no `IndexedParallelIterator`, so pair each chunk with its destination instead of zipping two parallel iterators.

`foo.txt` and `Foo.txt` are one file on case-insensitive file systems. Compare lowercased paths so they take the sequential path.
@shulaoda

Copy link
Copy Markdown
Member

I added a commit with two fixes:

  • Fix the wasm build. The wasm rayon shim has no IndexedParallelIterator.
  • Write foo.txt and Foo.txt sequentially. They are the same file on macOS and Windows, and writing them in parallel can mix their content.

@shulaoda

Copy link
Copy Markdown
Member

Writing 16,004 small files (600 B and 1.5 KB) into one directory, standalone program, median of 6 runs, speedup vs 1 thread in brackets:

Threads macOS APFS Linux ext4 Linux btrfs
1 (main) 957 ms 88 ms 137 ms
2 709 ms (1.35x) 69 ms (1.28x) 160 ms (0.86x)
4 515 ms (1.86x) 97 ms (0.91x) 172 ms (0.80x)
8 659 ms (1.45x) 134 ms (0.66x) 181 ms (0.76x)
All cores (this PR) 866 ms (1.11x) 186 ms (0.47x) 255 ms (0.54x)

bundle.write() on a synthetic app with 16,004 output files, interleaved A/B, median:

Run main this PR PR vs main generate phase vs main (same code)
1 1522 ms 1596 ms +4.9% +7.7%
2 2145 ms 1626 ms -24.2% -3.5%

Setup: M4 Pro (14 cores), macOS 27.0.1. Linux is an OrbStack VM (13 vCPUs) on the same Mac. The machine was busy during the runs, so the two bundle.write() runs disagree.


I will test again an update above data when my machine was not busy later

@hyfdev hyfdev left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On a 29k-file build on a Linux btrfs disk, 7f98f92's write phase is no faster than main and uses 8.8x its CPU.

The same writes in a 4-thread pool were 10% faster than main here.

Your Linux numbers show 4 threads slower than 1, so the best cap may depend on the platform.

Could your idle re-run compare a small fixed pool with the global one?

// `foo.js` and `Foo.js` are one file on case-insensitive file systems.
if destinations
.iter()
.all(|(_, dest)| seen.insert(PathBuf::from(dest.to_string_lossy().to_lowercase())))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lowercase key fixes the case-only pairs.

d/../c{i}.txt and c{i}.txt still race on 7f98f92: 203–432 of 500 files got the other content, and up to 2 mixed both.

Could the key apply dest.normalize() before lowercasing? That made all 5 runs last-writer-wins here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf: write output files in parallel in bundle.write()

3 participants