Visitar URL original
[Bench] Fixes for pure Fizz bench by gaearon · Pull Request #96771 · vercel/next.js · GitHub
Skip to content

[Bench] Fixes for pure Fizz bench - #96771

Merged
gaearon merged 3 commits into
canaryfrom
fix/sandbox-bench-profile-order
Aug 5, 2026
Merged

gaearon merged 3 commits into
canaryfrom
fix/sandbox-bench-profile-order

Conversation

@gaearon

@gaearon gaearon commented Aug 5, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Harness fixes, all surfaced while benching a Fizz-only React PR (react/react#37022):

  • Profile pass now alternates arm order by VM index. The timed runs are ABBA, but the CPU profile pass ran base-then-cand on every VM. The second arm runs warmer (page cache, governor), which deflated every function in the cand profile by a uniform few percent — untouched Flight frames "improved" ~7%. With balanced order the same untouched frames diff to ~1.5% across 14 paired VMs while real movers keep their magnitude. Each VM records its capture order in the profiles tarball (prof-order.txt) so analyses can check order drift directly.
  • The fingerprint now covers every server-side React layer. FP_FILES hashed only the vendored Flight server files, which a Fizz-only React change leaves byte-identical — both arms of a valid A/B reported the same fingerprint. Added the vendored Fizz bundle and the shared react-server runtime (hooks/cache) in both suites. The VM boot loop also warns loudly when arms fingerprint identically, instead of leaving it for the analysis footer.
  • --prepare writes a terminal status phase. It previously exited leaving phase: "measuring", so bench-status prescribed collecting measurement data that never existed. bench-status now reports prepare-only runs as complete rather than "DEAD — relaunch".

Verification

  • Full 16-VM e2e run with the ordering + Fizz-fingerprint changes (run-pr37022-e2e-prof2): arms fingerprint distinctly (18ea275e vs f9805be5), profile order alternates per VM, untouched hot frames diff to ~zero in the cross-VM profile aggregate.
  • bench-status.mjs against a real --prepare run dir reports complete (--prepare: caches only).
  • --dry-run smoke of the modified launcher; node --check on all three scripts.

@github-actions

github-actions Bot commented Aug 5, 2026 •

Copy link
Copy Markdown
Contributor

Tests Passed

Commit: cf8a1ce

@github-actions

github-actions Bot commented Aug 5, 2026 •

Copy link
Copy Markdown
Contributor

Stats skipped

Commit: cf8a1ce
View workflow run

@gaearon gaearon changed the title sandbox-bench: balance profile-pass arm order, cover Fizz in fingerprint [Bench] Fixes for pure Fizz bench Aug 5, 2026
@gaearon
gaearon marked this pull request as ready for review August 5, 2026 22:46
@gaearon
gaearon requested a review from timneutkens August 5, 2026 22:46
Comment thread .agents/skills/sandbox-bench/scripts/sandbox-e2e.mjs Outdated
@gaearon
gaearon enabled auto-merge (squash) August 5, 2026 23:00
@gaearon
gaearon merged commit adcfa7f into canary Aug 5, 2026
71 checks passed
@gaearon
gaearon deleted the fix/sandbox-bench-profile-order branch August 5, 2026 23:03
m-kawafuji pushed a commit to m-kawafuji/next.js that referenced this pull request Aug 8, 2026
## Summary

Harness fixes, all surfaced while benching a Fizz-only React PR
(react/react#37022):

- **Profile pass now alternates arm order by VM index.** The timed runs
are ABBA, but the CPU profile pass ran base-then-cand on every VM. The
second arm runs warmer (page cache, governor), which deflated every
function in the cand profile by a uniform few percent — untouched Flight
frames "improved" ~7%. With balanced order the same untouched frames
diff to ~1.5% across 14 paired VMs while real movers keep their
magnitude. Each VM records its capture order in the profiles tarball
(`prof-order.txt`) so analyses can check order drift directly.
- **The fingerprint now covers every server-side React layer.**
`FP_FILES` hashed only the vendored Flight server files, which a
Fizz-only React change leaves byte-identical — both arms of a valid A/B
reported the same fingerprint. Added the vendored Fizz bundle and the
shared react-server runtime (hooks/cache) in both suites. The VM boot
loop also warns loudly when arms fingerprint identically, instead of
leaving it for the analysis footer.
- **`--prepare` writes a terminal status phase.** It previously exited
leaving `phase: "measuring"`, so `bench-status` prescribed collecting
measurement data that never existed. `bench-status` now reports
prepare-only runs as complete rather than "DEAD — relaunch".

## Verification

- Full 16-VM e2e run with the ordering + Fizz-fingerprint changes
(`run-pr37022-e2e-prof2`): arms fingerprint distinctly (`18ea275e` vs
`f9805be5`), profile order alternates per VM, untouched hot frames diff
to ~zero in the cross-VM profile aggregate.
- `bench-status.mjs` against a real `--prepare` run dir reports
`complete (--prepare: caches only)`.
- `--dry-run` smoke of the modified launcher; `node --check` on all
three scripts.

<!-- NEXT_JS_LLM -->
@github-actions github-actions Bot added the locked label Sep 5, 2026
@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Sep 5, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants