Visitar URL original
fix(build): make the runtime chunk reproducible across build paths by gauthierpetetin · Pull Request #46829 · MetaMask/metamask-extension · GitHub
Skip to content

fix(build): make the runtime chunk reproducible across build paths - #46829

Closed
gauthierpetetin wants to merge 1 commit into
mainfrom
fix/amo-runtime-chunk-determinism
Closed

gauthierpetetin wants to merge 1 commit into
mainfrom
fix/amo-runtime-chunk-determinism

Conversation

@gauthierpetetin

@gauthierpetetin gauthierpetetin commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Description

runtime.[contenthash].js could differ between rebuilds of identical sources, which fails the
byte-for-byte mtree comparison Firefox AMO reviewers run and has been blocking Firefox
submissions. The visible symptom was ~190 bytes of c↔l swaps in mangled identifiers; nothing
else in the bundle changed. There are two root causes, and this PR fixes one of them and makes
the symptom impossible regardless of either.

Root cause 1 — the absolute build path leaks into every module hash (fixed here).
swcLoader.ts handed webpack the SWC source map
as a JSON string. webpack only strips absolute paths from object maps
(NormalModule.contextifySourceMap passes strings through), and the map is then hashed into
buildInfo.hash verbatim. So the absolute sourceFileName reached 11,331 / 11,435 module hashes and
214 / 216 chunks' provisional [contenthash]. This is why prepare_release.sh (different path)
flaked while the regular CI build and bundle.sh (same path) never did.

Root cause 2 — SWC's process.env inlining cache carries stale positions (not in this PR).
jsc.transform.optimizer.globals.envs caches the parsed env values per process with the spans of
whichever file was transformed first, so later modules get one bogus source-map segment that
depends on thread timing. Already fixed upstream in @swc/core 1.16.2
(swc-project/swc#12129, #12166); we are on 1.13.3. This is the flake seen at a fixed path and
will be closed by the @swc/core bump.

Why either shows up as c↔l. The runtime chunk embeds the provisional hashes of all other
chunks as hex string literals. SWC orders its mangling alphabet by letter frequency over the chunk,
string literals included; c is a hex digit, l is not. On v13.47.1 c and l were adjacent
ranks with the flip at exactly 199 vs 200 cs in that table against a natural count of 201 ± 14
(≈45 % of paths). RealContentHashPlugin then rewrites the hashes, leaving the swapped identifiers
as the only trace.

Changes

  1. helpers.ts — mangle the runtime chunk with SWC's fixed alphabet
    (mangle: { disableCharFreq: true }) instead of not mangling it at all (fix(build): disable SWC mangling on the runtime chunk for AMO determinism #46236). Identifiers
    can then no longer depend on the chunk's string contents, so neither root cause can reach the
    output. Mangling is back on for the runtime chunk:

    runtime chunk raw gzipped
    mangle: true (original, flaky) 339,679 B 86,913 B
    mangle: false (main today, fix(build): disable SWC mangling on the runtime chunk for AMO determinism #46236) 352,356 B 88,869 B
    disableCharFreq (this PR) 339,679 B 86,944 B
  2. swcLoader.ts — hand webpack a parsed source-map object so it relativizes sources
    (root cause 1). Also makes the persistent build cache portable between checkouts.

  3. loaders.swcLoader.test.ts — the existing test asserted the string form; it now asserts
    the object contract and documents why.

Verification

  • CI experiment: https://github.com/MetaMask/metamask-extension/actions/runs/36839769618 —
    v13.47.1 (the release that flaked) built at 20 different absolute paths, untouched (control)
    and with this PR's changes applied as a patch (fixed). The Report job's summary tabulates
    the runtime chunk name and a digest of all of dist/firefox per build. Result: control
    produced 2 different runtime chunks (12 × 6197764d…, 8 × d269eac6…, i.e. a 40 % flip
    rate across path lengths 89–204); fixed produced 1 runtime chunk and 1 dist digest across all
    20 paths
    . (Hashes differ from the published release because the experiment uses placeholder
    API keys; both variants share them.) Workflow and patch live on
    test/amo-determinism-proof.
  • Local: same tree built at 43- and 115-character paths — all 663 emitted files byte-identical;
    runtime alphabet is the fixed abcdef…. On v13.47.1, disableCharFreq produced zero identifier
    swaps across the full range of c counts.
  • yarn test:unit:webpack 516/516; yarn webpack:tsc clean.

Full plain-language walkthrough with diagrams, kept for reference:
#46829 (comment)

Follow-ups

  • Bump @swc/core to ≥ 1.16.2 (root cause 2); bump/swc-core-amo-determinism already carries
    most of the fallout (IN_TEST inlining for npm loaders, LavaMoat policy updates — themselves a
    consequence of swc#12166 keying the env cache correctly).
  • Once both are in, disableCharFreq can optionally be dropped (+31 B gzipped to keep it).

Changelog

CHANGELOG entry: null

Manual testing steps

  1. Make two copies of this branch in directories whose names have clearly different lengths.
  2. In each: yarn install, then
    SOURCE_DATE_EPOCH=1700000000 ENABLE_MV3=false yarn webpack:lavamoat:build:mv2 --env production
  3. diff -r <copyA>/dist/firefox <copyB>/dist/firefox — verify no differences. On main before
    fix(build): disable SWC mangling on the runtime chunk for AMO determinism #46236, runtime.<hash>.js differed between the two roughly half the time.
  4. Load dist/firefox as a temporary add-on in Firefox (about:debugging), unlock or onboard,
    and verify the UI renders with no LavaDome / lockdown / module is undefined console errors
    (the runtime chunk hosts the SES/LavaMoat runtime and is mangled again by this change).

Pre-merge author checklist

Pre-merge reviewer checklist

  • I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed).
  • I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

🤖 Generated with Claude Code

`runtime.[contenthash].js` could differ between rebuilds of identical
sources, which breaks the `mtree` comparison Firefox AMO reviewers run and
has been blocking Firefox submissions.

SWC orders its base54 identifier alphabet by the character frequencies of
the chunk it is minifying, so anything that changes the chunk's *contents*
can reshuffle the alphabet and rename every short identifier. In the
runtime chunk `c` and `l` are near-tied, and that chunk embeds webpack's
provisional (pre-`RealContentHashPlugin`) chunk-filename hashes — hex, so
they contain `c` but never `l`. Those hashes varied with the absolute build
path because `swcLoader` returned its source map as a JSON string: webpack
rewrites `sources` relative to the build context only for *object* maps
(`NormalModule.contextifySourceMap` passes strings through), and the map is
then hashed into `buildInfo.hash` verbatim, so the absolute
`sourceFileName` reached every module's hash. `RealContentHashPlugin` later
rewrote the hashes, which is why ~190 swapped identifiers were the only
visible difference and the cause was invisible.

- Mangle the runtime chunk with SWC's fixed alphabet
  (`mangle.disableCharFreq`) rather than not mangling it at all, recovering
  the ~12.4 KB (~1.9 KB gzipped) that #46236 traded away.
- Hand webpack a parsed source map so absolute paths stay out of module
  hashes.

Verified by building the same tree at two paths of different lengths: all
663 emitted files are now byte-identical, where previously the runtime
chunk differed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

@github-actions github-actions Bot added the size-S label Oct 1, 2026
@metamask-ci

metamask-ci Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

✨ Files requiring CODEOWNER review ✨

👨‍🔧 @MetaMask/extension-platform (3 files, +50 -13)
  • 📁 development/
    • 📁 webpack/
      • 📁 test/
        • 📄 loaders.swcLoader.test.ts +8 -2
      • 📁 utils/
        • 📁 loaders/
          • 📄 swcLoader.ts +12 -1
          • 📄 helpers.ts +30 -10

👨‍🔧 @itsyoboieltr (3 files, +50 -13)
  • 📁 development/
    • 📁 webpack/
      • 📁 test/
        • 📄 loaders.swcLoader.test.ts +8 -2
      • 📁 utils/
        • 📁 loaders/
          • 📄 swcLoader.ts +12 -1
          • 📄 helpers.ts +30 -10

@metamask-ci metamask-ci Bot added the team-extension-platform Extension Platform team label Oct 1, 2026
@metamask-ci

metamask-ci Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor
Builds ready [949cd25]
⚡ Performance Benchmarks (Total: 🟢 10 pass · 🟡 1 warn · 🔴 2 fail)

Baseline (latest main): 171ed20 | Date: 7/28/2026 | Pipeline: 36819263976 | Baseline logs

Metricschrome-webpackfirefox-webpack
loadNewAccount
[Sentry log · main/release]
🔴 load_new_account(p95) [CI log]🔴 load_new_account(p95) [CI log]

Regressions (🔴 2 failures)

Interaction Benchmarks · Samples: 5 🔴 2
Benchmarkchrome-webpackfirefox-webpack
loadNewAccount
[Sentry log · main/release]
🔴 [CI log]
🔴 load_new_account
🔴 [CI log]
🔴 load_new_account
confirmTx
[Sentry log · main/release]
🟢 [CI log]🟢 [CI log]
bridgeUserActions
[Sentry log · main/release]
🟢 [CI log]🟢 [CI log]

📈 Results compared to the previous 5 runs on main

  • ↑ loadNewAccount/load_new_account: +414%
  • ↑ loadNewAccount/total: +414%
  • ↓ loadNewAccount/fcp: -15%
  • ↑ confirmTx/longTaskTotalDuration: +11%
  • ↑ confirmTx/longTaskMaxDuration: +28%
  • ↑ confirmTx/tbt: +30%
  • ↓ confirmTx/inp: -17%
  • ↓ confirmTx/fcp: -14%
  • ↑ bridgeUserActions/bridge_load_page: +18%
  • ↑ bridgeUserActions/bridge_load_asset_picker: +39%
  • ↑ bridgeUserActions/longTaskCount: +11%
  • ↓ bridgeUserActions/longTaskTotalDuration: -10%
  • ↓ bridgeUserActions/longTaskMaxDuration: -28%
  • ↓ bridgeUserActions/tbt: -52%
  • ↓ bridgeUserActions/fcp: -20%
  • ↓ bridgeUserActions/lcp: -16%
  • ↑ loadNewAccount/load_new_account: +416%
  • ↑ loadNewAccount/total: +416%
  • ↓ loadNewAccount/inp: -32%
  • ↓ loadNewAccount/fcp: -79%
  • ↑ loadNewAccount/lcp: +928%
  • ↓ confirmTx/longTaskCount: -100%
  • ↓ confirmTx/longTaskTotalDuration: -100%
  • ↓ confirmTx/longTaskMaxDuration: -100%
  • ↓ confirmTx/tbt: -100%
  • ↓ confirmTx/inp: -47%
  • ↓ confirmTx/fcp: -84%
  • ↑ confirmTx/lcp: +1091%
  • ↑ bridgeUserActions/bridge_load_page: +104%
  • ↑ bridgeUserActions/bridge_load_asset_picker: +65%
  • ↓ bridgeUserActions/bridge_search_token: -16%
  • ↓ bridgeUserActions/longTaskCount: -100%
  • ↓ bridgeUserActions/longTaskTotalDuration: -100%
  • ↓ bridgeUserActions/longTaskMaxDuration: -100%
  • ↓ bridgeUserActions/tbt: -100%
  • ↓ bridgeUserActions/inp: -15%
  • ↓ bridgeUserActions/fcp: -82%
  • ↑ bridgeUserActions/lcp: +1057%
Startup Benchmarks · Samples: 100
Benchmarkchrome-webpackfirefox-webpack
startupStandardHome
[Sentry log · main/release]
🟢 [CI log]🟢 [CI log]

📈 Results compared to the previous 5 runs on main

  • ↓ startupStandardHome/firstReactRender: -99%
  • ↓ startupStandardHome/initialActions: -33%
  • ↑ startupStandardHome/setupStore: +59%
  • ↑ startupStandardHome/numNetworkReqs: +10%
  • ↓ startupStandardHome/longTaskCount: -33%
  • ↓ startupStandardHome/domInteractive: -32%
  • ↓ startupStandardHome/firstReactRender: -98%
  • ↓ startupStandardHome/initialActions: -50%
  • ↑ startupStandardHome/setupStore: +163%
  • ↓ startupStandardHome/fcp: -27%
User Journey Benchmarks · Samples: 5 · mock API
Benchmarkchrome-webpackfirefox-webpack
onboardingImportWallet
[Sentry log · main/release]
🟢 [CI log]🟢 [CI log]
onboardingNewWallet
[Sentry log · main/release]
🟢 [CI log]🟡 [CI log]
🟡 total

📈 Results compared to the previous 5 runs on main

  • ↓ onboardingImportWallet/doneButtonToHomeScreen: -94%
  • ↓ onboardingImportWallet/openAccountMenuToAccountListLoaded: -75%
  • ↓ onboardingImportWallet/longTaskCount: -100%
  • ↓ onboardingImportWallet/longTaskTotalDuration: -100%
  • ↓ onboardingImportWallet/longTaskMaxDuration: -100%
  • ↓ onboardingImportWallet/tbt: -100%
  • ↓ onboardingImportWallet/total: -89%
  • ↓ onboardingNewWallet/srpButtonToPwForm: -20%
  • ↓ onboardingNewWallet/createPwToRecoveryScreen: -22%
  • ↓ onboardingNewWallet/skipBackupToMetricsScreen: -22%
  • ↑ onboardingNewWallet/agreeButtonToOnboardingSuccess: +21%
  • ↓ onboardingNewWallet/doneButtonToAssetList: -80%
  • ↓ onboardingNewWallet/longTaskCount: -69%
  • ↓ onboardingNewWallet/longTaskTotalDuration: -100%
  • ↓ onboardingNewWallet/longTaskMaxDuration: -52%
  • ↓ onboardingNewWallet/tbt: -100%
  • ↓ onboardingNewWallet/total: -77%
Dapp Page Load Benchmarks · Samples: 100
Benchmarkchrome-webpack
dappPageLoad
[Sentry log · main/release]
🟢 [CI log]

📈 Results compared to the previous 5 runs on main

  • ↓ dappPageLoad/pageLoadTime: -70%
  • ↓ dappPageLoad/domContentLoaded: -30%
  • ↓ dappPageLoad/firstPaint: -50%
  • ↓ dappPageLoad/firstContentfulPaint: -50%
Bundle Size Diffs [🚀 Bundle size reduced!]
Status Bundle Total Diff Change
✅ background 15.56 MiB +119 Bytes 0.00%
✅ ui 18.97 MiB -4.92 KiB -0.03%
✅ common 0 Bytes 0 Bytes 0.00%
✅ other 1.69 MiB 0 Bytes 0.00%
✅ content scripts 1.97 MiB -2 Bytes 0.00%
✅ zip 22.79 MiB -145.63 KiB -0.62%

@sonarqubecloud

sonarqubecloud Bot commented Oct 1, 2026

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
66.7% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud

@gauthierpetetin

Copy link
Copy Markdown
Contributor Author

Full explanation (reference)

The PR description is kept short; this comment preserves the complete plain-language walkthrough of the two root causes, with diagrams, for future readers.


1. The problem in one picture

Mozilla reviewers rebuild MetaMask from our source and compare their result to the .zip we
submitted. Every byte must match.

   We submit                    Reviewer rebuilds
   ─────────                    ─────────────────
   runtime.js      ────vs────   runtime.js      ← ✗ differs ~45% of the time
   background.js   ────vs────   background.js   ← ✓ always matches
   ui.js           ────vs────   ui.js           ← ✓ always matches
   ... 250 more files ...                       ← ✓ always matches

Exactly one file out of ~250 could differ, and only sometimes. That blocked Firefox releases.

2. What the difference actually was

190 bytes out of 790,000. Every single one was the letter c and the letter l trading
places
:

  ours:      let c=[];  ...  for(var c=1/0 ... l=!0 ... if(l){
  reviewer:  let l=[];  ...  for(var l=1/0 ... c=!0 ... if(c){
                  ↑                  ↑          ↑        ↑
            same code, same length — just two variable names swapped

That's it. The code is identical in behaviour. But different bytes = different checksum = the
reviewer's automated comparison fails.

3. Why c and l? The letter popularity contest

To make files smaller, our build renames every variable to the shortest possible name (e, t,
a…). This is done by a tool called SWC.

SWC doesn't hand out letters alphabetically. It runs a popularity contest: it counts how
often each letter appears in the file, then gives the most-used variable the most popular letter.
(Reusing popular letters compresses better when the file is gzipped.)

For our runtime.js, the ranking comes out like this:

  rank:    1   2   3   4   5   6   7     8   9     10  11  12
  letter:  e   t   r   a   s   o   n  │  c   l  │  i   m   p
  count: 20013 ... ... ... ... ... 7615│7194 7269│ ...
                                       └────────┘
                                   7194 vs 7269 — essentially TIED

c and l are neck and neck. A difference of one single character anywhere in the file can
flip which one wins
— and then every variable that was going to get rank 8 gets l instead of
c, and vice versa. 190 swaps.

Every other file in the build has a comfortable margin between its 8th and 9th letters, which is
why only runtime.js was ever affected.

4. So what changes the letter count? Temporary filenames

runtime.js is the file that knows where all the other files live. It contains a lookup table of
filenames, and those filenames contain content hashes written in hex:

// inside runtime.js — a table of ~160 filenames
{ "46":"e23a6c9490339b890118", "110":"682055cc237ca3ec5279", "133":"bedf0a792fcf955c01a1", ... }
       └─ hex: 0 1 2 3 4 5 6 7 8 9 a b c d e f ─┘
                                    ↑
                          hex contains 'c'  …but never 'l'

That's ~3,200 hex characters, holding roughly 200 cs and zero ls.

So the c-vs-l contest is decided almost entirely by how many cs happen to land in those
random-looking hashes
:

  number of 'c' in the table      who wins rank 8      resulting file
  ──────────────────────────      ───────────────      ─────────────────
         ≤ 199                          l              runtime.1fcc8942….js   ✗
         ≥ 200                          c              runtime.b574af00….js   ✓ (what we shipped)

  A random build produces:  201 ± 14  ← the threshold sits right in the middle
                                         → a ~45% / 55% coin flip

This is the "flakiness." It was never random corruption — it was a coin flip with a razor-thin
margin.

5. And why did the hashes change for the reviewer? The folder name

The last piece. Those hashes are computed from each source file's content plus its source-map
metadata
— and our build accidentally puts the full absolute path into that metadata.

  We build in:        /home/runner/work/metamask-extension/
  Reviewer builds in: /output/comparison_v13.47.1/metamask-extension-13.47.1/
                      └──────────────── different text ────────────────┘
                                          ↓
                      every file's hash changes → the hex table changes
                                          ↓
                      the 'c' count changes → coin flip → c/l may swap

This is why our earlier experiments behaved so strangely: renaming the build folder changed the
outcome, but it wasn't about the folder being longer — any different folder name reshuffles the
hashes and re-tosses the coin.

Technical detail, for the curious: webpack normally strips absolute paths out of this
metadata, but only when a plugin hands it a structured object. Ours hands it a plain text
string, which webpack passes through untouched. One-line bug, 11,000 affected files.

6. Why it was so hard to find: the evidence erases itself

The hex table is only temporary. After the renaming step, webpack replaces those placeholder
hashes with the real final ones — which are identical for everyone, because every other file
matches.

  Build step                          Is the hex table path-dependent?
  ──────────────────────────────      ────────────────────────────────
  1. compile all files                 —
  2. compute temporary hashes          ← path leaks in here
  3. SWC renames variables             ← decision made here, using the temp hashes  ⚠
  4. replace temp hashes with real     ← evidence wiped clean here
  5. write the .zip                    → table looks identical; only the
                                         variable names still differ

So if you looked at the two final files, you'd see only c↔l swaps and no reason for them.
The cause had already been overwritten. That's the whole mystery.

7. The full chain, end to end

  ┌─────────────────────────────────────────────────────────────────────┐
  │  Different build folder                                            │
  │  (or just a different build, see note below)                        │
  └────────────────────────────┬────────────────────────────────────────┘
                               ▼
  ┌─────────────────────────────────────────────────────────────────────┐
  │  Absolute path gets baked into every file's hash                    │
  └────────────────────────────┬────────────────────────────────────────┘
                               ▼
  ┌─────────────────────────────────────────────────────────────────────┐
  │  runtime.js's table of hex filenames changes                        │
  │  → the number of letter 'c' moves by a few                          │
  └────────────────────────────┬────────────────────────────────────────┘
                               ▼
  ┌─────────────────────────────────────────────────────────────────────┐
  │  SWC's letter popularity contest flips:  c ⇄ l                      │
  │  → 190 variable names swap                                          │
  └────────────────────────────┬────────────────────────────────────────┘
                               ▼
  ┌─────────────────────────────────────────────────────────────────────┐
  │  Temp hashes replaced by real ones — cause erased, swap remains      │
  └────────────────────────────┬────────────────────────────────────────┘
                               ▼
  ┌─────────────────────────────────────────────────────────────────────┐
  │  runtime.js has a different checksum → Mozilla's comparison fails   │
  └─────────────────────────────────────────────────────────────────────┘

Note — the second, smaller cause. Even in the same folder, roughly 1–2% of files get
slightly different metadata on each build. SWC keeps a reusable cache of our process.env
values, and entries in that cache carry stale position markers from whichever file happened to
be compiled first — which depends on thread timing. Harmless in itself, but it nudges the c
count by a unit or two, which is enough to flip the coin when it's already balanced on edge.
This is why the reviewer script failed ~2 times in 10 even at a fixed path.

8. The fix

Two independent things, ranked:

① Stop the contest (recommended, one line). Tell SWC to hand out letters in fixed alphabetical
order instead of by popularity — mangle: { disableCharFreq: true } on runtime.js. Then nothing
in the file's content can influence variable names, so neither cause above can reach us.

  Verified: 0 name swaps across the entire range of 'c' counts; identical file size.

This replaces the current workaround (PR #46236, which turned renaming off entirely for that one
file and cost ~12–16 KB).

② Stop the path leak (good hygiene, one line). Hand webpack a structured object instead of a
string, so it strips the absolute path as intended.

  Verified: path-dependent files drop from 11,331 → 203.
  Not sufficient on its own — the 203 are the cache-timing cause from the note above.

Plus two upstream reports worth filing: SWC's stale cache positions, and webpack's
string-vs-object inconsistency.


Recap in three sentences

Our runtime.js contains a list of temporary filenames written in hex. The minifier picks short
variable names by counting letters in the file, and hex digits tip a dead-even tie between c and
l — so anything that changes those temporary filenames (notably the build folder's name) swaps
190 variable names. A later build step rewrites the temporary filenames with identical final ones,
which is why the two files looked inexplicably different with no visible cause.

@gauthierpetetin

gauthierpetetin commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor Author

Superseded by #46845, which goes straight to the end state: it fixes the root causes (swc loader source map, @swc/core 1.16.2, html-bundler absolute requests, React Compiler stats in the hashed buildMeta) and reverts the #46236 workaround instead of adding disableCharFreq. The explanation comment above stays as the reference for causes 1–2; causes 3–4 are described in #46845.

@github-actions github-actions Bot locked and limited conversation to collaborators Oct 1, 2026

This branch was successfully deployed

1 active deployment
pr-comment — 949cd25d Deployed Oct 1, 2026 by gauthierpetetin via Publish prerelease / Publish prerelease #167716
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

size-S team-extension-platform Extension Platform team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant