Repository navigation
Conversation
…ths in the patch-id scan
## The bug
The patch-id squash-merge fallback counted every commit in
`merge-base..target` and bailed when there were more than
`PATCH_ID_SCAN_MAX_COMMITS` (500). On a fast-moving default branch that
count passes 500 within days, so a squash-merged branch whose files were
edited again after the merge (the only case that reaches this fallback)
stops being detected as integrated, and `wt step prune` / `wt remove`
keep it.
Seen on a GitOps repo where a bot commits image bumps to `main`
continuously: branches squash-merged days or weeks earlier sat 600–5,600 commits
behind `main`, every one was over the cap, and none were pruned. Their
squash commits' patch-ids matched the branches' exactly.
## The fix
A target commit can only share the branch's patch-id if it changes
exactly the branch's files. So the scan now lists only the target
commits that touch one of the branch's paths, and the cap applies to
that list:
git --literal-pathspecs rev-list --full-history --stdin
stdin: <merge-base>..<target>, then `--`, then the branch's paths
- The paths come from `diff-tree -r -z --name-only`, the same plumbing
and rename handling as the branch-side patch.
- `--stdin` keeps a branch touching thousands of files off the argv; a
new `Repository::run_command_with_stdin` runs it.
- `--literal-pathspecs` stops `*`, `?`, `[` or a leading `:` in a
filename from acting as pathspec magic.
- `--full-history` stops history simplification from dropping a commit
that reached the target through the side of a merge.
- A path containing a line break can't be written one per line, so that
branch falls back to the whole range.
- `diff-tree --stdin -p` still gets no pathspec: each candidate's patch
must cover its whole change, or a commit that also touched other files
would falsely match.
The filter only drops commits that could never match, so it adds no
false positives. The path-limited walk compares trees but builds no
patches: over the full 26,200-commit history of the repo above it takes
0.17–0.37s, against 0.05s for the old `rev-list --count` pre-flight. On
that repo, 19 of the 27 branches whose merged PR head equals the local
branch tip are now detected (0 before), with no false positives.
The other 8 merged branches edit files the bot re-touched over 1,000
times since, so they still exceed the cap.
## Tests
The `patch_id_tests` fixture's pads were empty commits, which the path
filter now drops, so it gains a `Padding` kind and a path:
- `bails_when_commits_touching_branch_paths_exceed_cap`: pads touching
the branch's file still hit the cap.
- `detects_squash_when_only_other_paths_exceed_cap`: the same count of
pads on another file no longer hides the squash (the bug).
- `branch_paths_are_literal_not_pathspec_magic`: a file named `*`.
- `rejects_target_commit_with_extra_changes`: a target commit making the
branch's change plus an unrelated one doesn't match.
Claude-Session: https://claude.ai/code/session_01XwVSv9XpxotCADuKxRyUbA
…stead of giving up ## The problem With the path filter, the patch-id scan still gave up entirely when more than `PATCH_ID_SCAN_MAX_COMMITS` target commits touch the branch's paths. That happens when a branch edits a hot file: on the GitOps repo from the previous commit, 8 squash-merged branches edit values files a bot re-touched 1,079–1,669 times since, so they were reported as not integrated. ## The fix List the candidates oldest first (`rev-list --reverse`) and diff the first `PATCH_ID_SCAN_MAX_COMMITS` of them instead of none. The cut is made in Rust: `--max-count` applies before `--reverse`, so it would keep the newest. A squash merge lands on the target soon after the branch forks, and updating the branch before merging (a rebase, a merge from the target) moves the merge-base up to just before it. What piles up behind it is the target's later edits to the same files. For all 8 branches above, the squash commit was the oldest candidate. Nothing changes up to the cap. Past it, a check now diffs `PATCH_ID_SCAN_MAX_COMMITS` patches where it used to diff none, which is the cost the cap already allowed for a branch just under it. Every match is still an exact patch-id match, so it adds no false positives; it only finds squash merges the old bail skipped. On that repo, all 27 branches whose merged PR head equals the local branch tip are now detected, and the 4 remaining branches are genuinely unmerged (2 closed PRs, 1 without a PR, 1 whose tip differs from the merged head). Still missed: a squash that landed after more than `PATCH_ID_SCAN_MAX_COMMITS` other commits to the branch's files, counted from the merge-base. That takes a branch left un-updated while the target edited its files hundreds of times, then squash-merged as is; by then the target has usually changed the patch's context lines too, so no patch-id would match anyway. ## Tests The fixture now takes padding before and after the squash: - `detects_squash_among_oldest_candidates_past_cap` (was `bails_when_commits_touching_branch_paths_exceed_cap`): 500 same-file pads after the squash no longer hide it. - `misses_squash_past_oldest_candidates`: 500 same-file pads before the squash push it past the cap. - `detects_squash_behind_commits_to_other_paths` (was `detects_squash_when_only_other_paths_exceed_cap`): the other-file pads move before the squash, since after it an unfiltered oldest-first scan would find it too and the test would stop pinning the path filter. - `branch_paths_are_literal_not_pathspec_magic`: likewise moved before. Claude-Session: https://claude.ai/code/session_01XwVSv9XpxotCADuKxRyUbA
…free Review follow-ups on the path-limited patch-id scan. - The branch's paths are now read as raw bytes (`run_command_output`) and passed to `rev-list --stdin` unchanged. `run_command` decodes stdout lossily, so a non-UTF-8 filename reached the pathspec as U+FFFD, matched nothing, and filtered the squash commit out — a regression against the unfiltered scan. The comment claiming "any filename survives" is now true. - `rev-list` gains `--no-merges`. `diff-tree` emits no patch for a merge and a squash merge has one parent, so every merge kept by `--full-history` only took a slot under the cap. - `branch_paths_are_literal_not_pathspec_magic` uses `[o]the[r]` instead of `*`, which git on Windows rejects as a path (`fatal: invalid path '*'`). As a pathspec the bracket expressions still match `other`; `typos` flags `[o]ther`. New tests: - `non_utf8_branch_paths_still_filter`: a Latin-1 `café.txt` behind `PATCH_ID_SCAN_MAX_COMMITS` pads on another file. Fails with a lossy decode, and would fail with a fall-back-to-whole-range fix too. - `branch_path_with_line_break_scans_whole_range`: fails without the line-break guard, which would split the path into two pathspecs. - `branch_that_changes_nothing_is_not_squash_merged`: covers the early return. The first two are Unix-only: Git for Windows refuses control characters in paths, and a Windows checkout can't hold a non-UTF-8 name. Claude-Session: https://claude.ai/code/session_01XwVSv9XpxotCADuKxRyUbA
…alone The `typos` pre-commit hook rewrote the fixture's `caf\351.txt` to `calf\351.txt`, in both the literal and the `build` doc comment, so the test comment's `café.txt` named a file the test never built. Use Latin-1 `été.txt` (`\351t\351.txt`) instead, which typos doesn't touch, and spell the literal out in the comment. Claude-Session: https://claude.ai/code/session_01XwVSv9XpxotCADuKxRyUbA
…h-limited-scan # Conflicts: # src/git/repository/mod.rs
… path The path-limited `rev-list` had no bound: it walked all of `merge-base..target` with every branch path as a pathspec, so its cost grew with range × path count. On git.git over a 10,000-commit range (no commit-graph, same machine load for each), one path took 1.5 s, 30 paths 16 s, and 1,659 paths 182 s. It reran every time the target moved, which on a fast-moving repo is constantly. - Filter on one path. `git patch-id` hashes each file's name with its change, so a commit with the branch's patch-id touches every path the branch changed, and any one is a sound filter. Prefer a file the branch added: the target can't touch it before the squash, so the squash is its oldest candidate. A path with a line break is never chosen; a branch whose every path has one falls back to the range. - Bring back the `rev-list --count` pre-flight, at 10,000 commits (`PATCH_ID_SCAN_MAX_RANGE`) instead of 500. It is graph-only and stays around a tenth of a second at that size. The GitOps repo that prompted this PR had branches 600–5,600 commits behind, all inside it. - `PATCH_ID_SCAN_MAX_COMMITS` becomes `PATCH_ID_SCAN_MAX_CANDIDATES`, since there are now two caps. - The name-status listing goes through `PlumbingDiff::Tree`, so `submodule.<name>.ignore=all` no longer hides a gitlink path from it, and reads raw bytes through `run_command_bytes`. - `wt remove`'s help states both caps as behavior, not internals. Tests: `filters_on_a_path_the_branch_added`, `detects_squash_at_range_cap` and `skips_scan_past_range_cap`. Each fails against the mutation it pins (no added-path preference, no pre-flight, `>=` for `>`) and only it.
The patch-id result was cached only per (branch SHA, target SHA), so a stale conflicting branch paid the whole scan again every time the target moved: on a clone of this repo with a branch 2,000 commits behind that conflicts on `src/cli/mod.rs` (228 candidates), about 1.15 s per `wt list` after each new commit on main. - A new `patch-id-scan` cache kind, keyed on (branch, merge-base, cap), records the target the scan reached, how many candidates it diffed, and whether one matched. When that target is an ancestor of the current one, the next check scans only `scanned_target..target` against what is left of the 500-candidate cap. A match, or a miss that used up the cap, answers before any walk, the range pre-flight included. Any other cached target means a fresh scan. On the same clone, `wt list` after a target move drops to 0.31 s. - When no target commit touches the filter path, return before the branch-side diff and the patch-id pipeline. - A `rev-list --count` output that doesn't parse is an error rather than 0, which would have lifted the range cap. A resumed scan counts earlier candidates first, where a fresh scan orders by commit date, so the two can disagree on which candidates fall inside the cap after a merged-in side branch. That moves how far back the check reaches, never whether a match is real. The sha_cache spec and the list caching table now describe the kind as progress rather than a pure SHA-pair result. Tests: `no_candidates_records_scan_without_diffing`, `resumes_scan_as_target_moves_forward`, `exhausted_scan_stays_negative_as_target_moves`, `rescans_when_cached_target_is_not_an_ancestor` and `cached_match_holds_past_range_cap`.
…h-limited-scan # Conflicts: # src/git/repository/mod.rs
Each patch-id candidate became a full `diff-tree -p` patch, and the branch side a second pipeline, so a conflicting branch cost seven git processes and every candidate's whole patch. In a monorepo where target commits touching the filter path also touch many other files, that was most of the scan. `git patch-id` hashes every line of a file's patch header except `index`, including each file's `diff --git` line and its mode lines. So one `diff-tree --stdin -r -z --raw` over the candidates, which diffs trees but reads no blobs, settles most of them: - A candidate whose changed paths aren't exactly the branch's can't match, however small the extra change (a mode flip or an empty file included). - A candidate whose every path has the branch's modes and blob ids produces the branch's patch byte for byte, so it matches. - Only the rest go through `diff-tree -p | patch-id`, in one pass with the branch's own diff, given as `<branch> <merge-base>` on stdin. The branch side's `--name-status` listing becomes `--raw --no-abbrev` in the same process, so the common squash costs three processes and no patches. Answers are the PR's: on clones of a GitOps repo and a monorepo, `wt step prune --dry-run` lists the same branches as before, and `wt list` marks the same ones integrated. `run_command_with_stdin` becomes `run_command_bytes_with_stdin`, since the raw output carries paths as bytes. Tests: `detects_squash_onto_a_file_edited_since_the_fork` and `rejects_different_change_to_the_same_paths` reach the patch-id pass, as `detects_squash_merge_under_nondefault_diff_config` now does too: its target edits a distant line before the squash, which otherwise matches by blob id and never builds a patch. `parses_raw_diff_with_and_without_commit_ids` and `classifies_candidates_by_raw_changes` cover the parsing and the rules.
…-prefilter # Conflicts: # src/git/repository/integration.rs
worktrunk-bot
left a comment
There was a problem hiding this comment.
Holding this for a maintainer decision rather than approving. This change widens which branches the integration check treats as squash-merged. That check is what lets wt remove and wt step prune delete a branch without -D. By the PR's own numbers, prune would remove 122 branches on the monorepo instead of 103, and 42 instead of 13 on the GitOps repo. The repo's review guidance (.claude/skills/running-tend/references/review-pr.md, Data-Loss Surface: Hold for Human Review) reserves changes that loosen this gate for a human.
The decision is narrower than the diff size suggests. None of the new steps loosens what counts as a match. A blob-id match requires identical paths, modes and old/new oids, so the patches are byte-identical. The path filter and both caps only remove candidates. A cached match is only reused when its target is an ancestor of the current one. The extra branches come from extending the existing patch-id rule from 500 to 10,000 commits back. So what needs deciding is whether patch-id --verbatim equality is trustworthy enough to auto-delete branches that much older. I found no correctness issues in the implementation, and the 23 patch_id_tests pass locally.
@max-sixty, requesting your review for that call.
|
@max-sixty one follow-up to the previous PR. This takes another attempt. In this case this basically trades complexity (more lines of code) for what is from my benchmarks on par or sometimes faster than the current |
|
nice @mbark !! @worktrunk-bot are there any cases that are significantly worse? fine if there are some cases, particularly cold caches, that are 50% worse. But does anything get 3x or 10x worse? |
|
Yes, there's one case, and per branch it's far worse than 10x. It's a branch that conflicts with the target, is between 500 and 10,000 commits behind, and changes only a file the target also edits on its own, such as a lockfile. A stale Dependabot-style lockfile bump is the realistic example. On main, that branch costs a single I timed the PR's git sequence directly against main's on synthetic repos, without building either binary:
That puts this one branch at roughly 20x–800x, and a few seconds in absolute terms on a big lockfile. A repo with several such branches pays that cost for each of them on a cold Some context on how bad that is:
If you want to close off that one case, a lower cap on just the patch-id step would do it, for example diffing at most 50 undecided candidates rather than 500. The blob-id step already settles the common squash without any patches. Only candidates with the same paths but different blobs reach ReproductionThe setup has an initial commit with a 30,000-line MB=$(git merge-base main feature)
git rev-list --count $MB..main # main stops here (>500)
git diff-tree -r -z --raw --no-abbrev $MB feature --
C=$(printf "$MB..main\n--\npackage-lock.json\n" | git --literal-pathspecs rev-list --full-history --no-merges --reverse --stdin | head -500)
echo "$C" | git diff-tree --stdin -r -z --raw --no-abbrev
( echo "feature $MB"; echo "$C" ) | git diff-tree --stdin --patch | git patch-id --verbatimBoth runs gave 2.49–2.50 s against 3 ms. The 3,000-line variant used 3,000 target commits. |
|
This is once again really up to you @max-sixty , it is back to the balance of speed and accuracy. Note The text below is quoting the clanker. @max-sixty, to answer the 3x/10x question directly: yes. One kind of branch gets more than 10x slower, on a cold cache, and the slowdown happens about once per branch. I reproduced the bot's case, and its numbers hold. The case: a branch that conflicts with the target, is 500–10,000 commits behind, and changes only a file the target keeps editing, such as a lockfile. Main gives up on it after one
Nearly all of the time goes to Why I think it's acceptable as is:
There's a second, smaller case of the same kind: the path-limited If you'd rather close it off, there are two options:
My preference is to keep it as is. I'm happy to add option 1 if you'd rather bound it. |
|
One refinement to option 1 that keeps the guarantee: tie the patch-id cap to how far behind the branch is.
Since the |
…-prefilter # Conflicts: # src/git/repository/mod.rs
Follow-up to #4100, which you closed because it was slower than main. This version detects the same squash merges as #4100, and it's faster than main on a large repo. On a small, fast-moving repo it's about 0.2 s slower than main on a cold cache, and slightly faster after a fetch.
Why main is cheap on some repos and slow on others
Main's patch-id fallback stops after one
rev-list --countfor a conflicting branch more than 500 commits behind, so it never detects that branch. For a branch under 500 behind, it runsdiff-tree -pon every commit in the range, with no path filter. In a monorepo, those commits touch many large files.What this PR does
Path filter, from fix(integration): find squash merges past the patch-id cap by scanning the branch's paths, oldest first #4100. Only target commits that touch one of the branch's paths can match:
git patch-idhashes each file'sdiff --gitline, so a commit with the branch's patch-id touches every path the branch changed. The filter prefers a file the branch added.Two caps instead of one, from fix(integration): find squash merges past the patch-id cap by scanning the branch's paths, oldest first #4100. The scan gives up past 10,000 commits behind (a graph-only
rev-list --count) and diffs at most the oldest 500 candidates.Resumable scan, from fix(integration): find squash merges past the patch-id cap by scanning the branch's paths, oldest first #4100. A
patch-id-scancache entry per (branch, merge-base) records how far the scan got, so after a fetch only the new commits are checked. A branch whose check already found a match, or used up its candidates, answers from that entry without running git. I tried dropping the cache now that scans are cheaper, but it's what keeps repeat runs fast: without it, "after fetch" on the GitOps repo below went from about 1.0 s to 1.7 s.New: blob ids before patches.
git patch-idhashes every line of a file's patch header exceptindex(seebuiltin/patch-id.c). Onediff-tree --stdin -r -z --rawcall diffs the candidates' trees without reading any file contents, and that settles most candidates:diff-tree -p | patch-id, together with the branch's own diff in the same pass.A typical squash merge now costs three git processes and builds no patches. fix(integration): find squash merges past the patch-id cap by scanning the branch's paths, oldest first #4100 needed seven processes and a patch for every candidate.
The rules only skip work: the answers match #4100's, and include every branch main detects.
Results
These are release builds of main (
64ead5388), #4100 (6cccbe62a) and this PR, run on bare local clones. "Cold" means an empty cache with the default branch 50 commits back. "After fetch" fast-forwards it to the tip with the cache kept. The machine was otherwise quiet, with a load average of 7 on 10 cores before the runs. During them it rose to 37–58, largely fromwt's own parallel git processes, so treat differences under about 10% as noise.A private monorepo: 31k commits, 452 local branches, median of 3 runs.
wt list --branches, coldwt step prune --dry-run, coldprunewould removeThe GitOps repo from #4100: 50 local branches, a bot committing to
mainconstantly, median of 5 runs.wt list --branches, coldwt step prune --dry-run, coldprunewould removeRuns with nothing changed (warm) are the same for all three: 0.1 s on the GitOps repo and 0.3 s on the monorepo. Across both repos, this PR's
pruneoutput is identical to #4100's, and so are the brancheswt listmarks integrated.On the GitOps repo, what remains is process count rather than diffing. For a conflicting branch more than 500 commits behind, main runs one
rev-list --countand gives up; this PR runs that count plus a raw branch diff and a path-limitedrev-list. At this repo's sizes each git process costs about 20 ms. Most of these branches match on blob ids, and only 5 of the 34 matches needed patch-ids.Limitations (unchanged from #4100)
These are both in the safe direction: the branch is kept, and
-Dstill removes it.There's still no hard time bound. Each check is bounded by commit counts, the same kind of bound main has, and the blob-id step means a typical candidate costs a tree diff rather than a patch.
Tests
detects_squash_onto_a_file_edited_since_the_fork: the target edited the file elsewhere before the squash, so the blobs differ but the patch-ids match.rejects_different_change_to_the_same_paths: the same paths with a different change don't match.detects_squash_merge_under_nondefault_diff_config: now edits a distant line first, so it still exercises patch-ids underdiff.contextanddiff.algorithm.classifies_candidates_by_raw_changesandparses_raw_diff_with_and_without_commit_ids.submodule.<name>.ignore=all.