Visitar URL original
ci: report corpus baseline drift on a cadence instead of by accident by robcohen · Pull Request #2196 · rustledger/rustledger · GitHub
Skip to content

ci: report corpus baseline drift on a cadence instead of by accident - #2196

Merged
robcohen merged 8 commits into
mainfrom
chore/corpus-baseline-drift
Aug 30, 2026
Merged

robcohen merged 8 commits into
mainfrom
chore/corpus-baseline-drift

Conversation

@robcohen

Copy link
Copy Markdown
Member

Refs #2186.

Part 1 is already done

The manifest is current. On a clean checkout with a freshly fetched corpus:

  • ./scripts/fetch-compat-test-files.sh → 735 files
  • ./scripts/regen-corpus-baselines.sh → zero diff
  • discovered files 735, manifest entries 735, 0 missing, 0 orphaned

#2185/#2189 carried the 3 modified hashes and 20 additions the issue lists. Nothing left to regenerate.

The stated cause isn't the cause

The issue says the gate is path-filtered. It isn't — parser-baselines.yml has no paths: filter and its header says it runs on every PR. The real mechanism is two deliberate decisions meeting:

  1. An unmanifested downloaded file is a warning, not a failure. That's correct as written: upstream can add a file at any moment, and failing would turn unrelated PRs red. But the warning goes to a CI log nobody reads, so the file sits with no parser-output coverage at all.

  2. CI's corpus is frozen. The job restores tests/compatibility/files from a cache keyed on hashFiles('scripts/fetch-compat-test-files.sh'), and the fetch step is skipped on an exact key hit. The key changes when the script changes — never when upstream does. So CI keeps replaying whatever corpus existed when that key was created, and every PR agrees there's no drift.

Neither is wrong alone, and tightening either costs exactly what its own comment explains. What's missing is a third thing that looks at a corpus nobody has frozen.

What this adds

A weekly job that fetches the corpus fresh — no cache, which is the entire point — regenerates, and turns any drift into an issue. It reports; it never gates; it closes its own issue when the manifest matches again.

The issue body separates the three cases, because they need different responses:

group meaning
New corpus files with no baseline no parser-output coverage at all since they appeared
Changed hashes source hash moved = upstream churn; output hash moved with source unchanged = a parser change reached main without regenerating
Entries whose file is gone upstream deleted or renamed; dead weight

The oracle is the repository's own BASELINE_UPDATE=1 path plus git diff — not a Python reimplementation of the fingerprint, which would be a second implementation of the thing under test, free to disagree with the Rust one in precisely the cases that matter.

Self-test

Modeled on nightly-health.yml, including the self-test that runs before the live check and is allowed to fail the job. The live check exits 0 whatever it finds, so a broken classifier would otherwise report a permanently reassuring "no drift" — the same silence this workflow exists to break, wearing a green check.

Verified it can fail, two ways:

break caught
rehash detection disabled (rehashed = []) 3 failures
classifier returns clean for everything 5 failures

It also pins that an empty diff reports clean and that comment-only lines aren't drift, so it can report both answers rather than only the alarming one.

🤖 Generated with Claude Code

https://claude.ai/code/session_012nNGPA44Dt32NyxWem26xr

The manifest is current -- a fresh fetch of the corpus regenerates to
zero diff, and all 735 files have an entry. #2185/#2189 carried the
drift #2186 describes. What is left is the reason it accumulated.

The issue attributes it to a path filter on the baseline gate. There is
none: `parser-baselines.yml` runs on every PR. The actual mechanism is
two deliberate decisions meeting:

  * A downloaded corpus file with no manifest entry is a WARNING there,
    not a failure, and the comment saying so is right -- upstream can
    add a file at any moment and failing would fire on unrelated PRs.
    But the warning goes to a log nobody reads, so the file sits with no
    parser-output coverage at all.

  * That job restores the corpus from a cache keyed on the FETCH SCRIPT
    and skips the fetch on an exact key hit. CI's corpus is frozen at
    whatever was fetched when the key was created, while the upstream
    repositories it mirrors keep moving. Every PR sees the same stale
    corpus and agrees there is no drift.

Neither is wrong on its own, and tightening either has the cost its
comment already explains. So this adds the missing third thing: a weekly
job that fetches the corpus FRESH -- no cache, which is the entire point
-- and turns drift into an issue. It reports, never gates, and closes
its own issue when the manifest matches again.

The oracle is the repository's own `BASELINE_UPDATE=1` path plus `git
diff`, not a Python reimplementation of the fingerprint, which would be
a second implementation free to disagree with the Rust one in exactly
the cases that matter.

Modeled on `nightly-health.yml`, including the self-test that runs first
and IS allowed to fail the job: the live check exits 0 whatever it
finds, so a broken classifier would otherwise report a permanently
reassuring "no drift" -- the same silence this exists to break, wearing
a green check. Verified by breaking it two ways (rehash detection
disabled, everything reported clean); the self-test catches both.

Refs #2186
@robcohen
robcohen requested a review from a team as a code owner August 30, 2026 17:56
Copilot AI lite review requested due to automatic review settings August 30, 2026 17:56
Comment thread scripts/corpus-baseline-drift.py Fixed
Comment thread scripts/corpus-baseline-drift.py Fixed

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The drift reporter currently swallows gh CLI failures, which can lead to a green workflow that silently fails to create/update/close the reporting issue.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a scheduled GitHub Actions workflow to detect and report drift between the freshly-fetched compatibility corpus and the committed parser baseline manifest, so baseline changes are surfaced on a cadence (via an auto-managed issue) rather than only when an unrelated PR happens to touch the parser.

Changes:

  • Adds scripts/corpus-baseline-drift.py to regenerate the manifest, compute drift from git diff, classify it (added/removed/rehashed), and open/update/close a dedicated issue.
  • Adds .github/workflows/corpus-baseline-drift.yml to run the drift reporter weekly (and on manual dispatch), explicitly fetching the corpus without the corpus cache and running a self-test before the live check.
File summaries
File Description
scripts/corpus-baseline-drift.py Implements manifest regeneration + diff-based drift classification and issue lifecycle management.
.github/workflows/corpus-baseline-drift.yml Schedules the drift check, fetches corpus fresh (no corpus cache), and runs a classifier self-test before reporting.
Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread scripts/corpus-baseline-drift.py Resolved
@codecov

codecov Bot commented Aug 30, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@github-actions

Copy link
Copy Markdown
Contributor

✅ Compatibility Test Results

Metric Result
Check Compatibility 100%
BQL Compatibility 100% (raw 93%, 170 masked)
Full AST Match 100%
Total Files 811
What's tested
  • Check: Exit codes and error counts match bean-check
  • BQL: Query results match bean-query
    The headline is the EFFECTIVE rate, after pairs where beanquery
    itself is wrong are masked. raw is the same run with nothing
    masked. A widening gap means the registry grew, not that rledger
    changed.
  • Files: ~800 test files downloaded from 10+ beancount repositories

Full results available in workflow artifacts

Review of the commit before this one. Fetching FRESH is what makes this
workflow able to see drift, and it is also what makes it able to invent
some.

The fetch tolerates up to 15 best-effort clone failures. On the PR gate
that is harmless: it restores a cached corpus, so a failed clone still
leaves the previous copy on disk -- `fetch-compat-test-files.sh` only
`mkdir -p`s its destination and never clears it. This job deliberately
skips that cache, so it fetches into an EMPTY checkout, where every
tolerated failure is simply missing files. Regenerating against that
drops their manifest entries, and the report would have called them
"upstream deleted or renamed these" -- a confident lie, longer and more
alarming the more repositories happened to fail.

So the corpus is counted first, against the same 50-file slack the
"Verify corpus is populated" step in `parser-baselines.yml` uses for the
same reason. Short by more than that and the run reports nothing and
says why. A partial fetch cannot manufacture a false "added" or
"rehashed", but it does mean the run saw an unrepresentative corpus, and
silence with a reason beats a plausible wrong answer.

Two more fixes to `--dry-run`, which regenerates and therefore rewrites
the manifest: it now restores the file afterward, because a flag named
dry-run must not leave a modified file for someone to commit by
accident -- and it does NOT restore it when the manifest was already
modified before the run, because `git checkout --` would delete the
caller's edits. A reporting script has no business destroying someone's
work to keep its own promise.

The guard is self-tested in both directions: always-complete (the bug
above) and always-refuse (the over-correction that would silence the
workflow) each fail it.
`tests/baselines/README.md` documents the CI behavior for this manifest
and is where someone goes to understand it, so a second workflow acting
on the same file belongs there. Without it the cadence job reads as
duplicate coverage, and the natural next move is to delete one.

The section names the two reasons the PR gate cannot see this class --
an unmanifested downloaded file warns rather than fails, and the corpus
cache is keyed on the fetch script rather than on upstream state -- so
the distinction survives past whoever remembers the incident.

Also two guards on the script: it refuses to run outside the repository
root, where its relative paths and its `git checkout --` would either
fail confusingly or resolve somewhere unintended; and the comment about
always exiting 0 now says what that covers. Drift is reported rather
than gated, but `regenerate()` runs under `check=True`, so a corpus file
that panics the parser still fails the step -- that is a real problem on
main and nightly-health picks it up from there.
`gh()` returned an empty string when the command failed, which is
indistinguishable from a successful empty result. For a job whose only
output is an issue, that conflation is the worst bug available:

  * a failed `issue list` looks like "no existing issue", so the next
    run opens a duplicate, every week;
  * a failed `issue create` yields a blank URL, prints "opened ", and
    leaves the workflow GREEN having reported nothing at all.

The second is exactly the silence this workflow exists to break, wearing
a green check -- the same failure `nightly-health.yml` was written for,
reproduced inside its sibling.

`gh()` now raises `GhError`, and `main` turns that into a non-zero exit.
Finding drift stays exit 0, because a red X for a successful report
would be one more failing scheduled workflow for nobody to notice;
failing to report is the opposite case. A job that cannot say what it
found must not look like a job that found nothing.

Self-tested in both directions by stubbing `subprocess.run`: returning
"" on failure fails the test, and so does raising on a successful empty
result, which is a real and different answer.

Also makes the two list-element string concatenations explicit --
CodeQL's `py/implicit-string-concatenation-in-list`, flagged in review.
Adjacent literals on consecutive lines inside a list read as a missing
comma, and in a list of issue-body paragraphs that is a plausible typo
rather than a pedantic one.
@robcohen

Copy link
Copy Markdown
Member Author

Fixed in the latest commit — both flagged spots now build the string explicitly.

The lead paragraph moved into a _summary() helper and the trailer uses a parenthesised single expression, so no list element is two adjacent literals any more. You're right that it's not pedantic here: in a list of issue-body paragraphs, a missing comma is a plausible typo rather than a theoretical one, and it would silently merge two paragraphs instead of erroring.

Same commit fixes something the review prompted me to go looking for: gh() returned "" on failure, which is indistinguishable from a successful empty result. A failed issue list would have read as "no existing issue" and opened a duplicate weekly, and a failed issue create would have printed a blank URL and left the run green having reported nothing — the exact silence this workflow exists to break. It now raises and the step exits non-zero, self-tested in both directions.

Comment thread scripts/corpus-baseline-drift.py Fixed
Comment thread scripts/corpus-baseline-drift.py Fixed
`_report` decides between opening, updating and closing the issue -- it
is the only code in the script that changes anything -- and nothing had
ever run it. `--dry-run` returns before reaching it, and a scheduled run
exercises exactly one branch per week, so three of its four paths would
have shipped unexecuted and the fourth would have been tried in
production.

All four now run against a recording stub: drift with no open issue
creates, drift with an open issue edits THAT issue number, clean with an
open issue comments then closes in that order, and clean with no issue
touches nothing at all.

Each is verified to fail for its own reason -- swapping edit for create,
dropping the close, and passing drift to the no-op case each break
exactly one assertion. The last one matters most: a `_report` that
opened an issue every clean week would still have passed a test that
only checked the drift paths.

Stdout is captured while they run. Left alone the stub prints "opened
https://github.com/x/y/issues/1" into a self-test log, which is the sort
of line someone skims and believes.
The baseline test already refuses to write a manifest from fewer than
`MIN_FULL_CORPUS_SIZE` (100) files, so a partial fetch cannot silently
produce a truncated manifest and a false "no drift" -- checked, and it
panics rather than skipping when `BASELINE_UPDATE` is set.

That backstop and this check are not redundant, and the note says why:
at a corpus of ~735 this one trips at ~685, so a merely partial fetch
gets a sentence explaining itself instead of a cargo stack trace, and
the test's assert stays what it was written to be -- a last resort
against truncating the committed manifest, not the routine path.

Someone reading either guard alone would reasonably conclude the other
is dead weight.
The "Changed hashes" section told the reader "the diff says which: a
changed source hash is upstream churn; a changed output hash with an
unchanged source hash is a parser change that reached main without
regenerating" -- and then printed a list of paths and no diff. It knew
the answer and sent the reader to find it.

`classify` already had both sides of every changed line, so it now
compares the source hashes and splits the group in two. They are not a
matter of emphasis:

  * source hash moved   -> the file's contents changed upstream, a
                           different output hash is expected, routine.
  * source hash IDENTICAL, output hash moved -> this repository produces
                           different output for the same bytes than the
                           committed baseline records. That is a parser
                           change that reached main unregenerated, and it
                           is the reason this job exists.

Collapsed into one list they look the same and read as churn, which is
how the second kind gets skimmed past.

Self-tested in both directions: forcing every rehash to read as upstream
fails, and so does forcing every one to read as a parser change. The
second control is the one that matters -- a split that always cries
"parser change" would be noise nobody acts on within a month.
…mpty

Two CodeQL findings from review, both in code this branch added.

`py/implicit-string-concatenation-in-list`: nine runs of adjacent string
literals now join with an explicit `+`. Wrapping them in parentheses was
not enough -- adjacency inside parentheses is still adjacency, and the
concern is real for these in particular, since they are paragraphs of an
issue body where a missing comma would silently merge two of them rather
than raise.

Verified the rewrite changed nothing: rendering the same diff through
the pre-rewrite and post-rewrite modules gives a byte-identical body
(sha e0194f7e3082a81a both sides).

`py/empty-except`: the self-test's `except GhError: pass` is the
assertion -- reaching that arm is the pass condition and the
`failures.append` above it is what runs otherwise -- but nothing said
so, and a bare `pass` under an `except` is worth explaining every time.
@robcohen

Copy link
Copy Markdown
Member Author

All three addressed.

Implicit string concatenation — fixed in cdbfd0d, nine runs now join with an explicit +. Wrapping in parentheses wasn't enough; adjacency inside parentheses is still adjacency. Worth doing here specifically because these are paragraphs of an issue body, where a missing comma silently merges two of them rather than raising. Verified the rewrite changed nothing by rendering the same diff through the pre- and post-rewrite modules: byte-identical body, sha e0194f7e3082a81a on both sides.

Empty except — same commit. Reaching that arm is the assertion (the failures.append above it is what runs otherwise), but nothing said so, and a bare pass under an except is worth explaining every time.

gh() swallowing non-zero exits — you're right, and it was already fixed two commits before your comment landed, in fix(ci): a gh failure must not read as "nothing to report". Your reasoning matches exactly what I'd written there: a failed issue list reads as "no existing issue" and opens a duplicate weekly, and a failed issue create prints a blank URL and leaves the run green having reported nothing — the exact silence this job exists to break. It now raises GhError and the step exits non-zero, self-tested in both directions (returning "" on failure fails the test; so does raising on a successful empty result, which is a real and different answer).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants