Visitar URL original
test(compat): vary lot-identity axes together in the booking fuzzer by robcohen · Pull Request #2117 · rustledger/rustledger · GitHub
Skip to content

test(compat): vary lot-identity axes together in the booking fuzzer - #2117

Merged
robcohen merged 2 commits into
mainfrom
chore/booking-fuzz-two-axes
Aug 23, 2026
Merged

robcohen merged 2 commits into
mainfrom
chore/booking-fuzz-two-axes

Conversation

@robcohen

Copy link
Copy Markdown
Member

The first campaign answered one question well and never asked the others. Every divergence it found had same-date lots — 12/12, against 0/189 for seeds without them — because random generation collided on date, cost and label only by accident. Two lots at the same highest cost under HIFO, or two on the same date under STRICT, essentially never appeared.

Lot identity has three components a method can tie on, so the generator now constructs each configuration deliberately (none, date, cost, both, label) and reports the tie mode with every divergence.

Coverage is now checkable

             none   date   cost   both  label
  STRICT       26     28     30     17     28
  FIFO         23     27     18     31     26
  LIFO         22     24     26     29     24
  HIFO         19     20     23     31     28

All 20 cells populated at --runs 500. That's what makes the empty results meaningful: HIFO with tied costs and STRICT with tied dates now have real coverage and agree with beancount. Previously "no HIFO divergences" just meant the case had never been generated.

Two corrections, both from measuring rather than reading

none was not a control. Costs were drawn with replacement from a four-element pool, so 79% of "no tie" runs quietly contained a cost tie — useless for attributing a divergence to the date axis. Now sampled without replacement: 0 date ties, 0 cost ties.

Tying exactly two lots lost a known bug. The FIFO coalescing divergence needs three lots sharing a date with two sharing a cost — a non-contiguous repeat — which a uniform draw produced about once in seven date-tied runs. I confirmed this by running the known-divergent ledger directly: the bug was still there, the generator had simply stopped reaching it. Ties now span a random-sized group, and date-tied groups draw costs from a narrow pool so repeats land non-contiguously. FIFO divergences reappeared immediately.

What it finds now

At --runs 500: 490/500 agreed, 10 divergences — 8 LIFO × date, 1 FIFO × date, and one new class:

LIFO × label — two lots differing only by label, where rledger rejects a ledger beancount books. That's the mirror of the acceptance divergence already in #2115, so the disagreement cuts both ways: depending on the ledger, either engine may be the one that refuses.

All are the same root cause and tracked in #2115.

The first campaign answered one question well and never asked the
others. Every divergence it found had same-date lots (12/12, against
0/189 for seeds without them), because random generation collided on
date, cost and label only by accident: two lots at the same HIGHEST cost
under HIFO, or two on the same date under STRICT, essentially never
appeared.

Lot identity has three components a method can tie on, so the generator
now constructs each configuration deliberately — none, date, cost, both,
label — and reports the tie mode with every divergence so the classes
stay separable. All 20 method x tie cells are populated (17-31 runs per
cell at --runs 500), which is what makes the empty ones meaningful:
HIFO with tied costs and STRICT with tied dates now have real coverage
and agree with beancount.

Two corrections during the extension, both caught by measuring rather
than reading:

`none` was not a control. Costs were drawn with replacement from a
four-element pool, so 79% of "no tie" runs quietly contained a cost tie
and could not be used to attribute a divergence to the date axis. It now
samples without replacement: 0 date ties, 0 cost ties.

Tying exactly two lots LOST the FIFO coalescing divergence the first
campaign had found. That shape needs three lots sharing a date with two
of them sharing a cost — a NON-CONTIGUOUS repeat — and a uniform draw
produced it about once in seven date-tied runs. Ties now span a
random-sized group, and a date-tied group draws its costs from a
deliberately narrow pool so repeats land non-contiguously. Verified
against the known-divergent ledger directly: the bug was still there,
the generator had stopped reaching it.

At --runs 500 this finds 10 divergences: 8 LIFO x date, 1 FIFO x date,
and one new class — LIFO x label, where two lots differ only by label
and rledger REJECTS a ledger beancount books. That is the mirror of the
acceptance divergence already reported, so the disagreement cuts both
ways. All are the same root cause and are tracked in #2115.
@robcohen
robcohen requested a review from a team as a code owner August 23, 2026 21:34
Copilot AI lite review requested due to automatic review settings August 23, 2026 21:34

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the booking compatibility fuzzer (scripts/compat-booking-fuzz.py) to deliberately generate lot-identity tie configurations (date/cost/label) and to report the tie mode alongside divergences, improving coverage of previously rare booking edge cases (notably LIFO/STRICT/HIFO ties).

Changes:

  • Adds explicit tie-mode generation (none, date, cost, both, label) rather than relying on incidental collisions.
  • Adjusts lot generation to tie a random-sized group and biases date-tied groups to increase repeated-cost patterns.
  • Extends divergence reporting to include tie=<mode> for easier classification.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread scripts/compat-booking-fuzz.py Resolved
Comment thread scripts/compat-booking-fuzz.py Resolved
@codecov

codecov Bot commented Aug 23, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@github-actions

Copy link
Copy Markdown
Contributor

✅ Compatibility Test Results

Metric Result
Check Compatibility 100%
BQL Compatibility 100% (raw 93%, 170 masked)
Full AST Match 100%
Total Files 811
What's tested
  • Check: Exit codes and error counts match bean-check
  • BQL: Query results match bean-query
    The headline is the EFFECTIVE rate, after pairs where beanquery
    itself is wrong are masked. raw is the same run with nothing
    masked. A widening gap means the registry grew, not that rledger
    changed.
  • Files: ~800 test files downloaded from 10+ beancount repositories

Full results available in workflow artifacts

Both from review, and both were real.

The generator gained a label axis but the comparison key did not, so two
lots differing only by label collapsed into one entry on BOTH sides. The
fuzzer could see a label divergence only when it changed acceptance; a
divergence in which labelled lot got consumed was invisible by
construction. Adding `cost_label` to the key took the campaign from
490/500 to 465/500 — 15 LIFO x label divergences where there had been 1.
Seed 30 is representative: same total, same cost, same date, but
rledger books lot-a=5/lot-b=1 where beancount books 3/3.

The narrowed cost pool was described as producing non-contiguous repeats
and did not. It drew independently per lot, so a group of two never
repeated and a group of three usually came out contiguous or all-equal.
The shape is now constructed outright ([a, b, a, ...]): 47 of 47
date-tied groups of three or more have it, against roughly one in seven
before. FIFO divergences went from 1 to 7.

Net effect of a comment that only claimed something and a key that only
covered part of the identity it was generating: two thirds of the
divergences in this space were not being reported.
@robcohen

Copy link
Copy Markdown
Member Author

Both correct, both fixed in 9de3960635 — and together they were hiding two thirds of the divergences in this space.

The comparison key didn't include the label. Exactly as you say: the generator gained a label axis but the key didn't, so two lots differing only by label collapsed into one entry on both sides. A label divergence was visible only when it changed acceptance; a divergence in which labelled lot got consumed was invisible by construction.

Adding cost_label took the campaign from 490/500 to 465/500 — 15 LIFO × label divergences where there had been 1. Seed 30 is representative:

Assets:Stock/HOOL/12/USD/2020-01-02/lot-a: rledger=5 beancount=3
Assets:Stock/HOOL/12/USD/2020-01-02/lot-b: rledger=1 beancount=3

Same total, same cost, same date — different distribution across labels.

The comment claimed a shape the code didn't produce. Also right: an independent draw per lot meant group==2 never repeated and group>=3 mostly came out contiguous or all-equal. The shape is now constructed outright ([a, b, a, …]) rather than hoped for — 47 of 47 date-tied groups of three or more have it, against roughly one in seven before. FIFO divergences went from 1 to 7.

I'd been treating that comment as documentation when it was really an assertion I hadn't checked, which is the more useful half of the catch.

@robcohen
robcohen merged commit b3d24f8 into main Aug 23, 2026
45 checks passed
@robcohen
robcohen deleted the chore/booking-fuzz-two-axes branch August 23, 2026 22:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants