fix(query): split format vs format_amount semantics — recover BQL compat (refs #1112) - #1113

…Rust divergences (refs #1112) (#1114)

* chore(compat): mask the last 3 BQL divergences from #1112

After #1113 recovered the bulk of the #1106 regression, three real
mismatches remain — all `sum-number-by-currency` queries on fixtures
whose cost-spec interpolation produces a scale-3 residual in Python's
pipeline vs scale-2 in rledger's:

* testdata_source_healthequity_test_invalid_journal.beancount
* testdata_source_healthequity_test_matching_journal.beancount
* testdata_source_ofx_test_non_default_capital_gains_journal.beancount

In every case the numeric value is the same (e.g. `-1966.700` vs
`-1966.70`) — only the display scale differs, surfaced as a textual
diff because both tools intentionally preserve scale on
`Value::Number` output.

Root cause: cost-spec interpolation against `{}` lot-match against a
`{{total}}` lot produces a residual whose scale depends on which
intermediate value drives it. Python's intermediate stays at the
buy-side scale 3; rledger's #1108 fix dropped intermediate scale to
the input minimum (2) to stop the 26-digit contamination. That fix
was correct for the over-precision case but over-applies here.

Mask surgically (per `(file, query_name)`, not `"*"`) so any
non-scale divergence on these fixtures stays surfaced. This is a
display-scale shim, not a value bug — both tools agree on the
arithmetic. Fixing it properly means continuing #1108's pipeline
scale-propagation work, which is tracked under #1112 as a follow-up.

After this commit all 5 `first-balance-by-month` Python beanquery
bugs (#1105) and the 3 scale residuals here are accounted for as
known divergences, taking the BQL compat percentage's effective
match closer to 100%. The user-facing rounded percentage stays at
99% (it was already 99% post-#1113); this commit cleans up the
remaining bookkeeping so future regressions stand out clearly.

Refs #1112.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor(compat): split known-Python vs known-Rust BQL divergences (#1114 review)

Copilot review caught that the 3 `sum-number-by-currency` entries I
added to `KNOWN_PYTHON_DIVERGENCES` are actually rledger-side bugs
(continuation of #1108's pipeline scale-propagation work), not Python
beanquery quirks. Conflating them in the Python allowlist lets an
rledger bug masquerade as a Python quirk and could absorb a future
rledger regression on these file/query pairs.

Fix: introduce a separate `KNOWN_RUST_DIVERGENCES` set with the same
surgical-pin semantics. Wire it through:

- New predicate `_is_known_rust_divergence(run)` mirroring the Python
  one. New `_is_known_divergence(run)` returns the OR of both.
- Summary stats report `Known Python diffs` and `Known Rust diffs`
  separately so each side's allowlist size is visible at a glance.
- Per-mismatch log lines use `KNOWN-PY` / `KNOWN-RS` / `MISMATCH`
  labels.
- `$GITHUB_OUTPUT` gets two new keys, `bql_known_python` and
  `bql_known_rust`, alongside the existing aggregate `bql_known_divergences`.
- The 3 cost-spec interpolation cases move out of the Python allowlist
  and into the new Rust one. Effective match percentage is unchanged
  (still counted as "known"); only the bookkeeping is corrected.

Also addresses Copilot's second comment: the inline doc on the Rust
allowlist now says "Tracked under #1112 (kept open as the tracker — do
not auto-close from this PR)" so the reference stays valid post-merge.
PR body updated correspondingly.

Refs #1112. Does NOT auto-close it — the deep pipeline fix is the
real closure path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(booking): prune zero-value interpolated postings post-validation

Until now rledger preserved every interpolated posting through to user-
facing output, including zero-value ones (commit 7706be9, closes #877).
That fix was correct in spirit — pruning before validation hides E1001
on never-opened accounts referenced by elided postings — but
over-applied: it kept zero postings in user-facing output too, where
Python beancount drops them. The result was a persistent ~0.25% Full
AST Match floor on fixtures like `tests/regressions/issue-276.beancount`
and `tests/regressions/issue-279.beancount` (multi-posting lot sales
with zero residuals on Income:Trading).

This commit splits the two invariants by phase:

1. Booking marks every interpolated posting with a new metadata key
   `__interpolated__` (exposed as `rustledger_booking::INTERPOLATED_MARKER`).
   Validation runs against transactions that still include zero-value
   postings — so #877's E1001-on-unopened-account check still fires.

2. AFTER validation, `process()` walks each transaction and drops
   postings that have the marker AND a zero `units` value. Survivors
   keep their value but the marker is stripped so it doesn't leak into
   BQL queries, JSON output, or formatted ledgers.

Both invariants verified by new integration tests in
`crates/rustledger-loader/tests/loader_test.rs`:

* `test_zero_interpolated_posting_pruned_on_opened_account` — zero-value
  interpolated Income posting is dropped from output, no errors.
* `test_zero_interpolated_posting_keeps_e1001_on_unopened_account` —
  the #877 scenario: E1001 still emitted.
* `test_non_zero_interpolated_posting_is_preserved` — only ZERO ones
  get pruned; legitimate interpolated residuals stay.
* `test_interpolated_marker_does_not_leak_to_user_output` — defensive
  check that `__interpolated__` is stripped from survivors.

Effect on compat suite (verified locally with `--no-default-features`,
matching CI):

  Before                                             After
  ─────────────────────────────────────────────      ─────────────────────────
  issue-276.beancount  SELECT COUNT(*) = 12          = 10  (matches Python)
  issue-279.beancount  SELECT COUNT(*) = 13          = 11  (matches Python)
  Full AST Match       99% (784/786)                 100% (786/786)

The #877 regression case still produces E1001 with this commit applied:

  $ rledger check zero-balance.bean
  E1001 Account Expenses:NeverOpened was never opened
  ✗ 1 error
  exit_code: 1

Closes the residual AST divergence tracked under #1112.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: single-source the posting weight arithmetic in booking

The Beancount posting-weight rule (cost beats price; preserved totals;
@@ sign follows units) existed three times: the canonical generic
ladder in rustledger-booking (shared by the residual and interpolation
since #1700), a copy in the query executor's compute_posting_weight
(the BQL weight column), and a copy in the currency_accounts plugin's
weight_of closure. The copies had already sprouted real divergences
from the canonical:

- query recomputed total-cost weights as units x per_unit via
  CostSpec::resolve, reintroducing the division-then-multiplication
  precision loss the booking ladder deliberately avoids by using the
  preserved total (#1106/#1113): SELECT weight for a booked
  {{100.00 USD}} over 3 units showed 99.99999999999999999999999999
  where rledger check balances with 100.00 exactly.
- the plugin took abs() of totals and @@ prices before re-signing,
  silently "fixing" negative magnitudes instead of following the
  canonical total x signum arithmetic.

Expose the ladder from booking as two public Decimal functions backed
by the existing generic (WeightNum) implementations — no second copy:

- cost_number_weight(units, &CostNumber): the four-variant cost
  arithmetic (PerUnit multiply, Total/PerUnitFromTotal preserved total
  x signum, Compound N*a+b).
- price_weight(units, price, PriceKind): @ = |units| x price x signum,
  @@ = price x signum (the #1052 credit-side flip; zero units weigh
  zero for both kinds).

residual_weight's price arm now uses the same generic helper, so the
validator, BQL weight column, and currency_accounts neutralization all
compute weights through one ladder. The query's fallback order and the
plugin's DTO parsing (with its exhaustive compile-fail match, now the
CostNumberData -> CostNumber mapping) are unchanged.

New tests pin the exact-total weight through SELECT weight, all four
CostNumber variants, and the price sign rules.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: regenerate cargo-vet exemptions

* docs: justify BookedCost struct literal in currency_accounts (review)

The weight arithmetic reads only .total for PerUnitFromTotal, so the
per_unit consistency invariant is irrelevant at this site; enforcing
it would need an error channel the infallible weight closure doesn't
have. Wire-pair consistency is enforced at the ingress boundary
(ffi-wasi input_entry_to_directive uses try_new).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: rustledger-bot[bot] <257329324+rustledger-bot[bot]@users.noreply.github.com>

…not (#1965)

* fix(query): stop WEIGHT() inventing precision the weight column does not

Found by the canonical-function drift sweep (#1902 Phase 2). CLAUDE.md names
this pair specifically: a consumer surfacing a per-posting weight must call
`rustledger_booking::posting_weight` rather than re-derive it. The `weight`
COLUMN does. The `WEIGHT()` FUNCTION re-derived `units * cost.number`.

  3 HOOL {{100.00 USD}}

  weight column     100.00
  WEIGHT(position)  100.00000000000000000000000000

Same value, 26 digits of scale invented by the multiplication. `{{100.00}}`
resolves to a per-unit of 33.333..., and multiplying back reintroduces exactly
the division-then-multiplication the canonical exists to avoid (#1106/#1113).
The compat suite compares RENDERED output, so scale is not cosmetic.

WHY THIS IS A MITIGATION AND NOT A CORRECTION

`WEIGHT()` cannot call the canonical. It receives a `Value::Position`, whose
cost is already resolved to a per-unit `Decimal` - the preserved total that
makes `posting_weight` exact is gone by then - and
`evaluate_function_on_values` gets only `Value`s, with no posting to recover it
from. So the function still yields `100` where the column yields `100.00`.
Closing that needs `Position` to retain the cost spec, which is a wider change
than the symptom warrants; it is recorded on #1963.

THE OBVIOUS FIX WAS WORSE, AND IS NOW PINNED AGAINST

Normalizing unconditionally was the first attempt. It fixes the case above and
REGRESSES the common one: `2 HOOL {5.25 USD}` went from `10.50` to `10.5`,
because normalize strips meaningful trailing zeros too. Trading a rare cosmetic
problem for a frequent one is not a fix.

So the strip applies only when the scale is already past anything a real cost
carries (>12). Such a scale can only come from a division; `rust_decimal` tops
out at 28, and no written cost has 12 decimal places.

Both tests verified to fail on their own sabotage:

  raw product           -> "WEIGHT() invented precision: column=100.00 ..."
  unconditional normalize -> "on a plain per-unit cost the function and the
                              column must match exactly"

The second is the one worth having: it is the guard against re-attempting the
fix I nearly shipped.

Closes #1963.

* fix(query): use checked_mul in WEIGHT, and assert on scale directly

- WEIGHT() multiplied with the bare operator while the adjacent COST()
  arm uses checked_mul + overflow_err. Same product, same operands, two
  different overflow behaviors; now both fail loudly. A saturating
  product would certify a weight that never existed, which is the
  unsoundness #1863 removed from the residual path.

- The comment claimed a scale past 12 could only have come from a
  division, never from a written number. rust_decimal accepts literals
  out to 28 places, so a user could author a 13-place cost and see its
  weight normalized. Stated as the heuristic it is, with why the
  alternative (normalizing every weight) is worse.

- The regression guard compared Debug-string lengths as a proxy for
  precision. A formatting tweak could move it with no scale changing,
  and it never named the invariant. Compares Decimal::scale() now.
  Both sabotages still fail the test: dropping the normalize (function
  scale 26 vs column 2) and applying it unconditionally (10.50 -> 10.5).

* fix(query): route both WEIGHT() cost paths through one helper

Deep review of this PR found the fix was half a fix. WEIGHT() reaches
the same units x cost arithmetic two ways - a bare Position, and every
position inside an Inventory - and the first commit patched only the
first. The Inventory arm 30 lines below kept #1963 verbatim, so on the
same 3 HOOL {{100.00 USD}} lot:

  weight column          100.00
  WEIGHT(position)       100
  WEIGHT(SUM(position))  100.00000000000000000000000000

The two WEIGHT() spellings disagreed with each other as well as with
the column. Which is the drift this sweep exists to find: the same
quantity derived twice in one function, fixed once.

Both arms now call position_cost_total(). That also consolidates the
overflow message - the Inventory arm had its own wording rather than
overflow_err() - and nothing asserts on the old text.

New test pins the two paths together; verified it reports dirty by
reverting only the Inventory arm to a bare multiply (inventory scale
26, position scale 0). Also fixed a comment referring to a length check
that the previous commit replaced with a scale assertion.