Repository navigation
Add covering inputs as @example(...)s in the code #13
Description
Here's a neat workflow, combining the benefits for PBT and fuzzing with deterministic tests:
- Use the fuzzer to find a reasonably diverse set of covering examples (already works)
- Automatically edit them into the code as explicit
@example(...)cases (this issue!) - Run your standard CI with only full-explicit deterministic examples (already works; see also GH-86275: Implementation of hypothesis stubs for property-based tests, with zoneinfo tests python/cpython#22863)
So what will it take to have automatically-maintained explicit examples? Some quick notes:
- This only works for test cases which can be written using the
@example()decorator, which rules out stateful tests or those usingst.data(). We'll also have trouble with reprs that can't be eval'd back to an equivalent object - we might get a short distance by representing objects fromst.builds()as the result of the call (also useful for Explaining failing examples - by showing which arguments (don't) matter HypothesisWorks/hypothesis#3411), but this seems like a fundamental limitation. - We need to know where the test is, and how to insert the decorator. Introspection works, albeit with some pretty painful edge cases we'll need to bail out on, and I think
LibCSTshould make the latter pretty easy - we can construct a string call, attempt to parse it, and then insert it into the decorator list. - My preferred UX for this is "HypoFuzz dumps a
<hash>.patchfile and the user doesgit apply ...". We can dump the file on disk, and also make it downloadable from the dashboard for remote use. The patch shouldn't be too ugly, e.g. one line per arg, but users are expected to run their choice of autoformatter. - I mentioned "automatically-maintained": it'd be nice to remove previously-covering examples when the set updates; or crucial if we haven't shrunk to a minimal covering example (and currently we don't!). This probably means using magic comments to distinguish human-added examples from machine-maintained covering examples. Note that fuzzer-discovered minimal failing examples might be automatically added to the former set!
This seems fiddly, but not actually that hard - we already report covering examples on the dashboard, after all. No timeline on when I'll get to this, but I'd be very happy to provide advice and code review to anyone interested in contributing 🙂
Metadata
Metadata
Assignees
Labels
No labels
Activity
mentalisttraceur commentedon Nov 15, 2022
So if I understand correctly, the big-picture benefit is to work better with graceful degradation of
hypothesistests to just plain unit tests with hard-coded cases?By automatically providing those hard-coded cases in a form accessible to stub implementations of the
hypothesisAPIs which want to stay so simple that they can't even read the example database?(Besides working around any hassle with making the example database available everywhere, which to me seems better solved by providing a separate solution with great UX for hosting/distributing/syncing the example database.)
Zac-HD commentedon Nov 15, 2022
Smoothly stepping down to a parametrized unit-test, yes. For some users (e.g. CPython) this is so that they can use a stub implementation; others might be happy to use Hypothesis'
phasessetting which already supports this!At scale, say 100+ people and 1M+ lines of code, it can be really important to have fast and deterministic CI because you're testing for regressions rather than bugs via that workflow above, and can run a separate bug-hunting program 'alongside' your CI system. Otherwise I agree that sharing the example database is almost always going to be a better and easier solution, via e.g. Hypothesis'
RedisExampleDatabaseor just writing your own trivial wrapper around whatever datastore you like.Zac-HD commentedon Apr 26, 2023
Turns out that HypothesisWorks/hypothesis#3631 is a rather nice feature for Hypothesis itself, so we'll ship almost all the code upstream and HypoFuzz can just call into the hooks I left😁
Zac-HD commentedon May 31, 2023
OK, we've shipped all the upstreamable internals in Hypothesis - including removal of
@example()s with a specific tag - and I have a MVP branch up at https://github.com/Zac-HD/hypofuzz/compare/write-patches. Further notes:# or any other generated valuecomments are actually pretty nice; on the other it can be a big of a performance hog at the moment. The branch doesn't included these at the moment..hypothesis/patches/latest-{covering,failing}.patchlocation. To support naive use oftimeout, we'll also need to register a handler forsigterm. Finally, of course, we'll need to document the full GitHub Actions configuration we recommend.