Visitar URL original
QuickChick 3.0: scheduled derivation, proof-local testing, Rocq scorers by segevem · Pull Request #436 · QuickChick/QuickChick · GitHub
Skip to content

QuickChick 3.0: scheduled derivation, proof-local testing, Rocq scorers - #436

Open
segevem wants to merge 286 commits into
masterfrom
3.0-release
Open

segevem wants to merge 286 commits into
masterfrom
3.0-release

Conversation

@segevem

@segevem segevem commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

This is a draft of QuickChick 3.0.

Derivation now goes through the scheduled engine. QuickChick Derive (or QC Derive) replaces QCDerive and derives generators, enumerators and checkers, registering their instances. The old engine has been removed, along with DeriveLegacy and the using clauses. QuickChick 3.0 no longer generates correctness proofs for derived instances: the proof tactics recognized only the old engine's terms, so the DerivingProofs tutorial is gone. Earlier releases remain the reference for porting that workflow. The plugin now has a single generic library.

The quickchick tactic keeps its test and helpers local to the proof. A saved test can be rerun without deriving again. Counterexamples are shrunk, and shrunk values must still satisfy the hypotheses.

Schedule selection is exact. Score-based pruning replaces the 1,000-candidate cap; its bounds are proved for the built-in rankings. QC Scorer <name> selects the ranking (density by default, budget_aware or cardinality) and replaces the QC_SCHEDULE_RANKING environment variable. A scorer can also be written in Rocq as a SchedulingPolicy.policy. It is compiled to native code and loaded into the plugin, and it can read the scores of its dependencies. If the policy proves its bound, the search is pruned; if it declares a grouping, independent hypothesis groups are searched separately. SchedulingPolicyLaws provides combinators that come with these proofs.

Derivation also covers more relations:

  • Fixed or repeated inputs are accepted.
  • Outputs can be requested in any order.
  • Repeated or nested outputs are rejected with an explicit error.
  • Mutual types are supported even when a member has no base constructor.
  • Constructor conclusions may apply functions, as in R n (double n) or ev (n + 2).
  • Show and Shrink can be derived for types with explicit parameters.

Large derivations are much faster, mainly because completed dependency derivations are shared and instance checks are cached. Several anomalies are now ordinary errors: goals over abstract types, premises that cannot be scheduled, missing instances, and types with no finite values.

Rocq 9.0 and 9.1 are supported alongside Coq 8.20. CHANGELOG.md lists all the changes, and doc/3.0Migration.md covers migration.

Testing

dune build @install @runtest passes on Coq 8.20 (OCaml 5.4.1 and 5.2.1), Rocq 9.0.1 and Rocq 9.1.1. During large derivations on OCaml 5.2, coqc occasionally aborts with "stack smashing detected", and a rerun succeeds. The same abort has been seen less often on 5.4. Pruned selections were compared with exact search. Verification mode checks every pruning bound against every completed schedule, and some test directories run in that mode.

Known limits

  • Correctness proofs for derived instances are not generated.
  • The cardinality ranking has no pruning bound yet, so it keeps the candidate cap.
  • Rocq 9.2 and later need a coq-simple-io release that supports them.

Issues

Merging this closes:

SchedulingPolicies.md had grown into a log of the scoring research, with
statements that later work made false (no public pruning, no grouped
policies). It now describes the search, the scorers, writing a scorer in
Rocq, the pruning bounds and their proofs, the memo, constructor weights and
the diagnostic variables. The migration guide, derivation notes, README and
CHANGELOG drop references to the removed engine and list the tested versions.
@segevem
segevem marked this pull request as ready for review November 5, 2026 18:51
EConstr.Vars.replace_vars takes the evar map from 8.17, not 8.18. The
bare-Derive plugin's grammar extension shared driver.mlg's name,
QuickChickDerive, which Coq 8.18 rejects when the second plugin loads; it
is now QuickChickBareDerive. The command syntax is unchanged.
The Rocq 9.0 job's dependency resolution now exceeds opam's default
60-second limit and fails before anything is built.
The simple derivers declared each function as a fixpoint over the type,
which Coq rejects for a record or a Variant since neither is recursive.
A function that never calls itself is now written into its instance
instead. Fixes #155 and #285.
localMin ignored the maxShrinks argument and always shrank until no
candidate failed. It now stops once that many successful shrinks have
been made. updMaxShrinks sets the limit, like the other Args updaters.

Fixes #293.
Sample, QuickChick, MutateCheck and the related commands applied
sampleGen, show, quickCheck and friends relying on their implicit
arguments, so they broke when a user changed those with Arguments.
They now apply the constants explicitly, with a hole for each leading
argument.

Fixes #351.
forAllShrinkNonDet repeated the shrink tree it had already evaluated,
so every retry saw the same result and it behaved like forAllShrink. A
shrink candidate whose test discards is now run again with a fresh
seed, up to the given number of times, before it is rejected.

Fixes #372.
Show QuickChickWith with stdArgs and the update functions, list the
analysis field, and document forAllShrinkNonDet.

Fixes #123.
Extraction toggled the warning flags by hand. The flags were not
restored if extraction failed, and the fuzzing command never restored
them at all.

Fixes #215.
The extraction directory command now has the same spellings as the
other QuickChick commands; QCExtractDir still works. Dir is read as an
identifier, so it does not become a keyword. A relative directory is
resolved where the command is run, which also avoids extraction's
warning about relative output paths.
Port the active DependentTest relations to scheduled public derivations. Check results and input behavior instead of the old generated term shape, and add parameterized relations, fixed and repeated inputs, reordered outputs, and explicit unsupported-output checks.

The regression compiles with local plugins and theories on Coq 8.20 and Rocq 9.1. Two random-generator checks each pass 20 trials; deterministic enumeration and checker assertions also pass.
Relations were referred to by their base names, so in a module M2
defining Test, a premise about M1.Test was read as a recursive call of
M2.Test: the derived checker either failed to typecheck or, when the
types matched, checked the wrong relation. Inductives are now named by
their shortest unambiguous name, and helper names spell M1.Test as
M1_Test.

Fixes #311.
Properties built from <->, True or False could not be checked even
when their parts were decidable.
A premise such as In x l, defined by a function rather than an
inductive relation, could only be scheduled as a producer, which has no
derivation, so quickchick failed. It is now checked after its variables
are generated. quickchick also tests a goal whose variables were
introduced, quantifying them and the hypotheses about them again. The
check for abstract types no longer crashes on a hypothesis that
mentions earlier variables.

Fixes #109.
A typing relation with a context Gamma : nat -> ty concludes with
Gamma x and extends the context with fun k => match k with ... in a
premise. The application of a constructor variable now becomes a fresh
variable with an equality premise, as applications of defined functions
already did, and lambdas and matches in premise arguments are renamed,
generated and scheduled as computed arguments that need their free
variables bound.
Derived generators and printers wrote a constructor by its base name.
When another global had that name, such as a definition Bool next to a
constructor Z.Bool, the generated code referred to the other global
and failed to typecheck. Constructors are now written by the shortest
name that denotes them, as inductives already were; Show still prints
the constructor's own name, and Merge declares base names.
A premise written with an alias such as Definition Step := R was taken
for a function-defined proposition, which is only checked, so it could
not produce its arguments and derivation failed. A constant whose body
is an inductive is now read as that inductive.
A conclusion that computed an index with a match, such as an if over a
sumbool, failed with an anomaly: the branches bind their proofs
anonymously. Such a match now becomes a fresh variable with an equality
premise, as a function call does, and anonymous branch binders get
fresh names.
A premise that passes a computed argument, such as an extended context
fun k => match k with ..., to another relation in a mode that needs an
instance check failed with "cannot infer a dependency containing a
binder". The check now takes the argument's free variables.
A derived checker or constrained enumerator that enumerates values of
an unconstrained variable sees only the values up to the size bound.
When the variable's type is infinite, such as a list, finding no
witness among them does not refute the premise, but the checker
answered Some false. A typing relation whose literals take any label
was refuted for well-typed terms this way.

Such an enumeration is now bound through enumIncomplete, which ends it
with None, so exhausting it answers None. Enumerations of finite types
(non-recursive inductives whose fields are finite) stay complete and
still refute; they replace the previous None marker that checkers
added to every enumeration. Generic code cannot know that a type
parameter's enumerator is complete, so a checker over one with an
empty enumerator now answers None.
Comment thread plugin/oldGenLibcppo

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delete?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

4 participants