Visitar URL original
Add conformance coverage matrix generator, report, and CI check by maskri17 · Pull Request #542 · cel-expr/cel-spec · GitHub
Skip to content

Add conformance coverage matrix generator, report, and CI check - #542

Open
maskri17 wants to merge 2 commits into
cel-expr:masterfrom
maskri17:stage1-conformance-matrix
Open

maskri17 wants to merge 2 commits into
cel-expr:masterfrom
maskri17:stage1-conformance-matrix

Conversation

@maskri17

Copy link
Copy Markdown
Collaborator

Summary

Adds the CEL Conformance Coverage Matrix (doc/conformance.md), generator tooling (tools/), Bazel targets, and a Stage 1 GitHub Actions consistency & unit test workflow (.github/workflows/conformance_matrix.yml):

  • Conformance Coverage Report (doc/conformance.md): Section-by-section coverage matrix across official CEL runtimes (Go, C++ (Modern), C++ (Legacy), and Java) linked directly to .textproto suites under tests/simple/testdata/, with curated non-conformance notes and callouts.
  • Matrix Generator & Validation Gates (tools/):
    • generate_conformance_matrix.py + generate_conformance_matrix_test.py (unittest.TestCase compatible)
    • conformance_manifest.yaml (pinned runtime release configurations) & conformance_notes.yaml (curated skip/failure explanations and callout blocks)
    • Enforces skip validation, corpus union coverage validation, strict manifest checks, note coverage, and baseline regression checks.
  • Bazel Support:
    • Adds py_proto_library targets under proto/cel/expr/conformance/{proto2,proto3,test}/BUILD.bazel, //tests/simple:simple_testdata, //doc:conformance.md, //tools:generate_conformance_matrix, //tools:generate_conformance_matrix_test, and a dev_dependency = True pyyaml archive in MODULE.bazel (isolated from downstream BCR consumers).
  • GitHub Actions CI (.github/workflows/conformance_matrix.yml):
    • Runs bazelisk test //tools:generate_conformance_matrix_test and bazelisk run //tools:generate_conformance_matrix -- ... --strict_manifest --baseline_markdown doc/conformance.md --check_consistency doc/conformance.md on PRs and pushes affecting conformance testdata, protos, tools/, or doc/conformance.md.

@maskri17
maskri17 force-pushed the stage1-conformance-matrix branch from 084f910 to 77bb388 Compare November 10, 2026 05:36
Comment thread MODULE.bazel Resolved
Comment thread tools/README.md Resolved
Comment thread tools/README.md Resolved
Comment thread doc/conformance.md Resolved
Comment thread tools/conformance_manifest.yaml Resolved
- Bump conformance_manifest.yaml, conformance_notes.yaml, and doc/conformance.md to cel-go v0.33.2, cel-cpp v0.17.0, and cel-java v0.15.0.
- Left-align runtime status columns (:---), emit 80-column wrapped blockquotes, and document auto-appended manifest sections.
- Clarify PyYAML http_archive layout in MODULE.bazel, drop unused suite_py_pb2 in proto/cel/expr/conformance/test/BUILD.bazel, set private visibility in tools/BUILD.bazel, and update tools/README.md Bazel workflows.
- Harden generate_conformance_matrix.py validation, YAML scalar formatting via PyYAML, pipe escaping, and relative workspace path resolution, and remove internal workspace path assumptions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants