Visitar URL original
feat: automate community workflow step submissions by mnriem · Pull Request #4873 · github/spec-kit · GitHub
Skip to content

feat: automate community workflow step submissions - #4873

Merged
mnriem merged 18 commits into
github:mainfrom
mnriem:mnriem-workflow-step-submissions
Oct 9, 2026
Merged

mnriem merged 18 commits into
github:mainfrom
mnriem:mnriem-workflow-step-submissions

Conversation

@mnriem

@mnriem mnriem commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator

Description

Add a community workflow step submission flow equivalent to the existing extension, preset, and bundle submission automation.

  • Add a [Workflow Step] issue form and a maintainer-triggered workflow-step-submission agentic workflow, including its compiled lock file.
  • Validate submission metadata, release-tag-pinned individual file URLs, per-file SHA-256 digests, documentation, and author testing attestations. Never install, import, execute, review, or audit submitted step code.
  • Restrict generated draft PRs to workflows/step-catalog.community.json and docs/community/workflow-steps.md, preserve release history, and apply validation-passed only after successful PR publication.
  • Wire catalog notifications and update contributor guidance, community documentation, and navigation.
  • Add positive and negative coverage for notification activation and post-publication label handling, including API failures.

This changes repository submission automation only; it does not change CLI commands, built-in step behavior, or catalog installation/trust policy.

Testing

  • Tested locally with uv run specify --help
  • Ran existing tests with uv sync && uv run pytest
  • Tested with a sample project (if applicable)

Validation commands and results:

  • uv sync --extra test — passed; installed test dependencies in this worktree's own virtual environment.
  • uv run specify --help — passed.
  • LC_ALL=en_US.UTF-8 .venv/bin/python -m pytest tests/test_github_workflows.py tests/test_workflow_step_submission.py -q — passed, 140 tests.
  • gh aw compile add-community-workflow-step --no-check-update — passed; generated the committed lock file with zero warnings.
  • gh aw compile add-community-workflow-step --no-check-update --no-emit — passed, zero warnings.
  • git diff --check — passed before commit.

The full pytest suite was not run; focused coverage exercises the changed execution wiring and existing repository workflow checks. No sample-project test is applicable because CLI behavior is unchanged. The new agentic workflow has not been run against a live GitHub submission; compilation and local wiring tests do not establish an end-to-end agent run.

AI Disclosure

  • I did not use AI assistance for this contribution
  • I did use AI assistance (fill in the disclosure below)

AI disclosure: GitHub Copilot using GPT-6.1 Sol, in autonomous execution under user direction with default session settings and no explicit reasoning-effort override, authored the issue form, agentic workflow, tests, documentation, commit, and PR description, and ran the reported validation commands. The workflow lock file was generated by gh aw v0.88.7. No human line-by-line review is claimed.

Add a maintainer-triggered issue submission flow with metadata-only validation, tag-pinned file digests, scoped draft catalog PRs, and contributor documentation.

Assisted-by: GitHub Copilot (model: GPT-6.1 Sol, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 7, 2026 22:33

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The trigger label is unprovisioned, setup failures bypass blocker handling, and validation permits CLI-incompatible step IDs.

4 open findings
What changed in this PR

Adds automation for community workflow-step submissions, aligned with existing catalog workflows.

Changes:

  • Adds issue-form, agentic workflow, and compiled workflow.
  • Adds catalog notification and label tests.
  • Documents submission, validation, and trust policies.
File Description
.gitattributes Marks workflow locks as generated.
.github/​ISSUE_TEMPLATE/​workflow_step_submission.yml Adds submission form.
.github/​workflows/​add-community-workflow-step.md Defines validation automation.
.github/​workflows/​add-community-workflow-step.lock.yml Adds compiled workflow.
.github/​workflows/​catalog-assign.yml Adds submission notifications.
CONTRIBUTING.md Documents submission process.
docs/​community/​overview.md Adds workflow-step catalog links.
docs/​community/​workflow-steps.md Adds catalog and submission guide.
docs/​guides/​agentic-sdlc.md Lists the new automation.
docs/​reference/​workflows.md References community submissions.
docs/​toc.yml Adds navigation entry.
tests/​test_github_workflows.py Extends label-processing coverage.
tests/​test_workflow_step_submission.py Tests workflow wiring and notifications.

🧠 Review effort: Balanced


💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .github/workflows/add-community-workflow-step.md
Comment thread .github/workflows/add-community-workflow-step.md Outdated
Comment thread .github/workflows/add-community-workflow-step.md
Comment thread .github/workflows/add-community-workflow-step.md Outdated
Keep setup failures recoverable, reuse CLI step ID validation, and gate per-file downloads and hashing through a tested repository-owned helper.

Assisted-by: GitHub Copilot (model: GPT-6.1 Sol, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 7, 2026 22:44
@mnriem

mnriem commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator Author

Pushed follow-up commit de4e745 for this review.

Created and verified the upstream workflow-step-submission label. Both metadata setup steps now continue on error so the agent can report environment blockers. Submission identity checks reuse the repository-owned installer.validate_step_id, rejecting Windows device names that passed the original naming regex.

Moved file fetching into a repository-owned helper: submitted URLs remain JSON data, URL/repository/tag/path and digest checks run before fetching, curl receives a direct argument list without a shell, and hashing occurs only after exit zero and HTTP 200. Added coverage for valid and malicious URLs, argument-boundary isolation even when validation is bypassed, download/HTTP failures, digest mismatches, binary extra files, and missing-parser blockers. No submitted step code is imported or executed.

Validation:

  • gh aw compile add-community-workflow-step --no-check-update — passed, zero warnings; regenerated the lock file.
  • LC_ALL=en_US.UTF-8 .venv/bin/python -m pytest tests/test_github_workflows.py tests/test_workflow_step_submission.py tests/specify_cli/workflows/step/test_installer.py -q — 285 passed.
  • git diff --check — passed before commit.

The setup regression failed before the fix because continue-on-error was absent; it now passes. Also reproduced the original regex accepting con while the CLI rejected it; the new submission verifier rejects it. Live end-to-end execution of the agentic submission workflow remains untested. Review threads are left unresolved for the reviewer.

AI disclosure: Posted on behalf of @mnriem by GitHub Copilot using GPT-6.1 Sol, autonomous execution under user direction with default session settings and no explicit reasoning-effort override. The agent authored the fixes, tests, commit, and this summary, and ran the reported checks.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Submission validation does not enforce the CLI’s package limits, and security-sensitive path guards lack negative tests.

4 open findings
2 resolved since last review

🧠 Review effort: Balanced

Comment thread .github/scripts/validate_community_workflow_step.py
Comment thread .github/scripts/validate_community_workflow_step.py
Validate file and directory counts and nesting before downloading, fetch complete packages with a cumulative byte budget, and cover rejected package paths and exact limit boundaries.

Assisted-by: GitHub Copilot (model: GPT-6.1 Sol, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 7, 2026 23:02
@mnriem

mnriem commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator Author

Pushed follow-up commit 64786b1 for this review.

Submission validation now reads the installer’s actual package limits: 512 entries counting files and distinct implicit directories, 32 directory levels, and 50 MiB cumulative bytes. Metadata count and depth checks run before any request. The workflow invokes one complete-package download pass, with each download bounded by the remaining package budget; it publishes the manifest and complete digest mapping only after every file succeeds. No submitted Python is imported or executed.

Added rejection coverage for traversal, absolute/backslash/empty/dot paths, excluded directories/files, case-insensitive required-file aliases, and file/directory collisions. Boundary tests cover exactly 512 entries including directories, exactly 32 directory levels, and totals immediately below, at, and above 50 MiB. A six-times-10-MiB regression confirms the sixth file fails the cumulative budget before hashing. Failed downloads cannot publish a partial or stale manifest.

The two older findings listed in this review are already addressed: the upstream workflow-step-submission label was provisioned and re-verified with gh api repos/github/spec-kit/labels/workflow-step-submission; URL validation, direct-argument isolation for malicious text, nonzero/HTTP failure handling, and prevention of hashing failed downloads are covered by the previous round’s tests and retained in this run.

Validation:

  • gh aw compile add-community-workflow-step --no-check-update — passed, zero warnings; regenerated the lock file.
  • LC_ALL=en_US.UTF-8 .venv/bin/python -m pytest tests/test_github_workflows.py tests/test_workflow_step_submission.py tests/specify_cli/workflows/step/test_installer.py -q — 316 passed.
  • git diff --check — passed before commit.

The three over-limit metadata regressions failed before the fix because no exception was raised; they now pass. Live end-to-end execution of the agentic submission workflow remains untested. Threads remain unresolved for reviewer verification.

AI disclosure: Posted on behalf of @mnriem by GitHub Copilot using GPT-6.1 Sol, autonomous execution under user direction with default session settings and no explicit reasoning-effort override. The agent authored the fixes, tests, commit, and this summary, and ran the reported checks.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Transient HTTP failures can currently be misclassified as submission defects.

3 open findings
1 resolved since last review
Previously missed (2)

In code that hasn't changed since last review

Medium severity Prioritize HTTP status over curl exit 63 classification

.github/​scripts/​validate_community_workflow_step.py:180

Check the HTTP status before classifying curl exit 63 as a file-size defect. curl reports exit 63 while still emitting the response status when an error body exceeds --max-filesize; for example, a 503 response with a 1-byte remaining budget is currently reported as a submission mismatch instead of the required environment blocker. Handle known HTTP statuses first, then apply the size classification for a 200 response, and add a regression case for this combination.

Medium severity Classify HTTP 408 as a blocked timeout

.github/​scripts/​validate_community_workflow_step.py:187

HTTP 408 is a server-side request timeout, but this branch classifies it as a submission defect. That can apply validation-failed for a transient network/service failure even though the workflow's Blocked contract explicitly includes timeouts. Include 408 among blocked statuses and cover it in the failure table.

🧠 Review effort: Balanced

Prioritize known HTTP responses over curl exit codes, treat HTTP 408 as a blocked timeout, and cover status/exit combinations with ordinary and one-byte remaining budgets.

Assisted-by: GitHub Copilot (model: GPT-6.1 Sol, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 8, 2026 11:29
@mnriem

mnriem commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator Author

Pushed commit 1d1cd6a addressing both findings under Previously missed in this review.

Known HTTP responses now take precedence over curl exit codes. HTTP 403/408/429 and 5xx remain Blocked even when an oversized error body causes curl exit 63, including with a one-byte remaining package budget. Exit 63 is a size defect only for HTTP 200; a missing HTTP status is not evidence of a submission size defect. HTTP 408 is now explicitly treated as a blocked timeout. Failed or blocked downloads still cannot reach hashing.

Added regression combinations covering ordinary and one-byte budgets, HTTP 408, transient HTTP responses with exit 63, confirmed 404/redirect responses with exit 63, missing status, and a transport timeout after HTTP 200. Twelve cases failed before the fix and now pass.

The three carried-over findings were checked again: the upstream label exists, the shell/URL/failure coverage added in de4e745 is retained, and the package-path rejection coverage added in 64786b1 is retained. Evidence replies are being posted on those original threads without resolving them.

Validation:

  • gh aw compile add-community-workflow-step --no-check-update — passed, zero warnings; regenerated the lock file.
  • LC_ALL=en_US.UTF-8 .venv/bin/python -m pytest tests/test_github_workflows.py tests/test_workflow_step_submission.py tests/specify_cli/workflows/step/test_installer.py -q — 342 passed.
  • git diff --check — passed before commit.

Live end-to-end execution of the agentic submission workflow remains untested.

AI disclosure: Posted on behalf of @mnriem by GitHub Copilot using GPT-6.1 Sol, autonomous execution under user direction with default session settings and no explicit reasoning-effort override. The agent authored the fixes, tests, commit, and this summary, and ran the reported checks.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Package-path collision validation misses case-insensitive aliases that break installation on Windows.

3 open findings
Previously missed (1)

In code that hasn't changed since last review

Medium severity Detect case-insensitive package path collisions before installation

.github/​scripts/​validate_community_workflow_step.py:134

This collision check is case-sensitive, so metadata such as Helper plus helper/module.py (or Foo.py plus foo.py) passes validation. On Windows those names alias the same path: catalog installation either overwrites one downloaded file or fails when it tries to create a directory over a file, even though this workflow accepted the package. Normalize package paths component-wise with casefold() when checking duplicate and file/directory collisions, and add these aliases to the pre-fetch rejection cases.

🧠 Review effort: Balanced


Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

Compare package file paths component-wise with casefold before downloads, rejecting duplicate file aliases and file/directory conflicts while preserving noncolliding paths.

Assisted-by: GitHub Copilot (model: GPT-6.1 Sol, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 8, 2026 12:08
@mnriem

mnriem commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator Author

Pushed commit 08d79dc addressing the Previously missed case-insensitive package-path collision finding.

The validator now compares paths as tuples of casefold()-normalized components before any download. It rejects duplicate file aliases such as Foo.py/foo.py and file/directory aliases such as Helper/helper/module.py, including nested conflicts and case variants of required-file ancestors. Original file URLs and path spelling are preserved; only collision checks use normalization.

Added seven pre-fetch rejection cases, including reversed declaration order, and four acceptance cases to preserve distinct files, similar-but-distinct prefixes, and nonconflicting files sharing case-aliased directory names. All seven rejection regressions failed before the fix by reaching the curl boundary; they now fail validation before fetching.

The three carried-over findings already have evidence replies on their original threads. The upstream workflow-step-submission label was re-verified for this round; URL-boundary/fetch-failure tests and package-path negative tests remain in the passing suite. Threads are left unresolved for reviewer verification.

Validation:

  • gh aw compile add-community-workflow-step --no-check-update — passed, zero warnings; regenerated the lock file.
  • LC_ALL=en_US.UTF-8 .venv/bin/python -m pytest tests/test_github_workflows.py tests/test_workflow_step_submission.py tests/specify_cli/workflows/step/test_installer.py -q — 353 passed.
  • git diff --check — passed before commit.

Live end-to-end execution of the agentic submission workflow remains untested.

AI disclosure: Posted on behalf of @mnriem by GitHub Copilot using GPT-6.1 Sol, autonomous execution under user direction with default session settings and no explicit reasoning-effort override. The agent authored the fix, tests, commit, and this summary, and ran the reported checks.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The label policy can mark validation as passed before PR publication, and path validation permits case-insensitive aliases of forbidden components.

3 open findings
Previously missed (2)

In code that hasn't changed since last review

Medium severity Case-sensitive forbidden path checks allow excluded aliases

.github/​scripts/​validate_community_workflow_step.py:111

Forbidden package components are compared case-sensitively here, so aliases such as .GIT/config, __PYCACHE__/x.py, and .ds_store pass validation even though they denote the excluded names on case-insensitive filesystems. This undermines the workflow’s portable path guarantees. Case-fold each component before checking the forbidden set, and add negative cases proving these aliases are rejected before any fetch.

Medium severity Agent can bypass conclusion guard by emitting validation-passed

.github/​workflows/​add-community-workflow-step.md:68

The safe-output policy still lets the agent emit validation-passed directly. That bypasses the conclusion step’s publication guard, so the label can be applied before PR creation or remain applied if the PR output later fails—contradicting this workflow’s stated conclusion-only behavior. Remove validation-passed from add-labels.allowed and assert that restriction in the workflow contract test; keep it only in the conclusion step and remove-labels.

🧠 Review effort: Balanced


Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

Preserve the canonical intake fields and Decision listing, adapt automation to PEP 440 versions and archive provenance, reject forbidden path aliases, and restrict validation-passed to the guarded conclusion job.

Assisted-by: GitHub Copilot (model: GPT-6.1 Sol, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 8, 2026 12:50
@mnriem

mnriem commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator Author

Pushed merge commit 4c3053a addressing both previously missed findings and resolving conflicts with upstream main at a63f7ee. GitHub now reports the PR as MERGEABLE.

Forbidden package components are compared with casefold() against the repository-owned installer's exclusion set, so .GIT, __PYCACHE__, and .ds_store aliases are rejected before fetching. Removed validation-passed from the agent's add-labels.allowed policy; it remains available only in the publication-guarded conclusion job and in remove-labels. Source and compiled policy tests enforce that restriction.

Resolved all four conflicts by preserving upstream's canonical issue-form fields, automatic triage-must-have intake verdict, AI disclosure, documentation, and Decision listing. Adapted the automation to the canonical archive/file/checksum fields rather than requiring a separate Proposed Catalog Entry. PEP 440 versions and all three archive formats remain supported, with additional checks tying archive provenance to the submitted repository and release. The Decision catalog entry is unchanged relative to upstream.

Validation:

  • gh aw compile add-community-workflow-step --no-check-update — passed, zero warnings; regenerated the lock file.
  • uv sync --extra test — passed; refreshed the worktree's environment after upstream dependency-manifest changes.
  • LC_ALL=en_US.UTF-8 .venv/bin/python -m pytest tests/test_github_workflows.py tests/test_workflow_step_submission.py tests/specify_cli/workflows/step/test_installer.py -q — 393 passed.
  • git diff --check and git diff --cached --check — passed before commit.
  • git diff --name-only --diff-filter=U — empty before commit.
  • gh pr view 4873 --repo github/spec-kit --json headRefOid,baseRefOid,mergeable — confirmed the published head and MERGEABLE.

Six regression cases failed before the fixes: four forbidden-name aliases reached curl, and the source/compiled agent policies still permitted validation-passed. All now pass. The existing upstream form tests were retained, with the intake-only phase expectation updated for maintainer-triggered automation. Review threads remain unresolved for reviewer verification; live end-to-end agentic execution remains untested.

AI disclosure: Posted on behalf of @mnriem by GitHub Copilot using GPT-6.1 Sol, autonomous execution under user direction with default session settings and no explicit reasoning-effort override. The agent authored the fixes, conflict resolutions, tests, merge commit, and this summary, and ran the reported checks.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Contributor guidance incorrectly describes both workflow activation and the role of the required archive download URL.

5 open findings

🧠 Review effort: Balanced


Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

Comment thread CONTRIBUTING.md Outdated
Comment thread CONTRIBUTING.md Outdated
Document maintainer-label activation and distinguish the required archive download URL from catalog per-file distribution URLs.

Assisted-by: GitHub Copilot (model: GPT-6.1 Sol, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The workflow enforces an undisclosed documentation requirement that can reject submissions satisfying the published contract.

3 open findings
Previously missed (1)

In code that hasn't changed since last review

Medium severity Undocumented validation rejects submissions meeting the published contract

.github/​workflows/​add-community-workflow-step.md:331

This introduces a hard validation requirement that submitters are not told about: the issue form (.github/ISSUE_TEMPLATE/workflow_step_submission.yml:200) and submission guide (docs/community/workflow-steps.md:133-136) enumerate required documentation content but do not require the external document to explain Spec Kit's discovery-only catalog policy. A submission satisfying the published contract can therefore be rejected. Either remove this check or add the requirement to both intake surfaces.

🧠 Review effort: Balanced

Keep discovery-only policy in Spec Kit guidance without rejecting complete external usage documentation for not repeating it.

Assisted-by: GitHub Copilot (model: GPT-6.1 Sol, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 8, 2026 18:06
@mnriem

mnriem commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator Author

Pushed commit 804f35b addressing the previously missed documentation-contract finding.

Removed the hard requirement for submitted usage documentation to repeat Spec Kit's discovery-only community catalog policy. The workflow now explicitly says not to reject otherwise complete usage documentation for omitting that policy. Spec Kit's own community guide continues to describe discovery-only behavior; catalog trust and installation policy are unchanged.

This aligns validation with the published form and guide without imposing a new requirement on third-party authors. Required installation, configuration, outputs, failure/side-effect, and example workflow documentation checks remain.

Validation:

  • gh aw compile add-community-workflow-step --no-check-update — passed, zero warnings; regenerated the lock file.
  • git diff --check — passed before commit.

This is a prompt-only correction, so deterministic tests were not rerun and no test is claimed to prove agent adherence. Live end-to-end agentic execution remains untested. Existing evidence replies on carried-over findings remain available; threads are left unresolved for reviewer verification.

AI disclosure: Posted on behalf of @mnriem by GitHub Copilot using GPT-6.1 Sol, autonomous execution under user direction with default session settings and no explicit reasoning-effort override. The agent authored the prompt change, commit, and this summary, regenerated the workflow, and checked the published documentation contract.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Release-history behavior and shared workflow invariants need deterministic coverage before approval.

3 open findings
Previously missed (2)

In code that hasn't changed since last review

Medium severity Add tests for catalog history migration and missing digest rejection

.github/​workflows/​add-community-workflow-step.md:353

Add deterministic coverage for the version-update path described here. The new helper and tests validate only the submitted current entry and downloads; they never start from an existing catalog entry and verify that the old current release is moved into releases, existing history is preserved, or missing historical digests block publication. Since preserving release history is a promised behavior, enforce it in a repository-owned generated-catalog verifier and cover both successful migration and rejection cases rather than relying only on agent prose.

Medium severity Include new workflow in shared community-submission contract tests

tests/​test_workflow_step_submission.py:29

Include the new workflow in the shared community-submission contract coverage. COMMUNITY_SUBMISSION_WORKFLOWS still enumerates only bundle, extension, and preset, so the common checks for compiled runtime defaults, activation/write permission boundaries, output-file isolation, and complete fail-closed detection wiring silently skip this workflow. The dedicated test checks selected fields but not all of those invariants; split the shared matrix into common versus archive-specific cases if the individual-file fetch flow cannot use every existing test.

🧠 Review effort: Balanced

Follow the preset verifier snapshot/generated pattern to enforce release migration, preserved history, complete existing digests, and publication gating. Include workflow-step in common community contract coverage while retaining archive-specific matrices.

Assisted-by: GitHub Copilot (model: GPT-6.1 Sol, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 8, 2026 18:27
@mnriem

mnriem commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator Author

Pushed commit a3e2261 addressing both previously missed findings, aligned with the existing submission flows.

The repository-owned step verifier now follows the preset verifier's pre-edit snapshot / generated-catalog-check pattern. It consumes the original catalog and complete download receipt, creates the exact expected update, and rejects generated differences before PR publication. Version updates migrate prior current release-specific metadata into releases, preserve existing history, original created_at, optional discovery metadata, top-level schema/catalog URL, and unrelated entries. Missing or malformed current/historical digests block publication for maintainer repair. Downgrades and unauthorized/content-changing same-version repairs fail; generated catalog errors have a distinct repair exit code and never count as successful validation.

The new workflow is now included in COMMUNITY_SUBMISSION_WORKFLOWS, covering all common runtime-default, activation/permission-boundary, scoped draft-PR/output-file-isolation, and full fail-closed detection tests. Archive-only URL/download/checksum tests use a separate ARCHIVE_SUBMISSION_WORKFLOWS matrix; existing cases remain. The established workflow-step exception that only the guarded conclusion may add validation-passed remains explicitly tested.

Added deterministic positive and negative coverage for current-to-history migration, unchanged prior history, missing/invalid digests, history loss or mutation, changed metadata/unrelated entries, new entries, downgrades, authorized metadata-only repairs, invalid download evidence, CLI snapshot/generated success, malformed generated JSON, and stale-snapshot removal on blocked checks. No new issue fields or separate maintainer contract were introduced.

Validation:

  • gh aw compile add-community-workflow-step --no-check-update and gh aw compile add-community-workflow-step --no-check-update --no-emit — passed, zero warnings.
  • LC_ALL=en_US.UTF-8 .venv/bin/python -m pytest tests/test_github_workflows.py tests/test_workflow_step_submission.py tests/specify_cli/workflows/step/test_installer.py -q — 444 passed.
  • uvx ruff@0.15.0 check tests/test_github_workflows.py tests/test_workflow_step_submission.py .github/scripts/validate_community_workflow_step.py — passed.
  • Session-local fully mocked integration, updated to invoke the production snapshot/generated verifier — 6/6 scenarios passed.
  • git diff --check — passed before commit.

The deterministic checks enforce generated catalog contents; agent adherence to the sequence and live Actions/Copilot execution remain untested. Existing evidence replies on carried-over findings remain available; threads are left unresolved for reviewer verification.

AI disclosure: Posted on behalf of @mnriem by GitHub Copilot using GPT-6.1 Sol, autonomous execution under user direction with default session settings and no explicit reasoning-effort override. The agent authored the verifier, workflow/docs changes, tests, commit, and this summary, regenerated the workflow, updated the session-local mock, and ran the reported checks.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Generated catalog read failures are incorrectly reported as environment blockers instead of repairable generation errors.

4 open findings

🧠 Review effort: Balanced

Comment thread .github/scripts/validate_community_workflow_step.py Outdated
Align with the preset verifier by honoring the caller's error type for filesystem and decoding failures. Cover missing, non-UTF-8, directory, and permission failures without changing original-evidence blocker semantics.

Assisted-by: GitHub Copilot (model: GPT-6.1 Sol, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 8, 2026 19:06
@mnriem

mnriem commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator Author

Pushed commit cec582c addressing the generated-catalog read/decoding classification finding.

The step verifier now uses the caller's error_type for filesystem and Unicode decoding failures, matching .github/scripts/validate_community_preset.py's reader behavior. Deleted, non-UTF-8, directory-valued, and permission-denied generated catalogs are repairable GeneratedError outcomes (exit 3); failures reading original catalogs, snapshots, or download receipts remain Blocked (exit 2). No issue-form or workflow contract change was needed.

Added twelve CLI cases covering missing/non-UTF-8/directory paths across generated/original/snapshot/receipt phases and two permission-error cases preserving exception causes. Four generated-file regressions failed before the fix and now pass; the ten blocker cases preserve existing behavior.

Validation:

  • LC_ALL=en_US.UTF-8 .venv/bin/python -m pytest tests/test_github_workflows.py tests/test_workflow_step_submission.py tests/test_community_preset_validation.py tests/specify_cli/workflows/step/test_installer.py -q — 509 passed, including the other submission verifier's suite.
  • uvx ruff@0.15.0 check tests/test_workflow_step_submission.py .github/scripts/validate_community_workflow_step.py — passed.
  • git diff --check — passed before commit.

No workflow prompt/frontmatter changed, so its compiled lock did not require regeneration. Live end-to-end agentic execution remains untested. Existing evidence replies on carried-over findings remain available; threads are left unresolved for reviewer verification.

AI disclosure: Posted on behalf of @mnriem by GitHub Copilot using GPT-6.1 Sol, autonomous execution under user direction with default session settings and no explicit reasoning-effort override. The agent authored the fix, tests, commit, and this summary, compared the preset verifier behavior, and ran the reported checks.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Duplicate JSON fields can pass validation even though the CLI rejects the resulting catalog.

4 open findings
1 resolved since last review

🧠 Review effort: Balanced

Comment thread .github/scripts/validate_community_workflow_step.py Outdated
@mnriem mnriem added the triage-nice-to-have Verdict: evidence-backed fix or greenlit feature — land after review label Oct 8, 2026
Detect duplicate object keys at every depth before snapshot comparison, preserving generated repair, original-evidence blocker, and submission-failure classifications.

Assisted-by: GitHub Copilot (model: GPT-6.1 Sol, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 9, 2026 12:56
@mnriem

mnriem commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator Author

Pushed commit dcfaba0 addressing the duplicate-key catalog validation finding.

JSON parsing now uses an object-pairs hook that rejects duplicate fields at every nesting depth, matching the runtime catalog loader's uniqueness invariant. Generated catalog duplicates produce repairable exit 3, original catalog/snapshot/receipt duplicates remain Blocked exit 2, and ambiguous submission JSON fails with exit 1. No last-value-wins parsing can approve a generated catalog that the CLI rejects.

Added generated-catalog regressions for duplicate top-level steps, step IDs, nested version fields, and checksum entries where ordinary json.loads would return the exact expected snapshot. Added phase-classification cases for duplicates in original catalog, snapshot, receipt, and submission input. All eight cases failed before the fix and now pass.

Validation:

  • LC_ALL=en_US.UTF-8 .venv/bin/python -m pytest tests/test_github_workflows.py tests/test_workflow_step_submission.py tests/test_community_preset_validation.py tests/specify_cli/workflows/step/test_installer.py tests/specify_cli/workflows/step/test_catalog_versions.py -q — 538 passed.
  • uvx ruff@0.15.0 check tests/test_workflow_step_submission.py .github/scripts/validate_community_workflow_step.py — passed.
  • git diff --check — passed before commit.

No issue template or workflow prompt/frontmatter changed; existing submission phase contracts remain and the lock file did not require regeneration. Live end-to-end agentic execution remains untested. Existing evidence replies on carried-over findings remain available; threads are left unresolved for reviewer verification.

AI disclosure: Posted on behalf of @mnriem by GitHub Copilot using GPT-6.1 Sol, autonomous execution under user direction with default session settings and no explicit reasoning-effort override. The agent authored the fix, tests, commit, and this summary, checked the runtime catalog invariant, and ran the reported checks.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The final catalog verifier currently accepts incomplete or mismatched canonical submission metadata.

4 open findings
1 resolved since last review

🧠 Review effort: Balanced

Comment thread .github/scripts/validate_community_workflow_step.py
Compare every required current-entry metadata field against independently captured form values, including archive and repository provenance, compatibility, and optional changelog. Reject incomplete or miscopied entries before generating expected catalog state.

Assisted-by: GitHub Copilot (model: GPT-6.1 Sol, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 9, 2026 14:43
@mnriem

mnriem commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator Author

Pushed commit ea4c437 addressing the incomplete/mismatched canonical metadata certification finding.

Before constructing the expected catalog, the snapshot verifier now requires nonempty string name, description, author, repository, download URL, documentation, and license values matching independently captured canonical form metadata. It also requires requires.speckit_version to match Spec Kit Compatibility and checks optional changelog correspondence. Existing ID/version/verified, file/digest, provenance, and release-history checks remain. The workflow explicitly captures canonical values separately from the constructed entry, following the preset verifier's comparison pattern; no new issue fields were added.

Updated fixtures to represent complete submissions rather than allowing incomplete entries to count as successful examples. Added missing/empty/whitespace/wrong-type/mismatched catalog metadata cases, malformed/mismatched compatibility cases, missing independent canonical values, optional changelog preservation/rejection, and CLI rejection without snapshot creation. The first 47 rejection cases failed before the fix and now pass. The same-version archive-change test still verifies rejection when both copied metadata and canonical input agree on the changed URL.

Validation:

  • gh aw compile add-community-workflow-step --no-check-update — passed, zero warnings; regenerated the lock file.
  • LC_ALL=en_US.UTF-8 .venv/bin/python -m pytest tests/test_github_workflows.py tests/test_workflow_step_submission.py tests/test_community_preset_validation.py tests/specify_cli/workflows/step/test_installer.py tests/specify_cli/workflows/step/test_catalog_versions.py -q — 590 passed.
  • uvx ruff@0.15.0 check tests/test_workflow_step_submission.py .github/scripts/validate_community_workflow_step.py — passed.
  • Session-local mock integration, supplied with independent canonical metadata — 6/6 scenarios passed.
  • git diff --check — passed before commit.

Canonical correspondence does not independently authenticate the original issue or prove license/documentation content checks; those remain agent responsibilities. Live end-to-end agentic execution remains untested. Existing evidence replies on carried-over findings remain available; threads remain unresolved for reviewer verification.

AI disclosure: Posted on behalf of @mnriem by GitHub Copilot using GPT-6.1 Sol, autonomous execution under user direction with default session settings and no explicit reasoning-effort override. The agent authored the verifier/workflow changes, tests, commit, and this summary, updated the session-local mock, regenerated the workflow, and ran the reported checks.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The security-sensitive agentic workflow processes untrusted submissions and has not been exercised end-to-end against a live issue.

3 open findings
1 resolved since last review

🧠 Review effort: Balanced

@mnriem

mnriem commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator Author

Rechecked review 5471651445 against head ea4c437. It contains no new inline implementation finding and confirms the canonical metadata certification finding is resolved. No new commit is claimed for this response.

The three carried-over findings remain addressed by committed coverage and previously posted inline evidence:

  • URL-boundary and fetch-failure tests reject invalid URLs before fetching, demonstrate exact argument isolation, prevent hashing failed downloads, and exercise real curl with hostile user configuration.
  • workflow-step-submission exists upstream and was re-verified with the label API.
  • Negative package-path tests cover traversal, excluded components and case aliases, duplicate file aliases, and file/directory collisions before requests.

Current-head verification:

  • LC_ALL=en_US.UTF-8 .venv/bin/python -m pytest tests/test_workflow_step_submission.py -q — 275 passed.
  • Session-local mocked integration harness — all 6 scenarios passed again, exercising the current production validator, catalog snapshot/generated verifier, compiled activation/policy configuration, and conclusion script. Agent behavior, downloads, GitHub API publication, and actual safe-output execution remain simulated.
  • git diff --check — passed.
  • PR checks — no failing or unfinished checks observed at verification time.

The live end-to-end concern remains open. The user previously selected mocked local testing rather than changing the fork's default branch for a live issue-triggered run. Mock success does not establish real Copilot inference, Actions authentication/network access, the real safe-output executor, or draft-PR publication. Those require a separately authorized live run. No further implementation change or identical Copilot re-review is being requested solely to relabel that limitation.

Threads remain unresolved for reviewer verification. The user authorized posting this evidence without another identical review request.

AI disclosure: Posted on behalf of @mnriem by GitHub Copilot using GPT-6.1 Sol, autonomous execution under user direction with default session settings and no explicit reasoning-effort override. The agent reverified the current tests, session-local mock, upstream label, and CI state and authored this response; no new code change was made.

@mnriem
mnriem merged commit 9fa05b9 into github:main Oct 9, 2026
16 checks passed
@mnriem
mnriem deleted the mnriem-workflow-step-submissions branch October 9, 2026 15:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

triage-nice-to-have Verdict: evidence-backed fix or greenlit feature — land after review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants