Visitar URL original
perf(test): shorten hostport and cli-kits test tail by mogul · Pull Request #654 · GSA-TTS/agentic-coding-quickstart · GitHub
Skip to content

perf(test): shorten hostport and cli-kits test tail - #654

Merged
mogul merged 2 commits into
mainfrom
perf/hostport-cli-kits-test-tail
Oct 30, 2026
Merged

mogul merged 2 commits into
mainfrom
perf/hostport-cli-kits-test-tail

Conversation

@mogul

@mogul mogul commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

AI assistance disclosure

Authored by an AI agent (OpenCode, dispatched by First Mate) and reviewed by two separate adversarial AI reviews (no blockers; non-blockers fixed). Human review requested; no human approval yet.

Context

The bash 3.2 CI job takes 8.5 to 16 min for the same code (1.7x runner spread). A few tests set the wall time. Root causes, measured locally:

Test Cause
hostport(A) x4, publish(B) WITHOUT_PORTS, cli-kits provision persists An in-process acq_backend_provision runs tens of thousands of commands. bats runs @test bodies under set -T with a DEBUG trap that captures a stack trace before every command, and subshells inherit it. Same provision: 12.1s under bats vs 2.0s outside.
help: every acq-owned verb 15 sequential acq VERB --help; each spends ~1s on exit in the kit-fetch guardian (wait on its 1s sleep tick after USR1)
kit fetch lifecycle 21 serial python cases, each ~1s for the same guardian teardown

Production code and defaults are unchanged (no changes outside test/bats).

Plan / Changes (test/bats only)

  • helper.bash: new acq_untrace_subshell (set +T; trap - DEBUG), called first in the provision subshells of 113-host-port-override.bats (4) and 95-kit-ref-persistence.bats (2). errexit/ERR trap still apply; only failure-location tracing inside those subshells is lost.
  • 60-help.bats: run the 15 --help invocations concurrently, then assert each (exit status, header, no banner) individually.
  • kit-fetch-lifecycle.py: same 21 cases, 4 at a time (ThreadPoolExecutor), every failure reported, failure-only ceilings raised (they cannot slow a passing run).

Before / after (local, 2 CPUs, same machine, 3 runs each, seconds)

Test / file Before After
hostport: two sandboxes same kit, different ports 29.5 / 24.1 / 22.0 5.5 / 6.5 / 5.6
hostport: explicit host: honored 15.0 / 16.8 / 11.7 2.3 / 2.5 / 3.5
hostport: two kits collapse to one -p 16.1 / 12.8 / 11.7 2.3 / 2.4 / 4.6
hostport: unprobeable host, kit-pinned 19.5 / 11.8 / 11.6 3.5 / 2.5 / 3.4
publish(B): WITHOUT_PORTS 11.8 / 9.6 / 8.6 2.4 / 3.4 / 2.4
cli-kits: provision persists --kit 14.7 / 12.6 / 7.8 3.4 / 3.6 / 4.2
help: every acq-owned verb 17.4 / 17.2 / 16.6 4.0 / 3.3 / 2.5
kit fetch lifecycle (test) 7.1 / 7.7 / 10.5 (see note) 5.5 / 4.7 / 5.4
113 file 210 / 155 / 145 72 / 75 / 84
95 file 27 / 22 / 16 10 / 11 / 13
60-help file 67 / 64 / 58 47 / 48 / 46
103 file 7.4 / 8.0 / 10.6 5.7 / 4.9 / 5.5

Note: the kit-fetch baseline was measured while another agent loaded the machine and the test also failed there (flaky under load, pre-existing). Measured unloaded, back to back: 23.2s before vs 2.3-3.4s after.

Verification transcript

  • Full offline suite (bash 5.3, scripts/test-acq-bats, 2 jobs): 1058 ok, 11m46s. The only failure, 117-msys-path-forms test 3 (mkdir: Permission denied), is a sandbox permission issue and fails identically on an untouched copy of origin/main.
  • Touched files run 3x concurrently under 2-way file parallelism (xargs -P 2, same as the suite's fallback): all 3 rounds passed (wall 67s / 58s / 62s). An earlier round with 8 workers and 3s ceilings flaked (exec SIGHUP timeout); fixed by 4 workers + raised ceilings, then re-verified as above. The same flake appeared again with 15s ceilings in two rounds only while a second agent's full suite shared the machine (load 4+ on 2 CPUs); not reproducible on a quiet machine.
  • Real bash 3.2.57 (musl build): 95 and 60-help pass; 113 subset passes. 103 cannot run under the 3.2 shim here because the shim path is not exec-able from python subprocess (environment limitation, not the change); python script itself passed 17/17 cases on 3.2 earlier via the shim in a direct run of the pre-edit version.
  • ShellCheck (0.10.0 available locally, not 0.9.0), --severity=warning: clean on touched shell files. py_compile OK, git diff --check clean. pre-commit itself could not run locally (no pre-commit/network in sandbox); CI must confirm.

Mutation checks (each test still fails when its behavior is broken)

  • SAME kit different ports: forcing the chosen host to the guest port fails the test (before and after).
  • explicit host honored: emitting guest:guest instead of host:guest fails.
  • two kits collapse: dedupe print made unconditional fails.
  • kit-pinned unprobeable: removing the unknown-probe notice fails.
  • WITHOUT_PORTS: removing the _portrecs="" drop fails.
  • cli-kits persists: disabling acq_cli_kits_write in provision fails.
  • help: wrong header for stop, version, non-zero exit, banner leak each fail.
  • kit fetch lifecycle (identical verdict before/after on 8 mutations): guardian ignores USR1, no cleanup signal, no parent-liveness check, guardian EXIT rm removed all fail; the other 4 (no parent EXIT trap, no rm in parent cleanup, startup HUP/INT handlers removed) pass both before and after, i.e. were never covered; not made worse.
  • Note: "counter does not advance" mutation of the port picker passes both before and after (the random base differs per process); not a regression.

What was not sped up

  • 113 still ~72s: the acq_child (publish B) tests run real acq child processes (1s guardian teardown + provisioning each, 1.3-3.4s) and malformed values (10s, 4 acq runs). Left alone: they are the 4-5s tail, not the tail that sets wall time, and parallelizing them risks the stub counters.
  • 60-help file still ~47s from many other 1s acq invocations.
  • The 1s guardian teardown itself (production acq) is the shared root of several slow tests; fixing it is a production change and is out of scope.

Rollback

Revert this commit; test-only.

Security impact

None. Test-only; no auth, data handling, network or attack-surface change.

Cut wall time of the slowest offline bats tests without changing what they
assert or any production behavior.

- helper: add acq_untrace_subshell. bats runs tests under set -T with a DEBUG
  trap that captures a stack trace before every command; subshells inherit it,
  and an in-process provision runs tens of thousands of commands (12s traced vs
  2.3s untraced). Used in the provision subshells of 113 and 95.
- 60-help: run the 15 per-verb --help invocations concurrently (each waits ~1s
  on the kit-fetch guardian teardown), then assert each individually.
- kit-fetch-lifecycle.py: run independent cases 4 at a time, report every
  failure, and raise failure-only ceilings so a loaded runner cannot flake.

Co-authored-by: OpenCode Agent <user@gsa.gov>

@basiliskus basiliskus left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: request changes — 2f7a0d6823778d1df3b89dcece1339df9ea47af6

Please remove test/bats/__pycache__/kit-fetch-lifecycle.cpython-314.pyc from the change. It is generated, interpreter-specific bytecode rather than a test source or fixture. Because it is tracked, running python3 -m py_compile test/bats/kit-fetch-lifecycle.py with another Python build rewrites it and leaves an otherwise clean checkout dirty; it also expands this test-performance PR beyond its stated source-only scope. The repository does not currently ignore __pycache__/ or *.pyc, so adding the standard ignore rule would also prevent recurrence.

The concurrency changes otherwise keep per-case assertions and isolate process roots/output. I did not find another correctness, timeout, or failure-reporting regression in the changed source.

Assisted-by: OpenCode [GPT 5.6 Sol Latest Guardrails Defaultv2] skill:change-review

Remove the accidentally tracked CPython 3.14 .pyc for kit-fetch-lifecycle.py and add
standard Python bytecode ignore rules so it cannot recur.

Co-authored-by: OpenCode Agent <user@gsa.gov>
@mogul

mogul commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

🤖: AI agent (First Mate) — addressed the review feedback in 6c78bd6: untracked the generated test/bats/__pycache__/kit-fetch-lifecycle.cpython-314.pyc (it was the only tracked bytecode in the repo) and added __pycache__/ and *.pyc to .gitignore under a new PYTHON section. No other open PR (651/652/653/664) touches those lines. Running the 60-help, 95, 113 and 103 bats files leaves the checkout clean, with no bytecode written. The 103 test's Python helper only runs as a script, so no PYTHONDONTWRITEBYTECODE change was needed. The PR diff vs main now contains no generated files. CI is green: pre-commit, bash 3.2 offline suite, and Conventional Commit Title all pass. AI-assisted change; needs human review.

@basiliskus basiliskus left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: approve — 6c78bd654815e9f1dafc33110d54da93e83cfd25

The prior generated-bytecode finding is resolved: the .pyc is untracked, the net PR diff contains no generated files, and .gitignore:59–60 prevents recurrence. No new correctness, concurrency, timeout, or failure-reporting findings in the changed source.

Minor PR-description correction: “no changes outside test/bats” and “test/bats only” should now mention the .gitignore addition.

Local Bats/full pre-commit and Bash 3.2 execution were unavailable; CI status is successful at this head, but raw CI logs were unavailable here. Performance and author mutation claims were not independently re-measured.

Assisted-by: OpenCode [GPT 6.1 Sol Latest Guardrails Defaultv2] skill:change-review

@mogul
mogul added this pull request to the merge queue Oct 30, 2026
Merged via the queue into main with commit a3acc7b Oct 30, 2026
5 checks passed
@mogul
mogul deleted the perf/hostport-cli-kits-test-tail branch October 30, 2026 16:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants