Visitar URL original
fix(bench): keep benchmark results free of symlinks by 0xAndoroid · Pull Request #2318 · a16z/jolt · GitHub
Skip to content

fix(bench): keep benchmark results free of symlinks - #2318

Merged
0xAndoroid merged 1 commit into
mainfrom
fix/bench-result-symlink
Nov 6, 2026
Merged

0xAndoroid merged 1 commit into
mainfrom
fix/bench-result-symlink

Conversation

@0xAndoroid

Copy link
Copy Markdown
Collaborator

Summary

Fix both failures in the first live calibration run, which compares hosted and local builds of the same source.

The local build failed because fetching from a depth-one checkout imported the commit without its parents or a shallow boundary. git log then failed, leaving GIT_SHORT_HASH and GIT_DATE undefined. Result export failed separately: the profiler created latest_* symlinks inside result/, and the executor refused them.

Changes

  • Fetch cached build snapshots with --depth=1, repairing already-incomplete imports too.
  • Run each profiler in disposable working and temporary directories outside result/. Keep metrics and process logs in the results.

The installed executor needs no reinstall: it already downloads the benchmark scripts for each new job.

Testing

All 50 affected tests passed on Linux:

uv run --no-project python -m unittest \
  scripts.tests.test_e2e_bench scripts.tests.test_e2e_ci \
  scripts.tests.test_e2e_entry -v

The two regression tests fail before the fix and pass after it. No Rust build or live calibration rerun was performed.

Security Considerations

The executor still rejects symlinks in exported results. No proof or verifier code changes.

Breaking Changes

None.

Accepted debt

  • One test requires TMPDIR to equal the working directory; the contract only requires both to stay outside results.
  • Old incomplete imports may remain in cache reflogs. Their effect on later Git garbage collection or repacking is unverified.
  • The existing build.rs still hides git log errors.

Run each profiler in disposable scratch outside the exported results, while keeping metrics and process logs in the result tree. Preserve the executor symlink refusal.

Fetch cached build snapshots at depth one to restore shallow boundaries, including after a prior incomplete import, so the CLI build script can read its commit metadata.
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

📏 PR diff

Note

Actual code changed: 0 lines

🟢 0 added    🔴 0 removed

Tests, docs, fixtures, and helper scripts excluded.

Category 🟢 Added 🔴 Removed
⌨️ Actual code 0 0
🧪 Tests +82 0
📚 Docs 0 0
📦 Fixtures 0 0
🔧 Helper scripts +9 −3
Total diff +91 −3

Every changed line is classified once.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Verifier classifier: risk-1

Head 5b23bc1523d37a6533a0d6f99489354d9ec5b970

In both revisions, changes are confined to benchmark orchestration and tests. Git history repair and temporary profiling directories do not alter production verifier logic, proof inputs, or protocol parameters.

Risk: risk-1

  • scripts/bench/run.py: risk-1 — Benchmark-only infrastructure changes: shallow fetching preserves the selected revision, while temporary cwd/TMPDIR isolate profiling artifacts. Explicit inputs and PCS cache configuration remain unchanged.
  • scripts/tests/test_e2e_bench.py: risk-1 — Adds benchmark regression tests for shallow-checkout repair and temporary artifact cleanup; no production effect.
  • scripts/tests/test_e2e_entry.py: risk-1 — Adds a test that benchmark result fetching still rejects symlinks; no production effect.

gpt-6.1-sol, reasoning effort xhigh

@github-actions github-actions Bot added the risk-1 Tests, scripts, CI or documentation only label Nov 6, 2026
@0xandoroid-ai-agent

Copy link
Copy Markdown

This pull request #2318 has been reviewed at commit 5b23bc1 using skill reviewer and model Claude Opus 5.5 with xhigh reasoning effort, and no issues were found.

@github-actions github-actions Bot added the merge-eligible AI merge gate review policy satisfied for the current head label Nov 6, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The AI merge gate approves 5b23bc1523d37a6533a0d6f99489354d9ec5b970. Review attested by 0xandoroid-ai-agent: An authorized comment attests a clean review of the current head 5b23bc1523d37a6533a0d6f99489354d9ec5b970, so the gate approves it.

@0xAndoroid
0xAndoroid merged commit 135914d into main Nov 6, 2026
43 checks passed
@github-actions github-actions Bot removed the merge-eligible AI merge gate review policy satisfied for the current head label Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk-1 Tests, scripts, CI or documentation only

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant