Visitar URL original
ci: switch dead link checker from wget to lychee by BrendanWalsh · Pull Request #2501 · microsoft/SynapseML · GitHub
Skip to content

ci: switch dead link checker from wget to lychee - #2501

Merged
Brendan Walsh (BrendanWalsh) merged 1 commit into
masterfrom
brwals/lychee-link-checker
Feb 14, 2026
Merged

ci: switch dead link checker from wget to lychee#2501
Brendan Walsh (BrendanWalsh) merged 1 commit into
masterfrom
brwals/lychee-link-checker

Conversation

@BrendanWalsh

@BrendanWalsh Brendan Walsh (BrendanWalsh) commented Feb 14, 2026

Copy link
Copy Markdown
Collaborator

Related Issues/PRs

Fixes flaky Check Dead Links GitHub Action (~7% failure rate since Dec 2025).

What changes were proposed in this pull request?

Replaces wget --spider --recursive with lychee, a purpose-built link checker.

Root cause of flakiness: wget blasts 1500+ requests without rate limiting, triggering transient GitHub Pages errors. Every "broken" link reported by CI actually returns HTTP 200 when tested individually. The check found 7 different "broken" URLs across 7 failures — all different, all actually working.

Why lychee:

  • Controlled concurrency (8 parallel requests vs unbounded)
  • Retry with exponential backoff (5 retries, 5s wait)
  • Proper timeout handling (30s per request)
  • Purpose-built for link checking (vs wget which is a downloader)
  • Used by Microsoft docs, Kubernetes, and many major projects

How was this patch tested?

  • Verified all 7 historically "broken" URLs return HTTP 200
  • Analyzed 210 workflow runs: 0% failures Oct-Nov, 3% Dec, 15% Jan, 8% Feb
  • The CI run on this PR itself validates the new lychee configuration

Does this PR introduce any user-facing change?

No

@github-actions

github-actions Bot commented Feb 14, 2026

Copy link
Copy Markdown

Hey Brendan Walsh (@BrendanWalsh) 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

  • fix: Fix LightGBM crashes with empty partitions
  • feat: Make HTTP on Spark back-offs configurable
  • docs: Update Spark Serving usage
  • build: Add codecov support
  • perf: improve LightGBM memory usage
  • refactor: make python code generation rely on classes
  • style: Remove nulls from CNTKModel
  • test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

@BrendanWalsh
ci: switch dead link checker from wget to lychee
36db9d0 
The wget --spider --recursive approach was causing ~7% flaky failures
because it blasts 1500+ requests without rate limiting, triggering
GitHub Pages transient errors. Every 'broken' link actually returns
200 when tested individually.

Lychee provides:
- Controlled concurrency (8 parallel requests)
- Retry with backoff (5 retries, 5s wait)
- Proper timeout handling (30s per request)
- Structured output and better error reporting
- Purpose-built for link checking (vs wget which is a downloader)
@BrendanWalsh
Brendan Walsh (BrendanWalsh) merged commit dfb2fd3 into master Feb 14, 2026
7 checks passed
@BrendanWalsh
Brendan Walsh (BrendanWalsh) deleted the brwals/lychee-link-checker branch February 14, 2026 00:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@BrendanWalsh