fix(ci): retain official Ubuntu mirrors with last-resort fallback - #1279

Summary

Add kernel.org only as a last-resort fallback after the three official Ubuntu mirrors fail for a requested file. This supersedes the initial hard-replacement implementation, which the maintainer rejected. Product code and release metadata are untouched; this branch starts from 62064bcfbafb9dc252f9ff730ceabd59ea49c471.

Original-source hosted retries at that commit still failed on November 11: Main CI failed at 16:46 UTC resolving azure.archive.ubuntu.com while fetching QEMU; Website failed at 16:52 UTC resolving security.ubuntu.com while installing Chromium dependencies. No additional retries were made. runner-images #15296 and Canonical's DNS incident corroborate the outage.

Official-first native APT fallback

The shared .github/scripts/configure-ubuntu-apt-mirror.sh now emits:

http://azure.archive.ubuntu.com/ubuntu/       priority:1
https://archive.ubuntu.com/ubuntu/           priority:2
https://security.ubuntu.com/ubuntu/          priority:3
https://mirrors.edge.kernel.org/ubuntu/      priority:4

The actual file uses a literal TAB between each URI and its metadata. It does not rely on line order. apt-transport-mirror(1) specifies that lower numerical priorities are preferred; equal priorities are randomized; unspecified priorities default last. Debian APT's methods/mirror.cc sorts by priority and supplies alternate URIs in that order.

This is native per-file transport failover, not a global switch after one failed request and not a custom retry workflow. A successful official request stays official. kernel.org is attempted only after the official alternatives for that requested file fail.

Safety and scope

• Preserve both deb822 archive/security stanzas byte-for-byte, including suites, components, and Ubuntu archive signing keys.
• Validate each stanza before writes. Reject missing/duplicate/unexpected source URIs or keyring declarations, guarded-field continuations (including extra URI/key values hidden on following lines), and explicit trust/freshness bypasses. Folded guarded fields deliberately fail closed rather than being interpreted loosely.
• Require all official mirror entries at their known priorities. Permit the fallback only at priority 4, making repeat invocation idempotent. Reject missing officials, unknown mirrors, ambiguous metadata, and preference inversions before mutation.
• Preserve apt signature and date verification. APT::Update::Error-Mode "any" rejects incomplete refreshes rather than silently reusing stale indexes; acquire retries are bounded to one and HTTP/HTTPS timeouts to 20 seconds. Affected install/probe steps have a 10-minute deadline.
• Wire the helper only into Main CI's QEMU probe, Website's Chromium installation, and the release Linux-x64-only QEMU probe. Keep Ubuntu 24.04 for QEMU, qemu-x86_64 -cpu Nehalem dist/hunk --version, compiled watch coverage, playwright install --with-deps chromium, and all browser smoke tests. ARM, macOS, and Windows provisioning remain unchanged.
• No global DNS changes, hard-coded IPs, auth bypasses, security-suite removal, or skipped coverage. Include an empty maintenance Changeset.

Validation

Real native transport, not just fixture text

The tests run /usr/lib/apt/apt-helper against ephemeral loopback HTTP servers and temporary source/state/cache paths. They retain the bootstrap's actual priority metadata while reversing the physical mirror-list order. Downloads are SHA-256 checked. Controlled HTTP 503 responses stand in for unavailable mirrors; these tests prove selection/failover semantics, not recovery of Canonical DNS.

Passed with 23 tests each in isolated containers, read-only checkout mount and --network none (loopback only):

• Ubuntu 24.04, APT 2.8.3.
• Ubuntu 26.04, APT 3.2.0.

Observed exact request sequences:

• Official 1 succeeds: [1]; fallback never requested.
• Official 1 fails, official 2 succeeds: [1, 2]; fallback never requested.
• All official mirrors fail: [1, 2, 3, 4], fallback succeeds.
• All four fail: [1, 2, 3, 4], APT exits 100.

These tests run automatically wherever the native APT helper is present in Linux x64 CI; they skip on hosts without APT. No host apt files were changed. The fixtures also cover realistic two-stanza Noble/Resolute files, idempotence, continuation attacks, missing keyrings/configuration, invalid priorities, and empty/unknown mirror lists.

Other local checks

• bun run test — 5,361 passed, 17 skipped, 0 failed on the Arch host (including the four native-APT skips).
• bun test ./scripts/quality/runner-ubuntu-mirror.test.ts ./scripts/quality/baseline-workflows.test.ts — 22 passed, 4 native-APT cases skipped on the Arch host; all native cases passed in both Ubuntu containers above.
• bun run typecheck, bun run lint, bun run format:check.
• bun run check:docs, bun run check:changelog.
• shellcheck .github/scripts/configure-ubuntu-apt-mirror.sh and shell syntax validation.
• git diff --check.

All reported PR checks passed on revised SHA d73847ac134a386e7a7d685fa485454cf3027d68, without reruns. Website run 43064175560 shows Azure failures followed by successful official archive.ubuntu.com index/package fetches at 17:02–17:03 UTC; no kernel.org fetches appear in the log. Chromium installation succeeded and browser smoke tests finished 109 passed, 1 skipped. This is hosted evidence for that run, not a claim that all official mirrors have recovered everywhere. Linux PR validation 43064175593 also ran and passed all four real native APT transport tests. The initial hard-replacement revision's green CI is not claimed as validation of this official-first revision. Main CI runs only on pushes to main, so actual QEMU provisioning/execution awaits an authorized merge; PR tests cover its wiring. The release workflow is not dispatched.

Ready for Charlie's independent review. Do not merge, move tags, dispatch release, or publish without separate approval.

This PR description was generated by Pi using gpt-6.1-sol