CI: Integration jobs fail at "pre-pull Docker images" when image downloads slow down #2826

Type of problem

CI infrastructure - flaky job failure, not a product bug

Describe the situation

Integration test jobs fail before any test runs, with:

ERROR: Failed to pull the following image(s) after 3 attempt(s): ...
Failed to pre-pull Docker images needed by the test batch

Since 2026-11-04 this has happened five times in eight days, killing up to 24 Integration jobs in one run. The latest nightly MasterCI on antalya-26.6 (run 42931630136) lost all 24 Integration jobs that started between 01:03 and 01:47 UTC.

This issue:

• Fails the job before any test runs, so the shard reports no test results at all
• Isn't caused by the PR or commit under test. The job labels it INFRA itself
• Passes on rerun, or for shards that start later
• Affects every branch that has the pre-pull step: antalya-26.6, antalya-26.8, stable-26.8. The 26.3 branches don't have it

Root cause analysis

This analysis was done with AI assistance (Claude), from the CI job logs and the CI database.

Where the pull stops

None of the failing pulls logs an HTTP error: no 429, 401 or 5xx. The manifest resolves and the layers are listed, but no layer starts downloading. Every layer stays in Waiting for the whole 300 s:

latest: Pulling from library/redis
37fb86c2a767: Pulling fs layer
...
37fb86c2a767: Waiting
e7012240c0fb: Waiting
Pull of redis failed on attempt 1 (300s elapsed)

Why a slow download becomes a failure

The pre-pull step comes from upstream ClickHouse#100039. ci/jobs/scripts/prefetch-integration-test-images:

1. Starts a docker pull for every image in the batch at once. A batch needs 9 to 37 images.
2. Gives each pull 300 s, and retries 3 times.
3. Fails the job if any image misses all three attempts.

The dockerd inside the integration runner downloads 3 layers at a time across all pulls. That's the default, because ci/jobs/scripts/docker_in_docker.sh doesn't set max-concurrent-downloads. The other layers wait in a queue, and each image's 300 s includes that wait. When downloads slow down, the images at the back of the queue time out before they start.

What slows the downloads

Download throughput per job drops by more than 10 times. In the good nightly of 2026-11-07, the whole prefetch took 19 to 76 s per job. In the 2026-11-11 nightly, it ran past 15 minutes and failed.

Load on the shared runner pool contributes, but it doesn't decide the outcome:

The cause of the slowdown is still open. The candidates are:

• The registry proxy. The integration dockerd pulls Docker Hub images through dockerhub-proxy.dockerhub-proxy-zone:5000. Upstream traced the same symptom to its own proxy running out of connections (ClickHouse#108975). The proxy can't explain everything: images from ghcr.io and quay.io, which bypass it, also stall.
• Hetzner outbound network, or a registry.

Upstream runs the same script and has had no pre-pull failures since 2026-10-09. The problem is in our download path, not in the script alone.

Occurrences

Jobs that failed at pre-pull on antalya-26.6, antalya-26.8, stable-26.8, their tags, and PRs against them. Source: gh-data.checks (test_name = '' and test_context_raw LIKE '%pre-pull%'), with the image list checked in job.log.

The table leaves out 84 jobs from 2026-09-13 to 2026-09-21. Those failed only on minio/minio and minio/mc, after Docker Hub removed those images. The clickhouse/minio-minio mirror fixed that, and it's a separate cause.

Expected behavior

The pre-pull step finishes when downloads are slow. The job runs late but doesn't fail.

Actual behavior

When download throughput drops, images wait in the dockerd queue past their 300 s limit. The job fails after about 16 minutes.

Proposed fix

1. Time out the whole stage, not each image. Give the prefetch one limit (for example 20 minutes) and retry only on real errors. Queue time then stops counting against single images.
2. Find what slows the downloads. Check the dockerhub-proxy and network logs for the windows in the table. During the next wave, inspect a stuck runner with ss to see where the stalled connections go.
3. Stagger the nightly MasterCI schedules. antalya-26.6, antalya-26.3 and stable-26.3 start in the same second today. This lowers the risk but doesn't fix the cause.
4. Port ClickHouse#120107. It stops the tests from pulling again the images the prefetch already pulled. It's missing from antalya-26.6, antalya-26.8 and stable-26.8.

ClickHouse#101970 added retry_infra_failures.yml to rerun these failures automatically. It's disabled in this repository, and it only covers PR runs, not MasterCI.