Visitar URL original
CI: migrate `ecr.aws` images to GHCR (to avoid the ECR rate limit) · Issue #5293 · containerd/nerdctl · GitHub
Skip to content

CI: migrate ecr.aws images to GHCR (to avoid the ECR rate limit) #5293

New issue
New issue
Open
#5326
Open
CI: migrate ecr.aws images to GHCR (to avoid the ECR rate limit)#5293
#5326
Assignees
mainguyen0112
Labels
area/cie.g., CI failurepriority/high

Description

@AkihiroSuda
AkihiroSuda
opened on Oct 27, 2026

nerdctl/pkg/testutil/images.yaml

Lines 86 to 97 in df3d4d4

ffmpeg_soci:
ref: "public.ecr.aws/soci-workshop-examples/ffmpeg"
tag: "latest"
# Large enough for testing soci index creation
ubuntu:
ref: "public.ecr.aws/docker/library/ubuntu"
tag: "23.10"
coredns:
ref: "public.ecr.aws/eks-distro/coredns/coredns"
tag: "v1.12.2-eks-1-31-latest"

CI has been recently failing frequently due to the ECR rate limit

https://github.com/containerd/nerdctl/actions/runs/40471935075/job/122080866973

    container_inspect_linux_test.go:898: 
        
        +------------------------------------------------------------------------------------------------------------+
        | ➡️      | ⚙️ /usr/local/bin/nerdctl build -t testcontainerinspectuser-8d0f9759 /tmp/TestContainerInspectUs |
        |         | er1469321892/001/ab939a5b                                                                        |
        +------------------------------------------------------------------------------------------------------------+
        |         | 🚫 command returned a non-zero exit code                                                         |
        +------------------------------------------------------------------------------------------------------------+
        |         | ⚠️ 1                                                                                             |
        +------------------------------------------------------------------------------------------------------------+
        |         | 🟠 #1 [internal] load build definition from Dockerfile                                           |
        |         | #1 transferring dockerfile: 202B done                                                            |
        |         | #1 DONE 0.0s                                                                                     |
        |         |                                                                                                  |
        |         | #2 [internal] load metadata for public.ecr.aws/docker/library/ubuntu:23.10                       |
        |         | #2 ERROR: failed to copy: httpReadSeeker: failed open: unexpected status from GET request to htt |
        |         | ps://public.ecr.aws/v2/docker/library/ubuntu/manifests/sha256:fd7fe639db24c4e005643921beea92bc44 |
        |         | 9aac4f4d40d60cd9ad9ab6456aec01: 429 Too Many Requests                                            |
        |         | toomanyrequests: Data limit exceeded                                                             |
        |         | ------                                                                                           |
        |         |  > [internal] load metadata for public.ecr.aws/docker/library/ubuntu:23.10:                      |
        |         | ------                                                                                           |
        |         | Dockerfile:2                                                                                     |
        |         | --------------------                                                                             |
        |         |    1 |                                                                                           |
        |         |    2 | >>> FROM public.ecr.aws/docker/library/ubuntu:23.10                                       |
        |         |    3 |     RUN groupadd -r test && useradd -r -g test test                                       |
        |         |    4 |     USER test                                                                             |
        |         | --------------------                                                                             |
        |         | error: failed to solve: public.ecr.aws/docker/library/ubuntu:23.10: failed to resolve source met |
        |         | adata for public.ecr.aws/docker/library/ubuntu:23.10: failed to copy: httpReadSeeker: failed ope |
        |         | n: unexpected status from GET request to https://public.ecr.aws/v2/docker/library/ubuntu/manifes |
        |         | ts/sha256:fd7fe639db24c4e005643921beea92bc449aac4f4d40d60cd9ad9ab6456aec01: 429 Too Many Request |
        |         | s                                                                                                |
        |         | toomanyrequests: Data limit exceeded                                                             |

Activity

  1. AkihiroSuda
    added
    area/cie.g., CI failure
    on Oct 27, 2026

  2. mainguyen0112 commented on Oct 27, 2026

    @mainguyen0112
    mainguyen0112
    on Oct 27, 2026
    Contributor

    Hi, I'd like to work on this issue. I'll check the ECR image references in pkg/testutil/images.yaml and look for suitable GHCR replacements, making sure the image tags and supported platforms are compatible with the existing tests.

    I'll also verify the replacements locally before opening a PR.

  3. AkihiroSuda
    assigned
    mainguyen0112
    on Oct 27, 2026

  4. AkihiroSuda commented on Oct 27, 2026

    @AkihiroSuda
    AkihiroSuda
    on Oct 27, 2026
    MemberAuthor

    @mainguyen0112 Assigning this to you, but public.ecr.aws/soci-workshop-examples/ffmpeg probably needs:

    Other ones can be replaced easily

  5. AkihiroSuda
    added
    priority/high
    on Oct 28, 2026

  6. mainguyen0112 commented on Oct 29, 2026

    @mainguyen0112
    mainguyen0112
    on Oct 29, 2026
    Contributor

    Hi @AkihiroSuda, here's an update on the work.

    I've investigated the three images in pkg/testutil/images.yaml and performed some local validation.

    Ubuntu

    • ghcr.io/stargz-containers/ubuntu:26.04-org successfully builds with the Dockerfile used by TestContainerInspectUser on linux/amd64.
    • The resulting container reports the expected user (test).
    • The candidate also has an ARM64 manifest, although I haven't tested it on ARM64.
    • This changes the Ubuntu version from 23.10 to 26.04.

    CoreDNS

    • The original EKS CoreDNS image supports both AMD64 and ARM64, runs as root, and exposes TCP/UDP port 53.
    • I tested a GHCR CoreDNS candidate (ghcr.io/quenchworks/images/coredns:1.15.0). DNS queries worked, but it defaults to UID 1001 and port 1053, so it isn't a drop-in replacement for the existing test.
    • I think mirroring the original EKS CoreDNS image to GHCR would be safer than changing the test behavior.

    FFmpeg SOCI

    • The original image is linux/amd64 with 11 filesystem layers.
    • Using ORAS, I confirmed that ECR exposes a SOCI index referencing the image digest (sha256:4bd5d3b7...).
    • The SOCI index manifest references 11 additional blobs.
    • A GHCR migration therefore needs to preserve the image, SOCI index, and referenced blobs so that TestRunSoci continues working.

    For CoreDNS and FFmpeg, is there a preferred GHCR namespace where these images and artifacts should be mirrored? I'm happy to validate the resulting mirrors before updating images.yaml.

    Also, would switching the Ubuntu test image to 26.04-org be acceptable, or would you prefer keeping an older Ubuntu version?

    Thanks!

  7. AkihiroSuda commented on Oct 29, 2026

    @AkihiroSuda
    AkihiroSuda
    on Oct 29, 2026
    MemberAuthor

    Ubuntu

    26.04 is fine.
    You can prioritize updating this one over other ones

    CoreDNS
    FFmpeg SOCI

    Can we mirror them in stargz-containers org?

  8. AkihiroSuda commented on Oct 29, 2026

    @AkihiroSuda
    AkihiroSuda
    on Oct 29, 2026
    MemberAuthor

    cc @ktock for stargz-containers org

  9. mainguyen0112 commented on Oct 29, 2026

    @mainguyen0112
    mainguyen0112
    on Oct 29, 2026
    Contributor

    Sounds good to me!

    I've done a preliminary feasibility check:

    • CoreDNS: Multi-platform mirroring with Skopeo looks straightforward.
    • FFmpeg SOCI: ORAS recursive copy looks promising, but we'll need to validate SOCI artifacts and referrer discovery on GHCR.

    @ktock, would hosting these mirrors under stargz-containers be possible? @AkihiroSuda should I wait for confirmation before proceeding?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area/cie.g., CI failurepriority/high

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions