docker save gives Error response from daemon: no suitable export target found: #7278

Activity

1. iraj-jelo

   added

   kind/bug

   status/0-triage

   on Sep 2, 2026
2. thaJeztah commented on Sep 2, 2026

   

   thaJeztah

   on Sep 2, 2026

   Member

   More actions

   What version of containerd is running inside the container? There may be some issues if the platform is not the native platform, but trying to reproduce, I don't get the error;

   docker pull --platform=linux/amd64 kserve/llmisvc-controller:v0.20.0
   docker run -d --rm --name dind --privileged docker:dind
   docker save --platform="linux/amd64" "kserve/llmisvc-controller:v0.20.0" | docker exec --privileged -i "dind" ctr --address /var/run/docker/containerd/containerd.sock --namespace=k8s.io images import --snapshotter=overlayfs -
   docker.io/kserve/llmisvc controller:v0.2	saved
   application/vnd.oci.image.manifest.v1+json sha256:4bf78942d5009b2684decfe68d8f1c904b4a0f41711ec750869f16df0fd7ef29
   Importing	elapsed: 0.8 s	total:   0.0 B	(0.0 B/s)

   docker exec --privileged -i "dind" ctr --address /var/run/docker/containerd/containerd.sock --namespace=k8s.io images ls
   REF                                         TYPE                                       DIGEST                                                                  SIZE     PLATFORMS LABELS
   docker.io/kserve/llmisvc-controller:v0.20.0 application/vnd.oci.image.manifest.v1+json sha256:4bf78942d5009b2684decfe68d8f1c904b4a0f41711ec750869f16df0fd7ef29 46.9 MiB -         -
   time="2026-09-02T16:01:52Z" level=error msg="failed resolving platform for image docker.io/kserve/llmisvc-controller:v0.20.0" error="content digest sha256:76953e89391382ee3fbecb0b41c1d87e45eb86f796c36eaccafcf73c90ca5f66: not found"
3. thaJeztah

   added

   status/more-info-needed

   containerd-integrationIssues and PRs related to containerd integrationIssues and PRs related to containerd integration

   version/29.7

   on Sep 2, 2026
4. mukeshbhandarkar commented on Sep 4, 2026

   

   mukeshbhandarkar

   on Sep 4, 2026

   More actions

   @thaJeztah, @iraj-jelo I dug into this a bit further, and I think we can narrow down what to check next.

   The docker save --platform=linux/amd64 error comes from Moby's single-platform export selection path. For the requested platform, Moby only accepts the manifest if the manifest, config, and required layers are readable from the local content store.

   I also mapped the digest from the later ctr images ls error:

   sha256:76953e89391382ee3fbecb0b41c1d87e45eb86f796c36eaccafcf73c90ca5f66

   That digest is not attestation content. It is the OCI image config referenced directly by the linux/amd64 manifest:

   • index: sha256:d502b5b56713b12d0e04127cdea76bd935402a0d7870d3c30dbf730349d86a95
   • amd64 manifest: sha256:4bf78942d5009b2684decfe68d8f1c904b4a0f41711ec750869f16df0fd7ef29
   • amd64 config: sha256:76953e89391382ee3fbecb0b41c1d87e45eb86f796c36eaccafcf73c90ca5f66

   I tested the relevant selection behavior on both current Moby HEAD and the exact docker-v29.7.2 release.

   A complete linux/amd64 image is still selected successfully when a sibling BuildKit attestation has missing content. As a control, removing a blob required by the actual amd64 image makes the selector reject that platform as expected.

   So at least from that test, an incomplete attestation sibling does not seem to be what is causing the platform-not-found error.

   One detail that seems important: in @thaJeztah's reproduction, docker save and ctr images import both completed successfully before the missing 76953e... config was reported by ctr images ls. That means the missing-config message there is from the destination containerd state; it doesn't by itself show that the config was missing from the reporter's source Docker Desktop daemon before docker save.

   @iraj-jelo, could you run these immediately before the failing docker save?

   $image = 'kserve/llmisvc-controller:v0.20.0'
   
   docker image inspect `
     --platform=linux/amd64 `
     --format '{{json .ManifestDescriptor}}' `
     $image
   
   docker image ls --tree $image
   
   docker image save `
     --platform=linux/amd64 `
     --output llmisvc-controller.tar `
     $image

   In particular, it would be useful to know whether the platform-specific inspect succeeds immediately before save, and what image ls --tree reports for linux/amd64.

   If inspect --platform=linux/amd64 succeeds but the immediately following save --platform=linux/amd64 still reports that the image does not provide that platform, that would be especially interesting because both operations should be able to read the amd64 config at that point.

   There is also a lower-level containerd content-store check we could do if necessary, but since Docker Desktop's internal containerd interface is not a stable user-facing API, I think the Engine-level checks above are a better first step.
5. iraj-jelo commented on Sep 4, 2026

   

   iraj-jelo

   on Sep 4, 2026

   Author

   More actions

   @thaJeztah thanks so much for your help. you mean the version of containerd running inside the container of kserve/llmisvc-controller:v0.20.0? actually I don't know. I'm going to run that container indirectly inside a Kind cluster as a service of Kubeflow services.
   I tried to run the following command in the respective node to inspect containerd version for the pod container, but it failed:

   # crictl exec 9c147df66c31a "containerd --version"
   FATA[0000] execing command in container 9c147df66c31a: Internal error occurred: error executing command in container: failed to exec in
   container: failed to start exec "b753362d8e8dfbb0bc283f345a9e8d955a3758a7448ec28cfee5dff2ef9dc1b1": OCI runtime exec failed: exec
   failed: unable to start container process: exec: "containerd --version": executable file not found in $PATH
6. iraj-jelo commented on Sep 4, 2026

   

   iraj-jelo

   on Sep 4, 2026

   Last edited by iraj-jelo

   Author

   More actions

   @mukeshbhandarkar , thanks for your help! yes, of course.

   Your first suggested command failed due to the missing key "ManifestDescriptor" in json response:

   $ docker image inspect   --platform=linux/amd64   --format "{{ json .ManifestDescriptor}}"   $image
   
   template parsing error: template: :1:8: executing "" at <.ManifestDescriptor>: map has no entry for key "ManifestDescriptor"

   The whole response of docker image inspect --platform=linux/amd64 $image without formatting is like this:

   [
       {
           "Id": "sha256:4bf78942d5009b2684decfe68d8f1c904b4a0f41711ec750869f16df0fd7ef29",
           "RepoTags": [
               "kserve/llmisvc-controller:v0.20.0"
           ],
           "RepoDigests": [
               "kserve/llmisvc-controller@sha256:d502b5b56713b12d0e04127cdea76bd935402a0d7870d3c30dbf730349d86a95"
           ],
           "Comment": "buildkit.dockerfile.v0",
           "Created": "2026-08-07T00:13:53.486986239Z",
           "Config": {
               "User": "65532",
               "Env": [
                   "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                   "SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt"
               ],
               "Entrypoint": [
                   "/manager"
               ],
               "WorkingDir": "/home/nonroot"
           },
           "Architecture": "amd64",
           "Os": "linux",
           "Size": 49214977,
           "RootFS": {
               "Type": "layers",
               "Layers": [
                   "sha256:50abe06dfc0957e14cb8332ed242e8b884ef2563a9eb202e5172f243f62792f7",
                   "sha256:621c35e751a51a9a9dc3e80aa0b7fe8be2a93402ea6ccd307d30852cd7776cda",
                   "sha256:c8b007d0206e4b10ed4d3b3d99dfeab47c2648e82011989fd78a5731baf33fc3",
                   "sha256:bec7e6bb35e05d1284f28b10d2150c259717d91c658c4c10c08424bb9466caba",
                   "sha256:275a30dd8ce958b21daa9ad962c6fbc09f98306ee2f486b65c9075dc257b1412",
                   "sha256:4d049f83d9cf21d1f5cc0e11deaf36df02790d0e60c1a3829538fb4b61685368",
                   "sha256:af5aa97ebe6ce1604747ec1e21af7136ded391bcabe4acef882e718a87c86bcc",
                   "sha256:6f1cdceb6a3146f0ccb986521156bef8a422cdbb0863396f7f751f575ba308f4",
                   "sha256:bd3cdfae1d3fdd83a2231d608969b38b82349777c2fff9a7c12d54f8ac5c9b38",
                   "sha256:4cde6b0bb6f50a5f255eef7b2a42162c661cf776b803225dcac9a659e396bb6b",
                   "sha256:ad51d0769d16ba578106a177987dfe3d2e02c1668c852b795b2f6b024068242a",
                   "sha256:187cfc6d1e3e8a40a5e64653bcd3239c140807dcf1c09e48021178705a5a6139",
                   "sha256:5fd2536c39c0700be8b7b4344e375196da2f126842fd8ede66996a18860a3890",
                   "sha256:06f079ef312dd96058e826c7df662acc6b009c6f996352fb265e8d35462db698",
                   "sha256:e44323d2dcb3ab81db82ffc82bc76eb661e3f1ae8bed11bb2e177fb797862909"
               ]
           },
           "Metadata": {
               "LastTagTime": "2026-09-02T17:16:54.990178764Z"
           },
           "Descriptor": {
               "mediaType": "application/vnd.oci.image.manifest.v1+json",
               "digest": "sha256:4bf78942d5009b2684decfe68d8f1c904b4a0f41711ec750869f16df0fd7ef29",
               "size": 3129,
               "platform": {
                   "architecture": "amd64",
                   "os": "linux"
               }
           },
           "Identity": {
               "Pull": [
                   {
                       "Repository": "docker.io/kserve/llmisvc-controller"
                   }
               ]
           }
       }
   ]

   and the response of your second suggested command (for this one, also check the screenshot I upload earlier in Additional Info section of my report. it seems suspicious):

   $ docker image ls --tree $image
   
   IMAGE                               ID             DISK USAGE   CONTENT SIZE   EXTRA
   kserve/llmisvc-controller:v0.20.0   d502b5b56713        184MB         49.2MB
   ├─ linux/amd64                      4bf78942d500        184MB         48.4MB
   ├─ linux/arm/v7                     be6453e834ba           0B             0B
   ├─ linux/arm64                      e0963c6adc62           0B             0B
   ├─ linux/ppc64le                    ff7e02dc8f4e           0B             0B
   └─ linux/s390x                      dcfa3ce26f74           0B             0B
   

   $ docker image save \
     --platform=linux/amd64 \
     --output llmisvc-controller.tar \
     $image
   Error response from daemon: no suitable export target found: image with reference kserve/llmisvc-controller:v0.20.0 was found but does not provide the specified platform (linux/amd64)
7. mukeshbhandarkar commented on Sep 4, 2026

   

   mukeshbhandarkar

   on Sep 4, 2026

   More actions

   @iraj-jelo Thanks, the latest output narrows this down significantly.

   The important observation is that, immediately before the failure:

   • docker image inspect --platform=linux/amd64 successfully resolves the AMD64 manifest (sha256:4bf78942d500...) and its image metadata/RootFS;
   • docker image ls --tree also sees the AMD64 platform;
   • but docker image save --platform=linux/amd64 rejects that same platform.

   Looking through the containerd image-store paths, these operations have different completeness requirements. Platform-specific inspect can return metadata for an image even when some of its compressed layer content is unavailable locally. Explicit-platform save is stricter: the selected manifest, config, and layer blobs must all be available for export.

   That makes this consistent with an incomplete local content closure rather than the AMD64 platform actually being absent.

   There is also a potentially related containerd behavior tracked in containerd/containerd#8973: during pull+unpack, an existing chain-ID snapshot can cause fetching of the corresponding compressed blob to be skipped. The unpacked image can remain usable while operations that require the original packed content later fail. Moby #52193 appears to report a very similar Docker Desktop/containerd-image-store pull → save failure.

   I don't think we have enough evidence yet to say that this is definitely the cause here, though. In particular, we have not identified a missing AMD64 layer digest on your daemon.

   I would avoid removing/re-pulling the image for now, since that could destroy the state that makes this reproducible.

   If a maintainer has a preferred supported way to inspect per-manifest content availability on Docker Desktop/Windows, that would be the most useful next diagnostic.