Proposal: Self-hosted in-cluster registry for air-gapped Cozystack (bundle bootstrap → in-cluster source of truth) #3029

TL;DR

• Problem: air-gap in Cozystack today = bring your own registry (creating mirrors is
  explicitly out of scope). And it's not just OCI images — Talos OS assets must reach
  both the management (root) cluster and the tenant (leaf) clusters, via different
  channels.
• Proposal: ship a Cozystack offline bundle (OCI images + Talos assets) → bootstrap
  from a throwaway registry on the admin machine → stand up a self-hosted in-cluster
  registry that becomes the source of truth after a one-time cutover.
• Redirection: reuse the existing containerd registry mirrors (works for system +
  node + tenant images). Admission policy (native CEL > Kyverno) can't see node/system
  pulls, so it's not the mechanism — Kyverno is optional, for phase-2 signature checks.
• Scope: PoC. Phase 1 = TLS + baked-in root CA on Talos-owned bundles
  (paas-full/distro-full); signing/provenance/encryption = phase 2. Tenant
  self-service registry is a separate concern (existing Harbor app).
• Open ends: bootstrap chicken-and-egg, the HTTPS/CA dance, paas-hosted (no node
  control), and the day-2 upgrade ordering.

Summary

Today Cozystack can be installed air-gapped, but the model is "bring your own
registry": the admin must stand up and populate external private registry mirrors, and
the docs state outright that creating such mirrors is out of scope. Image redirection
is done via Talos/containerd machine.registries.mirrors, which the platform already
exposes as a registries.mirrors value and propagates to tenant clusters.

This proposal adds the missing pieces so that air-gap becomes a first-class, batteries-
included experience:

1. A Cozystack-produced offline bundle (a single ~4–10 GB artifact) carrying both
   classes of artifact the install needs: OCI container images and Talos OS assets
   (boot media for bare-metal nodes + the VM disk image for tenant worker nodes).
2. A throwaway local registry on the admin's machine used only to bootstrap.
3. A self-hosted in-cluster registry that, once the cluster is up, becomes the
   source of truth for all image pulls — backed by the storage Cozystack already runs
   (LINSTOR/Piraeus + SeaweedFS S3), reachable on a dedicated internal domain.
4. Automatic redirection of all image references to that registry, reusing the
   existing Talos/containerd mirror mechanism (with admission policy — Kyverno or native
   CEL — discussed and rejected as the redirection mechanism; see below).

The neat part: image tagging is already standardized (everything is ghcr.io/cozystack/…
and a small set of upstream registries), and the platform.registries.mirrors knob
already exists. We're not inventing redirection — we're shipping the registry those
mirrors point at, plus the tooling to seed it.

Problem statement: what needs delivering, and where

Air-gap for Cozystack is not a single delivery problem. It's a matrix of two
artifact classes × two cluster tiers, and the existing air-gap docs only really address
one cell of it (OCI images via mirrors). Being precise here is the whole point of this
proposal.

The two cluster tiers (Cozystack terminology)

• Management cluster (a.k.a. root cluster) — the base Cozystack cluster, running
  on bare-metal/VM Talos nodes that the operator owns.
• Tenant Kubernetes clusters (the leaf clusters) — provisioned by the management
  cluster: the control plane runs as Kamaji pods inside the management cluster, and the
  worker nodes are KubeVirt VMs that themselves boot Talos (the nocloud profile).

The two artifact classes

1. OCI container images — pulled by containerd at runtime: Kubernetes components, CNI,
   CSI, all Cozystack components, and the Talos installer image (used for installs/
   upgrades). These are redirected via registry mirrors — the part today's air-gap docs
   cover, and what the in-cluster registry below serves.
2. Talos OS assets — the bootable OS itself, which is not a containerd pull and so
   is invisible to registry mirrors / Kyverno / CEL:
   • Bare-metal (management) nodes boot kernel + initramfs + iso/metal raw images,
     served over PXE/HTTP via matchbox (bootbox).
   • Tenant VM worker nodes boot the Talos nocloud disk image, imported as a KubeVirt
     CDI DataVolume (vm-default-images).
     (These profiles already exist in-tree under packages/core/talos/images/talos/profiles/:
     iso, kernel, initramfs, metal, installer, nocloud.)

The delivery matrix (and where each cell is solved)

Why the tasks differ but overlap

• Differ — delivery channel & format. Management Talos assets are PXE/HTTP boot media
  that must exist before any cluster does (bootbox seeded offline). Tenant Talos is a VM
  disk image pulled by CDI after the management cluster is up. Different artifacts,
  different servers, different points in time.
• Overlap — same source, same version, same registry. Both tiers derive from the same
  talos/profiles at the same Talos version; both pull the same installer OCI image; and
  both consume OCI images through the same mirror mechanism pointed at the same in-cluster
  registry. So one bundle + one in-cluster registry can feed both — except the management
  boot assets, which are irreducibly a bootstrap-time, admin-side responsibility.

This matrix is what the rest of the proposal builds on: the bundle must carry OCI
images and Talos assets; the in-cluster registry becomes the source of truth for
everything except the management-node boot media (which only the offline bootbox seed can
provide, because it predates the cluster).

Background: how air-gap works in Cozystack today

• Image references are rewritten at the containerd level via Talos machine config:

  machine:
    registries:
      mirrors:
        docker.io:
          endpoints:
            - http://10.0.0.1:8082
        ghcr.io:
          endpoints:
            - http://10.0.0.1:8083
        # … registry.k8s.io, gcr.io, quay.io, cr.fluentbit.io,
        #    docker-registry3.mariadb.com
      config:
        "10.0.0.1:8082":
          tls:
            insecureSkipVerify: true

• The platform Helm chart exposes the same structure as registries.mirrors /
  registries.config, and forwards it to tenant Kubernetes clusters too (tenant
  containerd config secret), so redirection is consistent platform-wide.

• The registries themselves are the admin's problem. Cozystack provides no bundle and
  no registry; you pre-mirror all images for Talos, Kubernetes, and every Cozystack
  component yourself.

So the redirection plumbing is solid and reusable. What's missing is the registry and
the images.

Goals

• One downloadable artifact gets you from "no internet" to a running Cozystack.
• After bootstrap, the cluster serves its own images; no permanent dependency on the
  admin's laptop or an external mirror.
• Reuse the existing mirror mechanism rather than introducing a parallel one.
• Be explicit and honest about the PoC security posture.

Non-goals (phase 1)

• Image signing / verification (cosign/sigstore, Talos image verification) — phase 2.
• At-rest encryption of the registry — phase 2 (storage-layer encryption is a
  LINSTOR/SeaweedFS concern and can be layered later).
• A tenant-facing artifact registry product (see "Tenant self-service registry" below —
  that's Harbor's job, not this in-cluster mirror's); this proposal is about the
  platform's own image supply.

Proposed design

Rule 1 — Standardized tagging (already true, just leverage it)

Cozystack images are already published consistently to ghcr.io/cozystack/…, plus a
known, finite set of upstream registries (docker.io, registry.k8s.io, gcr.io,
quay.io, cr.fluentbit.io, docker-registry3.mariadb.com). This means the full image
set for a given bundle is enumerable — which is exactly what makes a deterministic
offline bundle possible. Add a cozystack images list --bundle <name> that emits the
pinned digest list (the images/*.tag files in each package are already the source).

Rule 2 — Air-gap needs a private registry

Stipulated. The question is whose and where — and the answer changes before vs. after
bootstrap.

Rule 3 — Bootstrap with a local registry on the admin machine (+ the bundle)

Produce a single offline bundle per bundle-flavor/version, e.g.
cozystack-airgap-paas-full-v1.5.0.tar (~4–10 GB). Build it with cozystack images bundle
(wrapping skopeo/oras copy over the digest list into an OCI layout tarball). The bundle
carries both artifact classes from the matrix above: the OCI images and the Talos
OS assets (kernel/initramfs/iso/metal for bare-metal boot via bootbox, plus the
nocloud disk image for tenant VM nodes). The OCI half seeds the registry; the Talos half
seeds bootbox and the CDI source.

On the admin machine (which does have, or once had, internet to fetch the bundle):

# 1. Run a throwaway local registry
docker run -d --restart=always -p 5000:5000 --name cozy-bootstrap-registry \
  -v /srv/cozy-registry:/var/lib/registry registry:2

# 2. Load every image from the bundle into it (no internet needed)
cozystack images push \
  --bundle cozystack-airgap-paas-full-v1.5.0.tar \
  --to http://ADMIN_IP:5000
#   (equivalent to: skopeo copy --all oci-archive:bundle.tar \
#    docker://ADMIN_IP:5000/<repo> for each image)

# 3. Point Talos machine config mirrors at the admin machine
#    machine.registries.mirrors.{ghcr.io,docker.io,…}.endpoints = [http://ADMIN_IP:5000]
#    machine.registries.config["ADMIN_IP:5000"].tls.insecureSkipVerify = true   # PoC

The admin registry is temporary — it exists only long enough to bring the cluster up.

Rule 4 — Post-bootstrap: in-cluster registry becomes the source of truth

Once core components are up, Cozystack deploys an in-cluster OCI registry as a system
component:

• Implementation: a lightweight registry for the system mirror — CNCF distribution
  (registry:2) or zot — rather than full Harbor, to keep the bootstrap image set tiny
  (Harbor pulls ~8 images and a DB; see chicken-and-egg below). Harbor stays the
  tenant-facing artifact product, not the platform mirror.
• Storage: backed by what's already in-cluster — SeaweedFS S3 as the registry blob
  store, or a LINSTOR/Piraeus PVC. Storage is genuinely not a constraint here.
• Addressing: served on a dedicated internal name, e.g.
  registry.cozy-system.svc (cluster-internal) and/or a stable platform domain like
  registry.<cluster-domain>. The name must resolve both in-cluster (CoreDNS) and
  at the containerd level on every node (this is a subtle requirement — see weak spots).
• Cutover: after the in-cluster registry is populated (seeded from the same bundle, or
  re-pushed from the admin registry), the platform rewrites the mirror endpoints from
  http://ADMIN_IP:5000 to the in-cluster registry, and the admin's laptop can go away.

After cutover, every pull — platform, tenants, future installs of apps — flows through the
in-cluster registry.

Rule 5 — Redirection mechanism: registry mirror (recommended) vs. Kyverno

Two options; I strongly favor the first.

Option A (recommended): containerd registry mirrors with fallback ordering.
This is what Cozystack already uses. containerd tries mirror endpoints in order, so we
can express "prefer internal, fall back to external":

machine:
  registries:
    mirrors:
      ghcr.io:
        endpoints:
          - https://registry.cozy-system   # internal, source of truth
          - https://ghcr.io                 # fallback (only reachable in semi-air-gap)

Pros: works for everything — kubelet pause image, CNI, CSI, Talos system extensions,
and tenant clusters — because it operates below Kubernetes. No admission webhook on the
critical path. Already the supported path.

Option B (rejected as primary): admission-time image mutation (Kyverno or native CEL).
An admission policy could rewrite image: fields to the internal registry. This can be
done two ways:

• Native Kubernetes CEL admission — MutatingAdmissionPolicy (CEL, beta in recent
  K8s) for the rewrite and ValidatingAdmissionPolicy (CEL, GA since 1.30) for
  enforcement. No extra dependency — it's in the API server. This is strictly
  preferable to Kyverno if we go the admission route at all.
• Kyverno — more expressive, but not currently shipped in Cozystack, so it'd be a
  new platform dependency. Its real justification is phase-2 signature verification
  (verifyImages), which CEL cannot do (see Security) — not redirection.

Either admission approach shares the same fundamental blind spot for air-gap: it only sees
Pod specs. It cannot rewrite Talos/containerd-level pulls — the pause image, kubelet,
system extensions, anything pulled before the webhook/policy is ready, or images in tenant
clusters. That's exactly where air-gap matters most (bootstrap + node components). It also
adds an admission-time failure mode to image pulls.

→ Use containerd mirrors for redirection. Admission policy (native CEL preferred over
Kyverno) is at best belt-and-suspenders for workload-level enforcement, not the mechanism.
Kyverno stays optional, pulled in only when we want phase-2 signature verification.

Diagram 1 — Bootstrap → cutover timeline

 TIME --->   bootstrap phase (admin-side source)        steady state (in-cluster source)
 =======================================================|=====================================

 [0] admin machine: load offline bundle
       - docker registry:5000              (OCI seed)
       - seed bootbox/matchbox             (Talos boot media)
              |
              v
 [1] mgmt bare-metal nodes PXE/HTTP-boot Talos
       kernel/initramfs/iso  <-- bootbox          (Talos OS assets, NOT via any registry)
              |
              v
 [2] mgmt containerd mirror  -->  ADMIN_IP:5000  (http, insecureSkipVerify)
       pull core OCI: CNI, CSI, LINSTOR/Piraeus, registry itself...
              |
              v
 [3] storage + core up
       deploy in-cluster registry (distribution/zot) on SeaweedFS/LINSTOR
       populate it from the bundle
              |
              v
 [4] ============================ CUTOVER ============================
       rewrite mirror endpoints, mgmt + tenant containerd config:
          ADMIN_IP:5000   ----->   registry.cozy-system  (TLS + baked-in CA)
              |                                              |
              v                                              v
 [5] admin machine RETIRED                  [6] tenant (leaf) clusters come up:
       (laptop can leave the site)                - Kamaji control-plane pods start
                                                  - KubeVirt VMs boot Talos "nocloud"
                                                    via CDI  (served in-cluster)
                                                  - tenant containerd mirror -->
                                                    registry.cozy-system
                                                  => all images served in-cluster

The single irreversible-ish moment is [4] CUTOVER: until then the admin source is
load-bearing; after it, nothing outside the cluster is. Steps [0]–[1] (Talos boot media)
can only come from the admin side — they predate the cluster.

Diagram 2 — Registry access paths (management vs. tenant)

                    MANAGEMENT (root) CLUSTER   [bare-metal Talos nodes]
   +---------------------------------------------------------------------------+
   |                                                                           |
   |   +-------------------+      blobs    +--------------------------------+   |
   |   | in-cluster OCI    |-------------->| SeaweedFS (S3) / LINSTOR PVC   |   |
   |   | registry          |               | (image blob storage)          |   |
   |   | distribution/zot  |               +--------------------------------+   |
   |   | "registry.cozy-   |                                                    |
   |   |  system"          |<------------+---------------------+                |
   |   | (TLS, baked-in CA)|   OCI pull  |          OCI pull   |                |
   |   +-------------------+             |                     |                |
   |            ^                        |                     |                |
   |            | OCI pull               |                     |                |
   |            | (containerd mirror)    |                     |                |
   |   +--------+----------+   +---------+----------+   +------+-------------+  |
   |   | mgmt node         |   | Kamaji CP pods     |   | KubeVirt VM        |  |
   |   | containerd mirror |   | (tenant api/cm/    |   | = TENANT WORKER    |  |
   |   |  ghcr.io ->       |   |  sched, run here)  |   | (Talos "nocloud")  |  |
   |   |  registry.cozy-   |   +--------------------+   | containerd mirror  |  |
   |   |  system           |                            |  (tenant secret)-> |  |
   |   +-------------------+                            |  registry.cozy-    |  |
   |            ^                                       |  system            |  |
   |    PXE/HTTP boot                                   +---------+----------+  |
   |    (Talos kernel/                                            ^             |
   |     initramfs/iso)                                  CDI imports Talos      |
   |   +-------------------+                             "nocloud" disk image   |
   |   | bootbox/matchbox  |                             (from in-cluster)      |
   |   +-------------------+                                                    |
   +---------------------------------------------------------------------------+
              ^
              | bootstrap ONLY  (dashed: retired after CUTOVER)
   +----------+-------------+
   | admin machine          |
   |  docker registry:5000  |
   |  + offline bundle      |
   |  + seeded bootbox      |
   +------------------------+

  Notes:
  - Tenant (leaf) cluster = {Kamaji CP pods} + {KubeVirt VM worker nodes}. The worker
    VMs physically run ON the management cluster, but pull OCI images through their
    OWN containerd mirror config (the per-tenant containerd secret), not the host's.
  - Talos OS delivery differs by tier: bare-metal mgmt nodes via bootbox/matchbox
    (PXE/HTTP); tenant VM nodes via KubeVirt CDI from the in-cluster source.
  - Everything OCI converges on one endpoint: registry.cozy-system.

How this matches the different Cozystack distributions

This is where the design is not uniformly clean, and we should say so:

For paas-hosted, Cozystack does not control the node containerd, so the
containerd-mirror approach can't be assumed. Options there are weaker: rely on the host
platform's own registry-mirror config (out of our hands), use per-workload registry
overrides / imagePullSecrets, or fall back to Kyverno mutation for workload pods
(accepting the system-image blind spot, since on hosted the base K8s images are the host's
problem anyway). Recommendation: scope phase-1 air-gap to paas-full / distro-full
(Talos-owned), and treat paas-hosted as a documented limitation / later phase.

(Bundle names to be confirmed against current main — derived from the layer templates
iaas/naas/paas/system and the values-isp-{full,hosted,full-generic}.yaml files, read
2026-06-24.)

Tenant self-service registry (separate concern → Harbor)

A predictable question: "what about a tenant pushing their own images in an air-gapped
environment?" That is deliberately out of scope for the platform mirror described here.
The in-cluster registry.cozy-system is the platform's read-mostly source of truth for
Cozystack's own (digest-pinned) artifacts; it is not a place tenants push arbitrary builds.

For tenant self-service, the right vehicle is almost certainly a separate Harbor — and
Cozystack already ships Harbor, both as a system component (packages/system/harbor)
and as a tenant-deployable app (packages/apps/harbor), with RBAC, projects, and
robot-accounts suited to multi-tenant push/pull. So this isn't speculative: the building
block exists, and there is active work on it in the tracker (e.g. hardening issues
#2738, #2736, #2739, #2368 and the image bump #2880); see also #2455 "Support private OCI
registry credentials for platform source", which is adjacent to air-gap pulls.

I could not find a dedicated discussion/issue specifically proposing air-gapped tenant
self-service registry — so this proposal just draws the boundary and defers to Harbor
rather than claiming a design. Concretely, the relationship would be:

• registry.cozy-system (this proposal): platform images + Talos, fed by the bundle,
  consumed by mgmt + tenant containerd via mirrors. Not tenant-writable.
• Harbor (existing app, separate lifecycle): tenant-owned images, tenant RBAC. In
  air-gap it would itself be seeded/populated through the same offline-bundle mechanics,
  but its governance, quota, and access model are Harbor's, not ours.

(Caveat worth flagging to maintainers: the open Harbor security issues above — default
credentials/secret-key — should be resolved before Harbor is leaned on as a security
boundary in an air-gapped, higher-assurance deployment.)

Security — explicitly PoC

We should be blunt in the proposal: phase 1 does not give you image authenticity.

• What phase 1 does protect against: transport-level tampering / MITM if the
  in-cluster registry is served over TLS with a CA that Talos trusts. The platform can
  bake a root CA into Cozystack and into the Talos machine config
  (machine.registries.config[<host>].tls.ca / machine.files), so containerd validates
  the registry cert. That alone meaningfully reduces "image substitution" risk: a pull can
  only succeed against a cert chaining to the trusted root.
• The annoying reality: containerd/Docker really want HTTPS. The clean-but-insecure
  PoC shortcut is tls.insecureSkipVerify: true (fine for the throwaway admin-machine
  registry, not for the in-cluster one). The correct PoC path is: generate the
  registry cert from the baked-in CA, with a SAN matching the registry domain, and trust
  the CA on every node via machine config. No plaintext HTTP for the persistent registry.
• What phase 1 does not do (phase 2): cryptographic image signing/verification,
  provenance/SBOM attestation, and at-rest encryption. Important distinction on tooling
  here: native CEL admission (VAP) can enforce cheap provenance constraints with no
  extra dependency — "image must come from registry.cozy-system", "must be pinned by
  digest, not tag" — but it cannot verify a cryptographic signature (CEL has no crypto
  primitives and can't fetch sigstore attestations). So actual signature verification
  still requires a cosign-aware admitter (Kyverno verifyImages or the sigstore
  policy-controller) or Talos-level image verification. That's the real, optional role
  for Kyverno — phase 2, not redirection. In theory, TLS + a trusted root CA + a closed
  network is enough to keep the system from being fed substituted images for a PoC — but
  it is not supply-chain integrity, and we should not market it as such.

Storage — not a concern (with one caveat)

Cozystack's control plane is stateful and LINSTOR/Piraeus + SeaweedFS already run in every
cluster, so hosting registry blobs is trivial — it's just another stateful workload. The
one caveat is ordering, not capacity: see the chicken-and-egg below — the storage layer
itself must come up before the registry it will back.

HA and disaster recovery

Because the in-cluster registry sits on the critical path for every new image pull, its
availability matters — but the building blocks are already here:

• HA serving. Run the registry (distribution/zot) as a multi-replica Deployment
  behind a Service/VIP, with blobs on replicated storage (LINSTOR/Piraeus replication,
  or SeaweedFS's own replication for the S3 backend). No single node loss should stall
  pulls. Note the registry is stateless compute over shared blob storage, so replicas are
  cheap.
• Blast-radius reminder. Even with HA, a total registry outage only blocks new
  pulls (scale-ups, new app installs, node replacements); already-pulled images on running
  nodes keep their pods alive. So it's a serious-but-not-instant outage — sized like losing
  the platform's storage, which Cozystack already treats as a first-class HA concern.
• Disaster recovery is just re-seed. The registry holds no unique state that isn't
  reproducible — its contents are exactly the (digest-pinned) bundle. If the blob volume is
  lost beyond replication recovery, restore = re-run cozystack images push from the
  bundle. This is the same operation as install/upgrade seeding, so DR needs no separate
  tooling; it just requires that the matching bundle (per installed version) is retained
  off-cluster. Worth stating as an operational requirement: keep the bundle for every
  version you're running.

Day-2: upgrading an air-gapped Cozystack

There is no magic here — and pinning down the non-magic precisely is the point. An
upgrade is: "get the new version's artifacts onto removable media, carry them across the
gap, pre-load them into the in-cluster registry, then bump the version." Unlike the
initial install, there's no throwaway admin registry — you push straight into the live
in-cluster registry while the cluster keeps running on the old images. The ordering is the
whole game: bump first and Flux/Helm will try to pull image digests that don't exist yet →
cluster-wide ImagePullBackOff.

Flow

 CONNECTED side (has internet)        AIR-GAPPED side (admin machine --> cluster)
 ============================         ==========================================
 [A] cozystack images bundle \        [C] verify bundle by checksum / digest list
       --version vNEXT \                   (the only integrity gate in PoC)
       -o cozy-vNEXT.tar                        |
            |                                    v
            v                             [D] cozystack images push \
 [B] write to USB / removable                  --bundle cozy-vNEXT.tar \
     media (the data-ingress point)            --to registry.cozy-system   (TLS+CA, authed)
            |                                   (skopeo/oras sync into the LIVE in-cluster
            |   -- physically carried -->        registry; old images stay untouched)
            |                                    |
                                                 v
                                          [E] push new Talos assets:
                                                - nocloud disk  --> in-cluster CDI source
                                                - reseed bootbox boot media
                                                 |
                                                 v
                                          [F] PREFLIGHT: cozystack images verify --version vNEXT
                                                (assert every digest vNEXT needs is present)
                                                 |
                                                 v
                                          [G] bump platform version / Flux reconcile
                                                --> every pull resolves locally
                                                 |
                                                 v
                                          [H] Talos OS upgrade (if needed):
                                                talosctl upgrade --image \
                                                  registry.cozy-system/.../installer:vNEXT
                                                 |
                                                 v
                                          [I] verify healthy --> GC old images (keep N-1)

Why each step matters / what bites you

• Pre-load before the bump ([D] before [G]). The single most common air-gap upgrade
  failure: reconcile references digests that were never pushed. Make [G] refuse to run
  unless [F] passed.
• Digest-pinned bundle ([A]). The bundle's image list is vNEXT's images/*.tag, so
  what you pre-load is exactly what the upgrade will request — no latest/tag drift.
• Preflight verify ([F]). Cheap insurance against the 2 a.m. ImagePullBackOff: walk
  vNEXT's digest list against the live registry and fail loudly if anything is missing,
  before touching the running platform.
• Talos arrives two ways ([E] + [H]). OS assets aren't OCI, so they need their own
  channels: reseed bootbox for new/replaced bare-metal nodes, and pull the installer
  OCI image from the in-cluster registry for in-place Talos upgrades. Tenant VM nodes pick
  up the new nocloud image via CDI on next roll.
• GC last, keep N-1 ([I]). Do not garbage-collect the previous version until vNEXT
  is verified healthy — the old images in the registry are your rollback. The registry
  grows over time; GC is a deliberate, post-verification step, not automatic.
• USB is the ingress point ([B]/[C]). In PoC the only integrity check is the bundle
  checksum / digest list; phase-2 signing turns this into a real trust boundary. The flow
  is push-only, so a strict one-way transfer (data diode) is compatible.

Optimization: delta bundles

Full bundles are 4–10 GB, but most images don't change across a minor bump. A
cozystack images bundle --since vCUR --version vNEXT that emits only the changed digests
would cut the USB transfer to the actual delta — meaningful when the transport is someone
walking a flash drive between buildings.

Who builds the bundle

Ideally the Cozystack project publishes a per-release air-gap bundle as a release asset
(built in CI from the same pinned digests), so operators download a vetted artifact rather
than hand-rolling one on a connected machine. cozystack images bundle then becomes the
fallback / customization path, not the only path.

Weak spots / open critique (the honest list)

1. Bootstrap chicken-and-egg (the hard one), now in two layers.
   (a) Talos boot media for the management nodes predates everything — there is no
   cluster, no registry, no CDI yet, so those kernel/initramfs/iso/metal assets
   must come from an offline-seeded bootbox/matchbox on the admin side. This cell can
   never be served by the in-cluster registry; it's the true root of the dependency chain.
   (b) Then the OCI core set — the in-cluster registry runs as pods that need images, and
   the storage layer (LINSTOR/Piraeus, CNI, CSI) it depends on needs images before the
   registry exists. So there is an irreducible core image set that must come from the
   bootstrap source (admin registry / bundle), not the in-cluster registry. Documented
   ordering: bootbox seed (Talos assets) → mgmt nodes boot → bundle → admin registry →
   core/storage up → in-cluster registry up + populated → mirror endpoint cutover →
   tenant Talos nocloud + tenant images now served in-cluster → admin registry retired.
   Getting the cutover atomic and safe (no half-migrated nodes) is the real engineering
   risk.
2. The HTTPS/CA dance. As above — the persistent registry must be TLS with a
   node-trusted CA. Distributing/rotating that CA across all Talos nodes and tenant
   clusters, and handling cert SAN/domain correctly, is fiddly and a common source of
   "ImagePullBackOff at 2 a.m." Plain HTTP only acceptable for the throwaway bootstrap
   registry.
3. DNS for the registry domain. The dedicated name must resolve at the containerd
   level on every node, not just in-cluster. On Talos that means CoreDNS and node-level
   resolution — another potential chicken-and-egg if the thing resolving the name itself
   needs an image. Using a Service IP or a fixed VIP may be safer than a DNS name for the
   core mirror.
4. paas-hosted gap. No node-level mirror control → no clean solution; documented
   limitation (above).
5. Single registry = SPOF. A registry outage stalls all new pulls cluster-wide
   (cached images keep running pods alive; scale-ups and new installs block). This is
   addressed — not hand-waved — in "HA and disaster recovery" above (multi-replica over
   replicated storage; DR = re-seed from bundle). Listed here so the risk is explicit, with
   the mitigation cross-referenced rather than repeated.
6. Upgrade flow in true air-gap. New version = new images and new Talos assets, all
   of which must be pre-loaded before the version bump or the reconcile breaks. This is
   significant enough that it has its own section ("Day-2: upgrading an air-gapped
   Cozystack") — the risks are pre-load-vs-bump ordering, preflight verification, Talos
   asset re-seeding, and registry GC/rollback retention.
7. Mirror fallback semantics. "Internal first, external second" only helps in
   semi-air-gap; in true air-gap the external endpoint is dead and just adds pull latency
   on a miss. We should make the fallback list configurable, not assume it.
8. Admission-policy blind spots (Kyverno or native CEL, if anyone pushes Option B
   for redirection): Pod-spec only — can't touch system/node images, races with bootstrap,
   per-cluster burden. Native CEL avoids the dependency cost but not the blind spot.
   Listed so it's not forgotten as a weakness of the alternative.
9. Bundle size & drift. 4–10 GB per flavor/version; multiple flavors and versions
   multiply storage on the admin side. Digest-pinning (from images/*.tag) is essential so
   bundles are reproducible and verifiable by hash even before signing exists.

Phased plan

1. Phase 0 – this discussion: validate direction; confirm bundle names and whether
   maintainers want this in-tree vs. as tooling.
2. Phase 1 – PoC (Talos bundles only): cozystack images list/bundle/push tooling that
   bundles OCI images and Talos assets; offline-seeded bootbox for management boot
   media; throwaway admin registry flow; in-cluster distribution/zot on SeaweedFS/
   LINSTOR; tenant Talos nocloud + tenant images served from the in-cluster registry/CDI
   source; automated mirror-endpoint cutover; TLS with a baked-in root CA;
   insecureSkipVerify only for the bootstrap registry. Target paas-full / distro-full.
3. Phase 2 – integrity: native CEL ValidatingAdmissionPolicy for cheap provenance
   constraints (allowed-registry, digest-pinning) — no new dependency; plus cosign/
   sigstore signature verification via Talos image verification and/or a cosign-aware
   admitter (Kyverno verifyImages / sigstore policy-controller, optional). Provenance/
   SBOM, at-rest encryption.
4. Phase 3 – hosted & tenants: a story for paas-hosted, plus first-class tenant
   cluster seeding from the in-cluster registry; upgrade/GC tooling hardening.

Open questions

1. In-cluster registry implementation: distribution vs. zot vs. reusing Harbor for
   the system mirror too (heavier bootstrap, but one registry to rule them all)?
2. Registry addressing: stable Service/VIP vs. DNS name for the core mirror (DNS
   chicken-and-egg)?
3. Should the bundle be per-bundle-flavor (smaller) or one superset bundle (simpler
   to ship, bigger)?
4. Is scoping phase 1 to Talos-owned bundles acceptable, with paas-hosted deferred?
5. Where should the bundle/seed tooling live — in cozystack CLI, a separate repo, or the
   installer?
6. Should the project publish per-release air-gap bundles as release assets (vetted,
   CI-built), with cozystack images bundle as the fallback/customization path?
7. Are delta bundles (--since vCUR) worth the added tooling complexity for the
   upgrade transfer-size win, or keep it simple with full bundles only?

Asks

• Is there interest in evolving air-gap from "bring your own registry" to a
  Cozystack-provided bundle + self-hosted in-cluster registry?
• Agreement that the containerd-mirror mechanism (not admission policy — neither
  Kyverno nor native CEL) is the right redirection layer, with admission/CEL reserved for
  optional phase-2 policy enforcement?
• Is the explicitly-PoC security posture (TLS + baked-in CA now, signing later) acceptable
  as a first step?
• If yes, I'm happy to open a kind/feature issue and prototype the bundle tooling +
  in-cluster registry for paas-full.