📢 CloudStack Kubernetes Flavors — A Reference Guide I Wish I Had When I Started #13638

chunkyen · 2026-07-17T08:20:42Z

chunkyen
Jul 17, 2026

I've been exploring Kubernetes on CloudStack out of personal interest, and one thing that struck me early on was how scattered the information is. Documentation exists for each individual approach (CKS, CAPC, Talos), but there wasn't a single place that compared them side-by-side or walked through the practical gotchas you hit along the way.

So I started building that reference — a repository covering four approaches to running Kubernetes on CloudStack:

1️⃣ CKS (CloudStack Kubernetes Service) — Native CloudStack integration
2️⃣ CAPC (Cluster API Provider for CloudStack) — Infrastructure-as-Code with CAPI
3️⃣ Rancher + CAPC — Managed Kubernetes with Rancher as the management plane
4️⃣ Talos Linux — Minimal, immutable Kubernetes OS (standalone or with Sidero Omni)

Plus two cross-cutting components that apply to every flavor:

CloudStack Kubernetes Provider (CCM) — external cloud controller manager for node labels, load balancers, and firewall rules
CloudStack CSI Driver — persistent storage with dynamic provisioning and snapshots

Each flavor includes architecture analysis, setup guides, and — most importantly — the lessons learned from actually deploying them. Things like the CSI DaemonSet cloud-init-dir issue on Talos, the ClusterClass limitation with CAPC, or the self-signed CA trust dance with Omni. The kind of details that take hours to figure out but are obvious in hindsight.

A significant portion of the work went into air-gapped / offline CKS deployment — including a deep-dive into the ISO build process, container image pre-import workarounds, Cilium manifest patching for offline environments, and the various failure modes you hit when CKS assumes internet access is available. The offline guide (https://github.com/chunkyen/cloudstack-kubernetes-flavors/blob/main/setup/cks/cks-offline.md) documents all of this.

The Rancher + Turtles + CAPC combination was something I figured out on my own — I couldn't find any existing architecture or guide that documented this specific stack working together. It's a good example of how open source technologies can be combined even when there's no official reference architecture, and I hope the guide helps others who want to run this stack without having to piece it together from scratch.

I've also included suggested improvements for the approaches that could benefit from upstream changes:

CKS Improvements (https://github.com/chunkyen/cloudstack-kubernetes-flavors/blob/main/setup/cks/cks-improvements.md) — pre-import images during upgrades, restrictive default firewall rules, ISO build script fixes, and more
CAPC + Rancher Turtles Improvements (https://github.com/chunkyen/cloudstack-kubernetes-flavors/blob/main/setup/rancher-turtles-capc/capc-rancher-turtles-improvements.md) — dynamic credential injection, ClusterClass support, etcd backup automation, and other proposals

What's in the repo:

🏗 Architecture docs for each flavor and cross-cutting components
📖 Step-by-step setup guides
⚠️ Known issues and workarounds
💡 Suggested improvements for upstream projects
📊 Quick comparison table (complexity, GitOps, upgrades, multi-cluster)

How it was built: The entire repository was developed collaboratively with AI agents (Hermes / OpenCLAW) running on locally hosted models (Qwen, Gemma, etc.) — a testament to how far open-source LLMs have come for technical documentation and infrastructure work.

🔗 https://github.com/chunkyen/cloudstack-kubernetes-flavors

This is a lab project driven by curiosity, not production experience. If you're also exploring Kubernetes on CloudStack, I hope this saves you some of the head-scratching. Corrections, contributions, and suggestions are very welcome!

daviftorres · 2026-07-17T15:06:34Z

daviftorres
Jul 17, 2026

Dear @chunkyen,

I am personally interested in what you are doing with Talos inside ACS.

Are you working on alternative offering for Kubernetes using ~~Calico~~ Talos instead of the official K8s?

If that is where you heading, I am available to perform some tests of the Calico build on ACS because that is my preferred way of running K8s anywhere.

Feel free to ping me if you need assistance on that. Regards,

10 replies

marcusaram Oct 8, 2026

Sure, at a high level. Small correction, we switched to Cilium instead of Calico:

Talos Images. The operator picks a Talos version and image type in the portal. We build the schematic through the Talos Image Factory and register the result once as a CloudStack template.

Provisioning. Each cluster is driven by a small reconcile loop in the plugin. Every step is idempotent and its progress is stored, so a crash or restart just resumes. The steps, in order:

Create an isolated network for the cluster and get a public IP.
Add CloudStack load balancer rules on 6443 (Kubernetes API) and 50000 (Talos API), so there is no separate VIP to manage.
Set anti-affinity groups: strict for control planes.
Generate the cluster secrets and machine configs with the Talos Go library. Patches set the install disk and the cluster's own installer image, turn off the default CNI and kube-proxy, and turn on the external cloud provider.
Deploy the VMs from the template, with the machine config passed as userdata.
Bootstrap the first control plane through the Talos API (via the LB on 50000), wait until it's healthy, then fetch the kubeconfig.

Add-ons. The plugin creates a dedicated CloudStack service account per cluster, scoped to the customer's project. Its keys feed the CloudStack CCM and the CloudStack CSI driver. We install Cilium (kube-proxy replacement plus ingress) and CSI with Helm and apply the CCM manifest, all from the plugin, with no GitOps tooling.

Day 2. All of it runs through the Talos API from the portal:

Scaling workers: new VMs come up with the worker config. Removed workers are reset and their VMs destroyed.
Talos OS upgrades: node by node, workers first, using the factory installer image for the target version. The portal waits for each node to report the new version and be healthy, and offers rollback to the previous boot partition.
Kubernetes upgrades: node by node, control planes first. The new version is applied to the live machine config without a reboot, the portal waits for each node to be Ready on it, then refreshes the cluster's built-in manifests. It rolls back automatically if a node stalls.

Customers get the kubeconfig and all these actions in their panel, and state is kept encrypted on the portal side.

chunkyen Oct 9, 2026
Author

Mind sharing the techstack behind the portal and the plugin? Was there a consious decision to not use Omni (I know it is not free for Enterprise use but what else?) - Thanks

marcusaram Oct 9, 2026

We evaluated Omni when Sidero Labs stopped working on the Talos CAPI provider and put their effort into Omni. It still wasn't what we were looking for. Pricing was part of it, but mainly we wanted full self service with fully automated provisioning, inside our own customer portal.

Our portal started out in PHP (Laravel), which didn't always run smoothly. So we rewrote it in Go and made it more generic along the way, and it's now becoming a product of its own (shameless self-promotion: Nubriva).

At a high level, the stack is:

Core: a Go application that handles customers, ordering, billing and the panels.
Plugins: separate Go plugins. Each integration is a plugin: CloudStack, Talos, DNS, accounting, and so on.
Frontend: Vue.
Talos/Kubernetes: the plugin talks to the Talos API directly through Sidero's Go machinery library, no talosctl. Add-ons go on with the Helm Go SDK and client-go, and CloudStack is driven through its API.

That keeps the core generic: adding a new provider is "just" another plugin. And since most of the Kubernetes ecosystem is written in Go, we can use those tools directly as libraries instead of shelling out to their CLIs.

chunkyen Oct 9, 2026
Author

Thank you, that's very helpful. If you are attending the upcoming cloudstack collaboration conference, I would like to meet up with you to have a chat to find out more. Cheers

marcusaram Oct 10, 2026

Unfortunately I'm not attending the conference. Feel free to contact me.

Navigation Menu

Repository navigation

📢 CloudStack Kubernetes Flavors — A Reference Guide I Wish I Had When I Started #13638

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 10 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

📢 CloudStack Kubernetes Flavors — A Reference Guide I Wish I Had When I Started #13638

Uh oh!

chunkyen Jul 17, 2026

Replies: 1 comment · 10 replies

Uh oh!

Uh oh!

daviftorres Jul 17, 2026

Uh oh!

marcusaram Oct 8, 2026

Uh oh!

Uh oh!

chunkyen Oct 9, 2026 Author

Uh oh!

marcusaram Oct 9, 2026

Uh oh!

chunkyen Oct 9, 2026 Author

Uh oh!

marcusaram Oct 10, 2026

chunkyen
Jul 17, 2026

Replies: 1 comment 10 replies

daviftorres
Jul 17, 2026

chunkyen Oct 9, 2026
Author

chunkyen Oct 9, 2026
Author