Repository navigation
Description
What happened?
With --wait=watcher (kstatus), waiting on a single hook Secret makes Helm sync every Secret in the release namespace, including all helm.sh/release.v1 records of every other release there. In a shared namespace with many releases this is hundreds of MB per helm upgrade, and upgrades fail with:
pre-upgrade hooks failed: resource Secret/<ns>/<name> not ready. status: Unknown, message:
context deadline exceeded
(the same error also appears for non-hook Secrets and Deployments of the release when the namespace is large enough).
Our numbers, one namespace, ~140 releases:
- ~580
helm.sh/release.v1Secrets, ~190 MB in total (GET /api/v1/namespaces/<ns>/secrets?resourceVersion=0returns ~190 MB in one unpaginated response from the watch cache). - The rendered manifest of a typical release is ~27 KB; each release record is ~350 KB because it stores the whole chart (in our case including test fixtures in the chart directory).
- The Helm client runs in another region than the API server. When ~15 upgrades run in parallel, each one downloads the whole namespace's Secrets before it can see its own hook Secret.
apiserver_watch_list_duration_seconds{resource="secrets",scope="namespace"}sits at the top (60 s) bucket during those waves, the hook staysUnknownuntil the timeout, and the upgrade is rolled back. - With
KUBE_FEATURE_WatchListClient=falsethe reflector falls back to a plain LIST, which the API server cuts at its 60 s request timeout (stream error ... INTERNAL_ERROR; received from peer), so that is not a workaround.
Where it comes from (v4.3.1):
pkg/kube/statuswait.gocallssw.Watch(..., watcher.Options{RESTScopeStrategy: watcher.RESTScopeNamespace}).github.com/fluxcd/cli-utilsv1.2.2pkg/kstatus/watcherstarts one informer per GroupKind + namespace (object_status_reporter.go), andDynamicInformerFactory.NewInformerbuilds theListWatchwith no label or field selector, so the informer lists/watches all objects of that kind in the namespace.
Helm 3 (and --wait=legacy in Helm 4) polls each resource by name with GET, and watches hook Jobs/Pods with a metadata.name field selector, so its cost did not depend on what else lives in the namespace.
What did you expect to happen?
Waiting on a release's resources should cost roughly in proportion to the release, not to everything of the same kind in the namespace — in particular not to Helm's own release history stored in that namespace.
Possible directions (not sure which fits best):
- Skip kinds that have no status to wait for (
Secret,ConfigMap,ServiceAccount, ...) — kstatus reports themCurrentas soon as they are observed, so watching them adds nothing but the namespace-wide sync. - Narrow the informers, e.g. a label selector, or a
metadata.namefield selector when a single object of a kind is awaited. - At minimum, exclude
owner=helmrelease records from Secret informers.
How can we reproduce it (as minimally and precisely as possible)?
- In one namespace, create many releases with history (or one release with a large chart) so that the namespace holds a large volume of Secrets — e.g. 500 Secrets of ~300 KB each.
- Use a chart that contains a
pre-upgradehook Secret (or any Secret). - Run
helm upgrade --install --wait=watcher --timeout 300s ...from a client with limited bandwidth to the API server (or several upgrades in parallel). - Observe that the client downloads all Secrets of the namespace before the hook Secret becomes
Current; with enough volume or parallelism the hook staysUnknownuntilcontext deadline exceeded.
Workarounds we applied: --history-max 3 and excluding test fixtures from the packaged chart via .helmignore to shrink each release record.
Helm version
Details
$ helm version
version.BuildInfo{Version:"v4.3.1", GitCommit:"c0d08d0d6d9424d4a14febb5e8a3e51b0dd00dff", GoVersion:"go1.27.2", KubeClientVersion:"v1.37"}CI uses the official release tarball from get.helm.sh.
Kubernetes version
Details
$ kubectl version
Server Version: v1.35.6
Activity