Visitar URL original
--wait=watcher syncs every Secret of the namespace (incl. all Helm release records) to wait on one hook Secret · Issue #32900 · helm/helm · GitHub
Skip to content

--wait=watcher syncs every Secret of the namespace (incl. all Helm release records) to wait on one hook Secret #32900

Open

Description

@savchenkomaxim

What happened?

With --wait=watcher (kstatus), waiting on a single hook Secret makes Helm sync every Secret in the release namespace, including all helm.sh/release.v1 records of every other release there. In a shared namespace with many releases this is hundreds of MB per helm upgrade, and upgrades fail with:

pre-upgrade hooks failed: resource Secret/<ns>/<name> not ready. status: Unknown, message:
context deadline exceeded

(the same error also appears for non-hook Secrets and Deployments of the release when the namespace is large enough).

Our numbers, one namespace, ~140 releases:

  • ~580 helm.sh/release.v1 Secrets, ~190 MB in total (GET /api/v1/namespaces/<ns>/secrets?resourceVersion=0 returns ~190 MB in one unpaginated response from the watch cache).
  • The rendered manifest of a typical release is ~27 KB; each release record is ~350 KB because it stores the whole chart (in our case including test fixtures in the chart directory).
  • The Helm client runs in another region than the API server. When ~15 upgrades run in parallel, each one downloads the whole namespace's Secrets before it can see its own hook Secret. apiserver_watch_list_duration_seconds{resource="secrets",scope="namespace"} sits at the top (60 s) bucket during those waves, the hook stays Unknown until the timeout, and the upgrade is rolled back.
  • With KUBE_FEATURE_WatchListClient=false the reflector falls back to a plain LIST, which the API server cuts at its 60 s request timeout (stream error ... INTERNAL_ERROR; received from peer), so that is not a workaround.

Where it comes from (v4.3.1):

  • pkg/kube/statuswait.go calls sw.Watch(..., watcher.Options{RESTScopeStrategy: watcher.RESTScopeNamespace}).
  • github.com/fluxcd/cli-utils v1.2.2 pkg/kstatus/watcher starts one informer per GroupKind + namespace (object_status_reporter.go), and DynamicInformerFactory.NewInformer builds the ListWatch with no label or field selector, so the informer lists/watches all objects of that kind in the namespace.

Helm 3 (and --wait=legacy in Helm 4) polls each resource by name with GET, and watches hook Jobs/Pods with a metadata.name field selector, so its cost did not depend on what else lives in the namespace.

What did you expect to happen?

Waiting on a release's resources should cost roughly in proportion to the release, not to everything of the same kind in the namespace — in particular not to Helm's own release history stored in that namespace.

Possible directions (not sure which fits best):

  • Skip kinds that have no status to wait for (Secret, ConfigMap, ServiceAccount, ...) — kstatus reports them Current as soon as they are observed, so watching them adds nothing but the namespace-wide sync.
  • Narrow the informers, e.g. a label selector, or a metadata.name field selector when a single object of a kind is awaited.
  • At minimum, exclude owner=helm release records from Secret informers.

How can we reproduce it (as minimally and precisely as possible)?

  1. In one namespace, create many releases with history (or one release with a large chart) so that the namespace holds a large volume of Secrets — e.g. 500 Secrets of ~300 KB each.
  2. Use a chart that contains a pre-upgrade hook Secret (or any Secret).
  3. Run helm upgrade --install --wait=watcher --timeout 300s ... from a client with limited bandwidth to the API server (or several upgrades in parallel).
  4. Observe that the client downloads all Secrets of the namespace before the hook Secret becomes Current; with enough volume or parallelism the hook stays Unknown until context deadline exceeded.

Workarounds we applied: --history-max 3 and excluding test fixtures from the packaged chart via .helmignore to shrink each release record.

Helm version

Details
$ helm version
version.BuildInfo{Version:"v4.3.1", GitCommit:"c0d08d0d6d9424d4a14febb5e8a3e51b0dd00dff", GoVersion:"go1.27.2", KubeClientVersion:"v1.37"}

CI uses the official release tarball from get.helm.sh.

Kubernetes version

Details
$ kubectl version
Server Version: v1.35.6

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions