feat(helm): probe liveness on /health/live and separate DNS lookup timeout from cache time - #89

Merge only after SocketDev/socket-nginx-firewall#611 is merged and released, and the release bot has bumped appVersion to that release. #603 (/health/live) shipped in firewall 2.17.0. On an image without #611 the DNS change makes the firewall reuse resolved addresses for 10s instead of 5s and leaves the 30s lookup wait in place.

What & why

Two changes, both about a pod whose worker processes have lost DNS.

1. Liveness on /health/live. Pods can stay alive while one worker inside them has lost DNS. Each worker resolves names on its own, so one of several workers can fail every lookup while /health, a static OK, keeps the pod Ready. Firewall 2.17.0 adds GET /health/live, which returns 503 when one worker cannot resolve a host (the Socket API or outbound proxy, and Redis) that another worker in the same pod resolves. This PR points the liveness probe at it by default, so Kubernetes restarts those pods.
2. Separate the DNS lookup timeout from the DNS cache time. Firewall images up to 2.18.0 use nginx.resolverTimeout as the time a resolved address is reused, and every lookup waits 30s for an answer, so requests on a worker with no DNS answers hang 60-90s. With #611, resolver_timeout is the lookup timeout and a new resolver_valid is the cache time. This PR adds nginx.resolverValid (default 5s, the cache time charts have always produced) and sets nginx.resolverTimeout to 10s.

How it works

• New healthCheck.livenessPath, default /health/live. The liveness probe uses livenessPath | default path; readiness keeps healthCheck.path (/health). Readiness deliberately stays on /health: sharing /health/live would pull a pod with one bad worker out of the Service about 60s before liveness restarts it.
• /health/live does not fail when DNS fails on every worker (cluster DNS down, a missing Redis name), so a cluster-wide DNS outage does not restart every pod at once. livenessPath: "" restores the previous probe.
• Images before 2.17.0 exempt only /health from the client auth gate, so with socket.bearerToken set /health/live returns 401 there and liveness would restart the pod in a loop. New healthCheck.livenessMinImageVersion (default 2.17.0): when the liveness path would be /health/live and image.tag is a semver older than that, the probe uses healthCheck.path instead. Non-semver tags (latest, digests) are assumed new enough, the same rule as metrics.minImageVersion; the checks are nested so a non-semver tag never reaches semverCompare, which errors on it.
• resolverTimeout: "10s": the resolver resends an unanswered query every 5s, so 10s allows one resend; 5s would fail a lookup on one lost packet. resolver_valid is always rendered; images without the split ignore it.
• Chart version 0.12.0 because default probe and DNS behavior change, with the CloudFormation ChartVersion default and README pinned to match (the release bot requires it). README gains a Health Checks section, a DNS Timeouts section, value rows and upgrade notes.

What to look at first

• The one-line probe change in templates/deployment.yaml and the one added line in templates/configmap.yaml.
• Edge case: a user with a custom healthCheck.path now gets liveness on /health/live rather than their path. Acceptable because every image serves /health/live; the README calls it out with the opt-out.

Testing

helm lint clean. helm template probe paths (on a 2.18.0 tag unless noted):

Client auth gate on (SOCKET_BEARER_TOKEN set), no token sent:

Rendered socket.yml carries resolver_timeout: "10s" and resolver_valid: "5s". Fed to the config tool from #611, it generates resolver 10.96.0.10 valid=5s ipv6=off; and resolver_timeout 10s;.

End to end in a kind cluster with this chart (1 replica), with DNS replies to one worker blocked inside the pod:

The kind run used ports.disableHttps=true and small resource requests to fit a single-node cluster; both probes use the HTTP port. It predates the DNS change, which was checked by rendering only.

Notes

• The release bot also bumps version; expect to rebase over its bump and keep 0.12.0 (or the next minor) on top.