Visitar URL original
deploy: promote ingress stability fixes to production by rdimitrov · Pull Request #1738 · modelcontextprotocol/registry · GitHub
Skip to content

deploy: promote ingress stability fixes to production - #1738

Merged
rdimitrov merged 1 commit into
mainfrom
codex/promote-ingress-stability
Oct 25, 2026
Merged

rdimitrov merged 1 commit into
mainfrom
codex/promote-ingress-stability

Conversation

@rdimitrov

Copy link
Copy Markdown
Member

Promote the infrastructure changes from #1737 through the existing production workflow while retaining registry application image 1.8.4. The comment-only change to deploy/Pulumi.gcpProd.yaml deliberately triggers that workflow on merge; it does not change application configuration or encrypted secrets.

Staging deployment of merge commit 5b2a1a3 succeeded: https://github.com/modelcontextprotocol/registry/actions/runs/40241181207. Verified both registry replicas are available on different nodes with 250m CPU requests, ingress is available with a 300m request and 3s/5s readiness/liveness timeouts, and per-revision spreading is applied. All new containers had zero restarts. Health, list, and name-search requests returned HTTP 200 in approximately 0.6 seconds. Staging has one ingress replica, so production's two-replica placement still needs post-deployment verification; these smoke checks do not establish peak-load performance.

Production preview at this promotion commit succeeded: two intended Deployment updates plus existing Grafana empty-rule normalization. The refreshed preview of the identical infrastructure code also showed recreation of the two automatically cleaned-up ingress admission setup Jobs. No node, database, or backup changes, deletions, or replacements are planned.

After deployment, verify both production Deployments and node placement, then compare probe failures/restarts and ingress request latency at comparable traffic during a busy period. The disk upgrade remains a separate rollout. Related: #1252.

@rdimitrov
rdimitrov merged commit 4e2df3f into main Oct 25, 2026
9 checks passed
@rdimitrov
rdimitrov deleted the codex/promote-ingress-stability branch October 25, 2026 22:54
rdimitrov added a commit that referenced this pull request Oct 25, 2026
Production ingress stalls correlated with high I/O latency and queues on
the 20 GB `pd-standard` boot disks, which also serve the ingress cache
and container logs. Increase both hosted environments to 50 GB
`pd-balanced` boot disks and explicitly retain GKE's surge strategy
(`maxSurge: 1`, `maxUnavailable: 0`). Node count and machine type stay
the same; database and monitoring persistent volumes keep their existing
sizes.

**Merge and verify production promotion #1738 before merging this PR.**
Merging this PR automatically rolls staging; production needs a
subsequent promotion after staging verification. Production deployment
uses all infrastructure code at the triggering commit, so reversing this
merge order would combine the pod and disk changes. GKE updates the
existing node pool by replacing its VMs incrementally. The single
registry and Grafana PostgreSQL instances will be interrupted while
their separate persistent disks move; surge capacity does not provide
database HA. The README documents preparation and verification.

Validation:
- `make -C deploy build`, `go vet ./...` in `deploy`, and `git diff
--check` passed.
- Refreshed production Pulumi preview at `1048543` succeeded. The
node-pool diff contains only `nodeConfig.diskSizeGb` and
`nodeConfig.diskType`; no Pulumi resource deletions or replacements and
no persistent-volume/database changes. The rest of the plan is the
pending #1738 Deployment updates, two automatically cleaned-up ingress
admission Jobs, and existing Grafana empty-rule normalization.
Re-preview after #1738 deploys before production promotion.
- Both projects have regional CPU, SSD-disk, and address quota headroom
for a surge node; actual zonal capacity is only established during
provisioning.
- Both production PostgreSQL clusters are healthy with one instance and
PDBs disabled. Their disks use the GCE persistent-disk CSI driver. The
production registry backup last succeeded at 2026-10-25 04:47:13 UTC;
check freshness again at rollout.

No infrastructure has been applied from this branch. After rollout,
verify disk configuration, workload readiness/placement and API smoke
checks, then compare disk queues, ingress restarts and request latency
during busy traffic. Related to #1252; this change does not establish
that the latency target has been met.

[GKE surge upgrade
behavior](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/node-pool-upgrade-strategies)
rdimitrov added a commit that referenced this pull request Oct 25, 2026
Promote the 50 GB `pd-balanced` node boot disks from #1739 through the
existing production workflow while retaining registry application image
`1.8.4`. This comment-only change to `deploy/Pulumi.gcpProd.yaml`
deliberately triggers that workflow on merge. The two-node pool retains
`e2-standard-2` machines and uses surge upgrades with one extra node and
zero unavailable nodes.

Staging rollout succeeded:
https://github.com/modelcontextprotocol/registry/actions/runs/40243009855.
Verified both original nodes were replaced with Ready nodes whose actual
boot disks are 50 GB `pd-balanced`. All four original persistent volumes
retain their identities, volume handles and sizes, and both PostgreSQL
clusters recovered. Every active workload is ready; the two registry
replicas are on different nodes, and registry/ingress containers have
zero restarts. Health, listing and name-search checks returned HTTP 200
in about 0.6 seconds; a fresh unique search also returned HTTP 200.

**Allow for a multi-minute database interruption when merging.** During
staging replacement, registry PostgreSQL was unready from 23:16:45 to
23:20:29 UTC on October 25 (3m44s), including its configured 180-second
smart-shutdown timeout before disk reattachment/startup. This is an
observed staging duration, not a production guarantee. Grafana
PostgreSQL also moved. The single staging ingress replica caused three
failed health samples during its move; production has two ingress
replicas but still only one instance of each database. `/health` does
not query PostgreSQL, so database readiness was checked separately.
Surge capacity does not provide database HA.

Refreshed production Pulumi preview at `c71e488`, after #1738 deployed,
passed: node-pool boot-disk size/type update, registry deploy-commit
annotation update, recreation of two automatically cleaned-up ingress
admission Jobs, and existing Grafana empty-rule normalization. No Pulumi
resource deletions/replacements or database/persistent-volume changes.
GKE nevertheless replaces the node VMs incrementally. All PR CI checks
passed.

Check the latest successful registry backup when merging; it last
succeeded at 2026-10-25 04:47:13 UTC during preparation. Both projects
have surge quota headroom; actual zonal capacity is established during
provisioning.

After production rollout, verify both nodes have 50 GB balanced boot
disks, all original persistent volumes are retained, both databases
recover, and workloads/API smoke checks pass. Compare disk queues,
ingress restarts and request latency during busy traffic before
concluding that #1252 is resolved.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant