Repository navigation
Tags
Tags
Optimize incremental sync queries and add staged production capacity (#… …1840) Incremental-sync listings now select page keys through an all-status covering index before fetching payloads, preserving deleted records, filters and cursor ordering. Prepared statement reuse and binary payload transfer remain enabled. Adds an opt-in production capacity transition: `prepare` rolls out 20/5 connection pools while retaining the existing HPA; `enabled` allows 2–6 apps at 40% CPU, with at most two additions per 30 seconds. Live guards require every app to have the smaller pool and a pool-capable release before activation, and require lowering the HPA ceiling before restoring larger pools. Application rollback remains independent of the connection budget. Existing profiles remain unchanged until opted in. Validation: database/handler/router race tests, deployment Go tests, 61 Python deployment tests, live staging response-equivalence checks and a bounded staging capacity comparison. Migration 019 must be prebuilt concurrently before production promotion. Deployment and rollback sequencing are documented in deploy/README.md.
fix(deploy): refresh state before applying saved plans (#1810) A refreshed preview sees TTL-expired Helm jobs as missing, but preview refresh does not persist that observation. Applying with refresh disabled then treats the old checkpoint entries as existing resources and proposes replacements, violating the saved plan's create constraints. Refresh during apply as well as preview. Preserve the saved plan and exact Helm reference-ID checks. Also key the staging deployment Go cache on deploy/go.sum so infrastructure dependency updates use the correct cache. Validation: all 72 Python tests and actionlint passed. The corrected saved-plan apply successfully deployed merge 02bc451 to staging. One Ready app replica, zero restarts; public health and v0/v0.1 lists returned 200; Grafana login and database health passed; both monitoring scrape targets were up. Original nodes and volume identities were preserved. Production remains on 1.8.7. Verify the automatic staging run after merge. Gateway research remains excluded.
Gate prepared listing responses for backfill and rollback (#1767) Prepared listing responses need an activation gate so existing rows can be backfilled before reads use the new path. Add `MCP_REGISTRY_PREPARED_LISTING_JSON`, defaulting to false, for both API versions, with explicit Pulumi activation/rollback configuration. Disabling restores typed responses while preserving the additive schema and derived data. Validation: full CI and Go analysis passed. Focused local integration tests verify response equivalence, updates/deletes, old-writer invalidation, backfill/fallback, and enable/disable rollback. Five short runs over 685 saved public records measured median encoding time 647639→241701 ns, bytes allocated 742484→619838, and allocations 1790→222. This is a local encoding-cost result, not a production capacity estimate; allocation benefit depends on record shape. Broad staging replay did not pass: generator source-port exhaustion and CPU pressure caused drops and transport failures, with materially different delivered/cache workloads. The user chose focused correctness/efficiency validation instead of further broad replay; that failed result remains explicit. Production activation is separate: fresh backup, versioned release with flag false, additive migration and bounded backfill, response/coverage checks, then flag enablement with rollback retained. This PR does not change production stack configuration.
Optimize broad searches and validate staging burst capacity (#1752) Broad uncached searches consume the shared PostgreSQL CPU budget and queue concurrent reads and publishes. Select matching page keys through covering indexes before fetching payloads, and use parameter-specific search plans so rare and broad terms do not reuse an unsuitable generic plan. Filters, cursors, payloads and API page limits remain unchanged. The tested fix also needs a 3-CPU PostgreSQL quota on the existing nodes. This PR adds that optional setting and enables it in staging; production configuration and the max 60/min 10 connection pool per app remain unchanged. Query changes alone at 1.5 CPUs did not pass. ### Staging evidence Using a fixed public production corpus, two app replicas and cold ingress caches, repeated old/new/new/old trials at the same 3-CPU quota showed: | Trial | Searches completed | Dropped | Full walks | Publishes | Publish p95 | |---|---:|---:|---:|---:|---:| | Existing A1 | 11,762/12,000 | 238 | 30/30 | 90/90 | 604 ms | | Candidate B1 | 12,000/12,000 | 0 | 30/30 | 90/90 | 290 ms | | Candidate B2 | 12,000/12,000 | 0 | 30/30 | 90/90 | 450 ms | | Existing A2 | 8,716/12,000 | 3,284 | 30/30 | 90/90 | 705 ms | Both candidate repeats had zero request failures. All 83 complete response hashes matched across 37 terms, both APIs, filters and pagination. Both controls also had the new indexes; the control is staging b96c5b2, including the earlier namespace fix, not production v1.8.5. The initial 100-second cutoff prematurely cancelled progressing full walks even without searches. The repeated comparison extended only observation to 180 seconds; search arrivals (200/s for 60 seconds), publish traffic, crawl waves and per-request deadlines stayed unchanged. Original failed experiments remain documented in [the full results](tools/readbench/STAGING_SEARCH_RESULTS.md), alongside machine-readable evidence and limitations. ### Validation and rollout - Full internal/cmd race tests, deployment package compilation, golangci-lint and `make validate` pass. - Source-built Pulumi previews pass: staging changes the DB quota from 1.5 to 3 CPUs; production has no capacity change from this PR. - Staging test workloads, diagnostic settings and benchmark rows were removed; the previous app and 1.5-CPU quota were restored after testing. Permanent node/ingress parity remains. - No production writes, release or promotion were performed. Before production promotion, concurrently build and verify both migration 017 indexes, then promote the application with the tested 3-CPU quota. The current singleton DB restarts for a resource change; deployment instructions document this requirement. This establishes capacity for the specified burst and corpus, not a guarantee for arbitrary future load.
feat(observability): diagnose individual slow listing requests (#1743) Slow listing requests currently expose separate HTTP and database histograms, but there is no request-level timing record. Database timing also includes application JSON decoding, making a slow query indistinguishable from slow row processing. Add bounded structured diagnostics for GET `/v0/servers` and `/v0.1/servers`: progress after 2 seconds and every 10 seconds while outstanding, plus completion for slow or failed requests. Records correlate ingress request ID, pod, route, query-shape flags, page size, PostgreSQL backend PID and SQLSTATE, row/JSON-byte counts, and exclusive handler/acquisition/query/fetch/decode/response timings. Raw SQL, arguments, cursors, search strings and payloads are excluded. Logging is capped per pod at 2 events/second with burst 10 and suppression accounting; fast successful requests remain silent. Validation: - Full internal/cmd unit suite passed with race detection and coverage against PostgreSQL 17. - Controlled tests prove a blocked pool is visible before cancellation, delayed queries are distinct from row processing, timings account for the full request, and request IDs/context propagate through the real Huma adapter. - Tests cover SQLSTATE-only error detail, panic/cancellation completion, timer shutdown and bounded log volume. - `make lint`, `make build`, `make validate`, and `git diff --check` passed. Follow-up to #1742. No query, index, resource, or cache-policy changes. After staging deployment, verify a bounded slow request and log ingestion before cutting the production release. Internal PostgreSQL wait attribution still requires contemporaneous database samples; sampled logs are not exhaustive tracing.
fix(publish): validate registries outside the transaction and time th… …e pool wait (#1562) ## Why Publish latency has been the only alert firing in production — 1,285 times since 2025-09-11, while the read-path rules fired 6 times in total (last one 2025-09-14). Digging into where that time actually goes turned up a contradiction: | Layer | Publishes (7d) | Over 10s | Max | |---|---|---|---| | HTTP histogram (`/v0/publish`) | 9,939 | **17** | 45.5s observed | | Per-phase publish log (`createServerInTransaction`) | 7,845 | **0** | **6.5s** | `validate_ms` — the external registry calls — peaks at 3.9s (p50 18ms, p99 1.1s). So the slow requests were spending their time somewhere the phase log could not see. **The log had a blind spot exactly where the problem is.** `InTransaction` calls `pool.Begin` and *then* invokes the transaction body, but the timing started inside that body. A publish that waited 40s for a free connection logged `total_ms=200` and looked healthy. ## What **Time the pool wait.** The gap between a timestamp taken before `InTransactionT` and the first line of its callback is exactly how long `pool.Begin` blocked. Reported as `pool_wait_ms`. **Move registry-ownership validation ahead of the transaction.** It fans out to npm, PyPI, NuGet, Cargo, OCI and MCPB with a 10s timeout per host. Running it inside the transaction pinned a pgxpool connection for the duration of those round-trips — so under a publish burst (up to 276 in a 4-minute window) connections are held by requests doing no database work at all, starving a pool already sitting at 150 of 200 connections and stalling unrelated requests in `pool.Begin`. The check reads none of our own state, so hoisting it introduces no race, and the per-server advisory lock was already acquired *after* validation. To keep one log event per publish, timing and logging move up into `CreateServer` via a shared `publishTimings`; `createServerInTransaction` fills in the phases that still run inside. ## Note on `total_ms` `total_ms` now covers the whole `CreateServer` call rather than just the transaction body, so the residual after the named phases is commit plus overhead. Nothing consumes these log fields — all four Grafana rules are built on `mcp_registry_*` metrics. ## Out of scope The **edit path** keeps its validation inside the transaction, with a comment explaining why: `skipRegistryValidation` is derived from a read inside that transaction, so hoisting needs a second read before it opens plus a decision about the resulting race. It carries a fraction of publish traffic — worth revisiting if it shows up in `pool_wait_ms`. ## Testing New tests in `internal/service/registry_service_publish_transaction_test.go`, using a small `database.Database` double to observe the transaction boundary (which a real database cannot): - **validation failure opens no transaction** — the invariant this change introduces - **`pool_wait_ms` covers time `pool.Begin` blocked** — via a double whose `InTransaction` sleeps - a validation failure still emits one `publish failed` naming `failed_phase=validate` - a failure raised *inside* the transaction is still captured by the now-outer deferred logger Existing tests pass unchanged. `go test ./...` (12 packages), `-race` clean on the service package, `golangci-lint` 0 issues. ## Verifying this in production The payoff is only visible once deployed: `pool_wait_ms` will either confirm pool starvation as the mechanism or rule it out. If it comes back near zero on the slow publishes, the remaining suspects are JWT validation, body decode, and the per-request JSON schema compile (`schema.go:196-210` builds a fresh compiler per call — measured at 676µs and 882KB per call, deliberately not touched here). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
fix(database): index server name substring searches (#1730) Substring searches use `server_name ILIKE '%term%'`, which was scanning the production servers table and contending with ordinary listings. Add a GIN trigram index using the existing pg_trgm extension, plus instructions for concurrent construction on populated databases. The index was built concurrently in production on 2026-10-23 and verified valid/ready. Migration 015 skips the existing index, creates it on fresh databases, and rejects an invalid index left by an interrupted build. The transactional migration runner is unchanged. Production benefits immediately; a release is not required to activate this index. ### Validation - CI build, lint, validation, full unit/integration tests and CodeQL all passed for commit `5d358d3`. - Applied the migration twice against a fresh isolated staging schema, then rolled the test back. - Production EXPLAIN ANALYZE uses the new index for a filesystem search: 96 ms execution. - Extended production verification: 38 sequential read-only API checks across v0 and v0.1, all HTTP 200 and all response assertions passed; 30 cache misses. Covers three-page listings, search pagination, case-insensitive matching, short/common/missing terms, literal `%`, `_` and backslash matching, latest-version filters, limit=100, exact/latest details and version lists. Median response 0.77 seconds; maximum 6.8 seconds on a one-character search. - A fresh 65-second sample measured 232 ordinary searches averaging 214 ms and 28 latest-version searches averaging 59 ms. Pagination averaged 12–16 ms across 8,952 calls. Earlier pre-change search averages were approximately 26 seconds. Traffic/query mixes differ; these are operational observations rather than a controlled benchmark. - Latest five-minute monitoring window: approximately 39,700 listing requests with zero 5xx responses. At an earlier check in the same validation session, application-observed mean/p95 were 29/100 ms; these exclude client network time and are distinct from the direct API timings. - Index remains valid/ready; PostgreSQL has not restarted. `git diff --check` passed. ### Remaining limitations Brief database readiness failures and connection errors occurred during the build, and another readiness failure occurred at approximately 17:20 UTC after completion (18 listing 5xx responses recorded in subsequent metrics). The database recovered readiness at 17:20:54 UTC. These later failures mean the index must not be described as resolving every availability issue. One- and two-character searches still took 4–7 seconds in some direct checks. Keep this PR focused on the index; short-search cost and readiness resilience warrant separate investigation. An optional read-only comparison against forced sequential scans hit its five-second statement timeout and was stopped; result equivalence for that additional check was not established. No production records were modified or restored by these checks.
fix(ui): respect display density and default icon sizes (#1728) Icon selection currently compares raster dimensions with a 40 CSS-pixel tile, which selects undersized images on high-density displays. Use the current device pixel ratio when scoring candidates: for example, a 2× display now chooses a 96×96 icon over a 48×48 icon for that tile. Treat omitted or empty `sizes` like `"any"`, matching the schema's default that an icon without sizes is usable at any size. This keeps an SVG without size metadata from losing to a fixed-size raster icon. Follow-up to #1586. Validation: - Focused JavaScript assertions against the actual HTML helpers passed for default/1×/1.5×/2×/3× density, omitted/empty/`any` sizes in both candidate orders, undersized fallback, theme preference, unsafe schemes, and attribute escaping. - `go test ./internal/api/handlers/v0 -run '^TestUIHTML_' -count=1` passed. - `git diff --check` passed.
fix(deploy): cache bulk registry reads (#1509) ## Summary - cache successful public list responses for `/v0/servers` and `/v0.1/servers` at ingress-nginx for 30 seconds - give only those exact cached paths a 600 RPM per-controller budget - preserve the existing 180 RPM limit for publish, auth, mutation, detail, and other routes - expose `X-Registry-Cache` to make cache MISS/HIT behavior directly observable ## Why Production ingress logs show that the current 180 RPM per-controller limit rejects legitimate bulk pagination before it reaches the application. The affected traffic is overwhelmingly registry-list reads, not publishing. A ten-minute production sample contained 17,836 successful list responses but only 3,188 unique URLs: 82.1% were repeated requests, representing roughly 1 GB of duplicate response bodies. Raising the list limit without caching would increase pressure on the application and its shared PostgreSQL pool. The short ingress cache absorbs repeated cursor-page reads before raising their scoped client budget. Publish latency is a separate concern. Reducing read pressure may help indirectly, but slow external package validation is intentionally out of scope for this PR. Relates to #826. ## Safety - caching is limited to exact public GET list paths and keyed by host plus full request URI - only 200 responses are cached - cache lifetime is 30 seconds; newly published entries can therefore take up to 30 seconds to appear on a cached page - cache locking coalesces simultaneous misses - stale cached responses may be served while refreshing or during transient upstream 5xx/timeouts - each ingress replica uses at most 1 GB of ephemeral on-disk cache ## Validation - `go test ./...` in `deploy/` - `go vet ./...` in `deploy/` - `git diff --check` - ingress-nginx chart 4.13.0 in a disposable kind cluster - generated NGINX configuration passed `nginx -t` - repeated list request verified `X-Registry-Cache: MISS` followed by `HIT` - non-list route verified uncached - exact list locations verified with the higher rate zone while the catch-all retained 180 RPM Nothing was applied to production during investigation or validation.
fix(validators): distinguish a missing package from a missing version… … on PyPI/NPM (#1411) Fixes #553. The PyPI and npm validators fetch the version-specific metadata endpoint (`/pypi/{name}/{version}/json` and the npm equivalent) and report any non-200 as `<pkg> not found`. That endpoint 404s for two different reasons though: the package doesn't exist, or it exists and only the *version* is missing (e.g. a release that hasn't propagated yet). So a valid package reads as gone: ```diff - PyPI package 'requests' not found (status: 404) + PyPI package 'requests' exists, but version '99.99.99' was not found (status: 404). A newly published release can take a moment to appear on PyPI. Wait and retry, or publish version '99.99.99' before registering it ``` On a version 404 it probes the package-level endpoint (`/pypi/{name}/json`, `/{name}`) with a `HEAD` to tell the two apart, and reports 429/5xx as transient rather than "not found". `HEAD` so it reads a status without pulling the whole packument. The probe carries its own 3s deadline: it only refines the error message, so a hung probe must not stretch the validator past the ~10s-per-registry budget the publish path assumes. Same probe-and-classify shape as the existing cargo validator. The fetch is split into `validatePyPIPackage` / `validateNPMPackage` behind `export_test.go` so the branches are testable with `httptest`, the same seam cargo uses. ### Testing - [x] hermetic `httptest` for the status matrix (version-missing, package-missing, 5xx, 429, inconclusive probe, positive path, scoped npm); the mocks pin the expected method per endpoint (`GET` fetch, `HEAD` probe) - [x] hermetic deadline test: a hung probe is cut off at ~3s and reported as inconclusive, not "not found" - [x] existing live package tests still pass, now also hitting the HEAD probe; on a live 429/5xx they now `t.Skip` as inconclusive instead of flaking (deliberate: CI runs these against the real registries with no short-mode gating) - [x] `gofmt` / `go vet` / `golangci-lint` clean Out of scope: auto-retry for the propagation race, and SSRF redirect-pinning parity for the pypi/npm clients (pre-existing, shared with nuget).
PreviousNext