Repository navigation
Tags: git-pkgs/proxy
Tags
Filter cached packages by ecosystem using pills (#319) * experimental filtering pills * align data in the dashboard table more nicely (#318) * Bump zizmorcore/zizmor-action from 0.6.2 to 0.6.3 (#333) Bumps [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) from 0.6.2 to 0.6.3. - [Release notes](https://github.com/zizmorcore/zizmor-action/releases) - [Commits](https://github.com/zizmorcore/zizmor-action/compare/3dc1ecc9bcb9e94e9b2c709687979e1298497054...70fb788f84895a7701f5643d103d587e460b5c99) --- updated-dependencies: - dependency-name: zizmorcore/zizmor-action dependency-version: 0.6.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump docker/setup-qemu-action from 4.2.0 to 4.3.0 (#331) Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 4.2.0 to 4.3.0. - [Release notes](https://github.com/docker/setup-qemu-action/releases) - [Commits](https://github.com/docker/setup-qemu-action/compare/96fe6ef7f33517b61c61be40b68a1882f3264fb8...1f40c72289eff860ee54a304f1438e3cff362e0a) --- updated-dependencies: - dependency-name: docker/setup-qemu-action dependency-version: 4.3.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump azure/setup-helm from 4.3.1 to 5.0.1 (#332) Bumps [azure/setup-helm](https://github.com/azure/setup-helm) from 4.3.1 to 5.0.1. - [Release notes](https://github.com/azure/setup-helm/releases) - [Changelog](https://github.com/Azure/setup-helm/blob/main/CHANGELOG.md) - [Commits](https://github.com/azure/setup-helm/compare/1a275c3b69536ee54be43f2070a358922e12c8d4...9bc31f4ebc9c6b171d7bfbaa5d006ae7abdb4310) --- updated-dependencies: - dependency-name: azure/setup-helm dependency-version: 5.0.1 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump modernc.org/sqlite from 1.57.0 to 1.58.0 (#337) Bumps [modernc.org/sqlite](https://gitlab.com/cznic/sqlite) from 1.57.0 to 1.58.0. - [Changelog](https://gitlab.com/cznic/sqlite/blob/master/CHANGELOG.md) - [Commits](https://gitlab.com/cznic/sqlite/compare/v1.57.0...v1.58.0) --- updated-dependencies: - dependency-name: modernc.org/sqlite dependency-version: 1.58.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump github.com/aws/aws-sdk-go-v2/config from 1.32.40 to 1.33.2 (#336) Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.32.40 to 1.33.2. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.32.40...config/v1.33.2) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/config dependency-version: 1.33.2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump github.com/prometheus/client_model from 0.6.2 to 0.6.3 (#330) Bumps [github.com/prometheus/client_model](https://github.com/prometheus/client_model) from 0.6.2 to 0.6.3. - [Release notes](https://github.com/prometheus/client_model/releases) - [Commits](https://github.com/prometheus/client_model/compare/v0.6.2...v0.6.3) --- updated-dependencies: - dependency-name: github.com/prometheus/client_model dependency-version: 0.6.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump golang.org/x/sync from 0.22.0 to 0.23.0 (#335) Bumps [golang.org/x/sync](https://github.com/golang/sync) from 0.22.0 to 0.23.0. - [Commits](https://github.com/golang/sync/compare/v0.22.0...v0.23.0) --- updated-dependencies: - dependency-name: golang.org/x/sync dependency-version: 0.23.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump github.com/aws/aws-sdk-go-v2/service/ecr from 1.61.0 to 1.64.0 (#334) Bumps [github.com/aws/aws-sdk-go-v2/service/ecr](https://github.com/aws/aws-sdk-go-v2) from 1.61.0 to 1.64.0. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.61.0...service/s3/v1.64.0) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/service/ecr dependency-version: 1.63.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix(handler): let the ProxyCached path request a per-call Accept-Encoding (#324) * fix(handler): fetch conda repodata gzip-compressed on both hops #304 made the ProxyCached path request Accept-Encoding: identity so the metadata cache stores upstream bytes verbatim. That is required for the signed / hash-pinned index ecosystems, but conda's repodata.json is large plain JSON: linux-64 repodata.json is ~441 MB uncompressed (over the metadata_max_size cap, so it 502s today) versus ~34 MB gzip. Replace the ProxyCached path's verbatim bool with an explicit acceptEncoding string ('' = leave unset / transparent, 'identity', or 'gzip'), reusing #304's existing store-and-replay of Content-Encoding unchanged. ProxyCached keeps its exported signature and continues to send identity, so the nine other ecosystems and helm/maven are untouched; only conda's repodata.json / current_repodata.json now request gzip. Setting Accept-Encoding explicitly disables Go's transparent decompression, so the compressed bytes and the Content-Encoding: gzip header are cached and replayed exactly as identity bytes are. conda, mamba and pixi solicit and decode gzip on .json URLs; repodata.json.bz2 stays identity. Fixes #305 * fix(handler): pass metadata content-encoding with the body it describes The adversarial review of the conda gzip route found a reachable regression: writeMetadataCachedResponse took Content-Encoding from a fresh cache-row read while cacheMetadataBlob skips the row write when Storage.Store fails. Under identity that was benign (the body was plain anyway), but on the new gzip route a disk-full or object-store outage served raw gzip bytes as Content-Type: application/json with no Content-Encoding and HTTP 200 -- conda, mamba and pixi fail to parse them, with no HTTP signal and only a Warn log, on every request until a cache write succeeds. fetchOrCacheMetadata now returns the encoding of the body it hands back (the upstream value on a fetch, the stored row's value on a TTL hit or stale fallback) and proxyCachedWithEncoding passes it to writeMetadataCachedResponse, so the header always describes the bytes actually written. cachedMeta drops its now-unused content_encoding field. helm and maven pass "" -- both fetch transparently, so their stored encoding was always empty and behaviour is unchanged. Also fixes a vacuous assertion in the new conda test: the upstream request counter incremented behind the availability gate, so the cached-replay block could never observe a refetch. * fix(handler): pin the stale-fallback content-encoding and drop a dead guard Follow-ups from the adversarial review of the #305 branch, limited to code this branch introduced: - proxyMetadataStream is only ever reached with an explicit Accept-Encoding (ProxyCached passes identity, conda passes gzip or identity), so the guard around the header set was unreachable; replace it with the plain one-token substitution of the former literal, which is the smallest change from main. - The stale-fallback return of fetchOrCacheMetadata (encoding taken from the cache row) was the one #305 return site no test pinned: replacing it with an empty encoding survived the whole suite. Add a conda test that expires the entry, fails the upstream, and asserts the stored gzip blob is served with Content-Encoding: gzip. Not changed, by scope: cacheMetadataBlob still discards the UpsertMetadataCache error (pre-existing on main). If Storage.Store succeeds and the row write fails, a later stale fallback or TTL hit can serve the gzip blob with the row's stale encoding; that needs a DB write failure plus a second event and is tracked separately. * fix(handler): restore the pre-existing cachedMeta content-encoding field The third adversarial review classified deleting cachedMeta.contentEncoding and its lookupCachedMeta populate as elective: neither line was created by this branch nor forced by the fix (writeMetadataCachedResponse now reads the encoding from its parameter and ignores the row value). Under the rule that pre-existing code this branch did not have to touch stays untouched, restore both as they are on main. No behaviour change. Residuals the review documented, unchanged by scope (both share one root cause: the encoding lives in the cache row and the bytes in the blob, and neither is written or read atomically): - cacheMetadataBlob discards the UpsertMetadataCache error, so after a successful gzip Store and a failed row write a later stale fallback or TTL hit can serve the gzip blob with the row's stale encoding. - During the one-time identity->gzip rollout, a request that read a pre-branch identity row, lost the upstream race to a request that stored the gzip blob, and then failed upstream serves the gzip bytes with no Content-Encoding for that one response; later requests self-heal. - helm and maven now pass an empty encoding; on main a spec-violating upstream that answered a transparent gzip request with an encoding Go does not decode (e.g. br) would have had that header replayed from the row. Degenerate; documented rather than changed. * fix(handler): keep conda's proxyCached and .bz2 route as on main Threading acceptEncoding through CondaHandler.proxyCached changed the form of two pieces of original code the fix did not need to touch: the repodata.json.bz2 route (method value rewritten as a closure) and proxyCached itself (new parameter, new call). Restore both exactly as on main; ProxyCached still sends identity, so the .bz2 route is unchanged in behaviour. handleRepodata's non-cooldown branch now derives the cache key inline and calls proxyCachedWithEncoding with gzip directly, so the only original conda.go line that changes is that one call. * fix(handler): keep writeMetadataCachedResponse and its callers as on main Adding a contentEncoding parameter to writeMetadataCachedResponse changed a signature that predates #304 and dragged its two pre-#304 callers (helm.go, maven.go) into the diff, even though #304 only ever added the cm.contentEncoding block inside the function body. Restore writeMetadataCachedResponse's doc and signature exactly as on main and make it a delegate that passes an empty encoding to a new unexported writeMetadataCachedResponseWithEncoding, which carries the original body with #304's block reading the parameter instead of the cache row. proxyCachedWithEncoding calls the sibling with the encoding returned alongside the body. helm.go and maven.go drop out of the diff; their behaviour is unchanged (both fetch transparently, so their stored encoding was always empty). Same split pattern as ProxyCached -> proxyCachedWithEncoding. * fix(handler): move the conda gzip change to its own branch The conda call site in handleRepodata predates #304 and #304 never touched it, so under the rule that this PR only corrects code and behaviour #304 introduced it does not belong here. Restore conda.go and conda_test.go as on main; the conda change continues on a stacked branch against its own issue. Replace the conda-route tests with tests that exercise proxyCachedWithEncoding directly, so this PR still pins its own plumbing: gzip is requested and the compressed bytes plus Content-Encoding are cached and replayed (cached and streaming paths), the header survives a metadata cache write failure, and the stale fallback keeps the stored encoding. * fix(homebrew): fetch the JSON API gzip-compressed on both hops Homebrew (#254) routes every API path through ProxyCached and so, since #304, fetches formula.jws.json (~33 MB plain, ~5 MB gzip) uncompressed on every refresh -- the case that motivated #305. Request gzip for the JSON API via proxyCachedWithEncoding: brew fetches every API download with curl --compressed and decodes Content-Encoding itself, so the compressed bytes and header are cached and served as-is and both hops stay compressed. The analytics endpoints are the one brew consumer fetched without --compressed; they stay on identity. * fix(handler): leave Accept-Encoding unset in proxyMetadataStream for an empty value fetchUpstreamMetadata treats an empty acceptEncoding as 'do not set the header'; proxyMetadataStream set it unconditionally, which would send an empty Accept-Encoding line if a caller ever passed . Guard it the same way so both paths agree. No caller passes today. * fix(handler): keep the metadata row and blob from describing different bytes Two ways the cache row could stop describing the stored blob once a caller requests gzip, both raised by the review of #324: - cacheMetadataBlob stored the blob and then discarded the UpsertMetadataCache error. After a successful gzip store and a failed row write, a later TTL hit or stale fallback served the gzip blob with the previous row's encoding. On a row-write failure, log it and delete the blob just written, so the next request refetches instead. - fetchOrCacheMetadata read the row once up front and reused it for the stale fallback. A request that read an identity row, lost the upstream race to a request that stored the gzip blob, and then failed upstream labelled the new blob with the old row. Re-read the row before falling back so the encoding matches the blob as it is now. Both only become harmful with an encoding change, which this branch introduces; the pre-existing validator-from-row read is tracked separately. * Drop unused cachedMeta.contentEncoding and fix stale doc reference The field was added by #304 and its only reader is replaced in this branch by the encoding parameter passed alongside the body. The proxyCachedWithEncoding comment named conda repodata, which was moved out of this branch in 5991d95; Homebrew is the caller that ships here. --------- Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com> * Fix duplicate fetches and 502s on concurrent cache misses (#329) * Return the stored artifact from storeArtifact, not a reader storeArtifact returned a CacheResult holding an open file handle. A handle has one read position, so it can only ever serve a single caller, which is what blocks sharing one fetch between concurrent requests. Return the artifact and its storage path instead, and let each caller open its own reader through openStoredArtifact. Threading that type through fetchAndCache, fetchAndCacheFromURL and their error paths is mechanical; behaviour is unchanged. * Coalesce concurrent cache misses A cache miss went from checkCache straight to an upstream fetch with nothing tracking in-flight work, so N concurrent requests for one uncached artifact produced N upstream fetches and N stores to the same key. That is the CI shape: parallel jobs installing overlapping dependencies against a cold cache. The duplicate stores also fail requests, racing fileblob's per-key ".attrs" sidecar into a partial read served as a 502. Over 12 runs of 8 simultaneous requests for one uncached tarball, against bb2205a: before, 8 fetches per run and 12 of 96 responses were 502; after, 1 fetch per run and none failed. Route both miss paths through a shared in-flight map keyed on the artifact, including the download URL and upstream-declared hash so callers expecting different bytes never share a fetch. singleflight does not fit: Do gives waiters no way to leave, while DoChan lets the caller running the fetch abandon it, breaking storeArtifact's scan-on-disconnect contract. Deciding roles under a mutex gives both behaviours. The fetch runs on the first caller's context and is seen through; waiters leave when their own clients do. This removes the sidecar trigger on this path. The race is in fileblob and three writers bypass this path entirely, so it is fixed separately. Fewer failures now reach the circuit breaker, so it trips later. Sixteen concurrent callers against real file:// storage fail 10 of 10 runs on main and pass 10 of 10 here. Other tests pin key discrimination, failure propagation, resolver-path coalescing, per-caller readers, waiter cancellation, key release and panic safety. allocs/op is unchanged. mockStorage gains a mutex so concurrent tests can use it. * Normalize digest case in the coalescing key artifactHashMatches compares digests with strings.EqualFold, but the coalescing key used the hash verbatim. The same digest in two casings produced two keys, so two callers for one artifact each ran their own upstream fetch and store, which is what the coalescing is meant to prevent. * Make the panic coalescing test deterministic The test timed the second caller's arrival with a sleep, so which caller became the leader was left to the scheduler. When it lost that race the second caller ran the fetch itself, and its panic was not recovered, so the test binary died instead of the test failing. Whether a caller has reached the wait is not observable from outside: it runs a cache lookup against the database first, so releasing the leader on a timer races that query. Drive coalesceFetch directly and hold the shared entry instead, which removes the timing entirely. The panicking fetcher is no longer needed. * Recheck the cache before running a shared fetch A caller checks the cache before it reaches coalesceFetch, so a fetch that commits in that gap is invisible to it. Arriving after the sharing entry is gone, it became a new leader and fetched, stored and scanned an artifact the cache already held. The leader now rechecks the committed record first. It serves that record only if its bytes still open, because a record can outlive them, and refetching is the recovery the cache lookup already makes for that case. Waiters are unaffected: the record fills the same shared value a fetch would, and each caller opens its own reader from it. The recheck is the leader's alone. A waiter has a fetch in flight to wait on, and rechecking would race it for no gain. * Lock the mock fetcher's bookkeeping Coalescing tests call the handler from many goroutines. The key keeps the fetch itself serialized, but the mock should not lean on that: it now locks the fields it records, so any concurrency the handler applies is safe under the race detector. * Wait for the leader's fetch instead of sleeping The canceled-waiter test slept 200ms and assumed the leader had taken the key by then. On a slow scheduler the canceled call could become the leader and the test would no longer cover waiter cancellation. The fetcher now signals when its first fetch begins, which happens only once the key is held. * Bump google.golang.org/grpc from 1.83.1 to 1.83.2 (#346) Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.83.1 to 1.83.2. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](https://github.com/grpc/grpc-go/compare/v1.83.1...v1.83.2) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-version: 1.83.2 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Stop writing fileblob's .attrs sidecar (#328) * Stop writing fileblob's .attrs sidecar fileblob stores blob metadata in an ".attrs" file per object and rewrites it with os.Create, truncating in place outside the atomic rename that protects the blob. A read overlapping a write decodes a partial file and fails with "opening reader: EOF", served as a 502. One writer against four readers on a single key failed 408 of 2000 reads. cacheMetadataBlob is most exposed to it, rewriting a key on every refresh while readers are served from it. Nothing in the proxy reads what the sidecar holds. gocloud.dev/blob is imported only by internal/storage, Store sets no ContentType, and Attributes is used only for Size, which comes from os.Stat. A missing sidecar already defaults cleanly, so "metadata=skip" removes the hazard rather than locking around it, and saves a write per store. * Clear .attrs sidecars left by earlier versions metadata=skip stops fileblob rewriting sidecars but does not delete ones already on disk, so a sidecar left partial by an interrupted write now fails every read of its key for good. Before, a later store repaired it by rewriting. Store therefore removes the sidecar for the key it writes. Removal is atomic where the rewrite was not, so a concurrent reader gets the whole old file or nothing. Delete already removes sidecars, so the two paths drain a cache between them. Deriving that path is necessary because fileblob's key escaping is unexported. It is the identity for a plain key and parts from one only for keys that are not valid local paths, which is what filepath.Localize rejects. That also keeps the removal inside the cache directory: without it a key holding ".." resolves outside. The clearing test runs one key per storage path the proxy builds, seeded through a bucket that still writes sidecars so the path under test is fileblob's own. * Explain why failed sidecar cleanup does not fail the store Move the removal into clearLegacySidecar and say why its error is dropped rather than returned. A failed removal leaves exactly the state this change inherited, while failing the write would turn a cleanup miss into a failed request. Windows makes that concrete: Go opens files with FILE_SHARE_READ and FILE_SHARE_WRITE but not FILE_SHARE_DELETE, so a reader holding the sidecar open blocks deletion, and that reader is the workload this change exists to protect. Propagating would fail stores during exactly the overlap being fixed. The next store of the key retries. A test pins it, using a non-empty directory at the sidecar path to make os.Remove fail with something other than not-exist on any platform. * Fail the concurrency test if its writer stops The writer returned silently when Store failed, so the test could pass with no concurrent writes at all. Its error is now reported, and the test also checks that at least one write completed. Reporting it showed the writer had been dying on Windows at its first collision: Go opens files without FILE_SHARE_DELETE, so a reader holding the file open makes the writer's rename fail with access denied. The test now skips there, since it cannot contend a writer with readers on that platform. * Clear legacy sidecars for keys fileblob escapes legacySidecarPath declined any key filepath.Localize rejects, which on Windows is every key with a colon: OCI digests and Debian epochs. Their sidecars were never cleared there, and a truncated one kept failing reads, since fileblob still reads a sidecar it finds under metadata=skip. fileblob hex-escapes such characters on the way to disk. The path is now derived the same way, so the sidecar is looked for where fileblob wrote it. Localize still validates the escaped form, which keeps the removal inside the cache directory. * Drop stale comment about declining colon keys on Windows 623ff3e made legacySidecarPath escape keys the way fileblob does, so colon-bearing keys are now cleared on Windows and the OCI and Debian rows in TestStoreClearsLegacyAttrsSidecar prove it. The comment described the behaviour before that commit. --------- Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com> * feat(debian): serve additional APT archives at /debian/{name}/ A Debian release is served by more than one archive. Security updates live on a different host than the main archive, so a single upstream.debian URL cannot serve a complete suite set and -security suites are unreachable. Add upstream.debian_repositories, a name-to-URL map served at /debian/{name}/, modelled on upstream.apk. The field is additive: upstream.debian keeps serving /debian/pool/... and /debian/dists/... with unchanged cache identities, so existing deployments and their warm caches are unaffected, and the scalar field and PROXY_UPSTREAM_DEBIAN are untouched. Named repositories scope both caches by name, since the same filename can hold different bytes in different archives. Metadata keys are hashed over name, upstream URL, and path, as APKHandler does. Names are validated through validateNamedUpstreams, and "pool" and "dists" are refused because they would shadow the main archive's own paths. An unconfigured first path segment stays a main-archive path rather than returning 404 as the APK handler does: the main archive is unnamed and serves paths of its own at the root. * fix(server): tune the shared upstream transport defaults (#351) * fix(server): tune the shared upstream transport defaults server.serve builds the shared client with safehttp.New, which clones Go's default transport: MaxIdleConnsPerHost stays 0 (an effective limit of two idle connections per host) and ResponseHeaderTimeout stays 0. Handing that client to fetch.NewFetcher via fetch.WithHTTPClient replaces the fetcher's own defaults of 10 idle connections per host and a 60-second response-header timeout. Set both on the shared transport, matching the fetcher defaults: a second burst of concurrent cache misses to one registry now reuses its connections instead of re-dialling most of them, and an upstream that accepts a request but stalls before sending headers is cut off after 60 seconds rather than only by the client's overall timeout. Tests measure connection reuse across two concurrent bursts against a TLS upstream that counts accepted connections (Go default: at most two reused; tuned: all eight) and assert that a stall before headers fails with the response-header timeout. Fixes #327 * fix(server): reconcile http_timeout docs and tighten the transport tests The http_timeout documentation and the config comment said "0" disables the upstream timeout entirely. With a fixed 60-second ResponseHeaderTimeout on the shared transport that is no longer the whole story, so both now say that waiting for response headers stays bounded independently of the setting. Test cleanup from review: the 50ms settle between bursts was dead time (the transport returns a connection to the idle pool before the body's final Read returns, so burst returning already means the pool is settled); the maxNewInBurst sentinel became explicit min/max bounds per case; the stall test dropped the client.Timeout override and the elapsed-time assertion, which was redundant with the error-text check in any realistic run and whose failure message misattributed the cause, and its comment now says plainly that the field assertions pin production while the behavioural half runs at a lowered timeout. * test(server): hold each burst at the upstream instead of sleeping The reuse test kept a burst in flight with a 100ms handler sleep, so a process stall longer than that between spawning the goroutines and their dials let a request finish early and hand its connection to a sibling. Review reproduced this with forced stalls: the default-transport case then dialled 5 instead of 6 connections. The handler now answers only once burstSize requests are waiting at the same time. With HTTP/1.1 pinned that forces every burst onto burstSize distinct connections regardless of scheduling, and the measured counts stay exactly 6 new for Go's default and 0 for the tuned transport, also under the same forced stalls. * fix(database): set connection pool limits for Postgres (#350) * fix(database): set connection pool limits for Postgres OpenPostgres returned sqlx.Open's handle with database/sql's defaults: no cap on open connections and two idle ones. Under load nearly every request opened a new Postgres session, ran its few statements and closed it again, paying a backend fork and SCRAM authentication each time. Set the pool limits the issue suggests: 32 open and 32 idle connections, idle connections closed after 5 minutes and every connection recycled after 30 minutes. The values live in named constants because the mnd linter rejects the literals inline. The test (skipped without PROXY_DATABASE_URL, like the other Postgres tests) takes 16 connections from the pool, releases them and checks that all 16 stay idle; with the default pool only two survive. Fixes #323 * test(database): pin the pool properties instead of the wiring Review pointed out that the pool test compared MaxOpenConnections to the constant it was set from, so the assertion followed the constant and would have accepted postgresMaxOpenConns = 0 (unlimited), and that its burst of 16 only proved MaxIdleConns >= 16. The burst now takes postgresMaxIdleConns connections, so every configured idle slot has to survive the release, and the open cap is checked to be finite and large enough for that burst before any connection is taken, so a cap below the idle count fails fast instead of blocking in db.Conn. The doc comment now says which settings the test covers; the idle-time and lifetime settings only show up in DBStats.MaxIdleTimeClosed and MaxLifetimeClosed after minutes of wall-clock time and stay unexercised. * test(database): guard the pool test against a vacuous burst Review showed that with the burst tied to postgresMaxIdleConns the test also passed for a constant of 2, database/sql's default, or of 0, where it took no connections at all. It now fails outright unless the configured idle count exceeds the default, and every connection it takes is released in a cleanup, so an assertion failure mid-burst no longer leaves sessions open for the rest of the test binary. * test(database): drop every schema table in the Postgres fixture (#349) createTestPostgresDB dropped artifacts, versions, packages and schema_info before calling CreateSchema, but not the migrations table. On a database that has seen one test the migration records survive, so the next CreateSchema fails while recording 001_add_packages_enrichment_ columns with a duplicate key on migrations_pkey. Running the package against one Postgres therefore failed from the second Postgres-backed test on. Drop vulnerabilities, metadata_cache and migrations as well, so the fixture clears every table CreateSchema creates. The package now passes repeatedly against the same database. * Discard a stale cache entry under the coalescing key (#348) The digest-aware cache check discarded a stale entry before its caller took the key. A slow caller could delete an entry another caller's fetch had just committed, and everyone sharing that fetch then failed to open it. The check now only reports the miss. The caller running the shared fetch discards the entry under the key, after the recheck, so one a previous fetch refreshed is served, not deleted. The mismatch warning fires once per refresh instead of once per request. Swift HEAD no longer discards either, having no fetch to do it under. It probes upstream as before, and the next GET replaces the entry. Different digests or URLs, or no digest, use different keys and can still collide on the storage path. That is the storage layout follow-up. * Separate OCI request timeout from readiness probes in loopback test (#347) The 250ms client is meant for the readiness poll, where a timeout is retried. Reusing it for the OCI manifest request makes the test flake under -race on Windows CI when fetch and cache I/O take longer, as seen on #328. The request checks upstream routing, not latency. * Fix NuGet cooldown enforcement for listings and downloads (#340) * fix(nuget): enforce cooldown across metadata and downloads * fix(nuget): support legacy registration and preserve valid metadata cache * refactor(nuget): address maintainer review cleanup * Fall back to embedded build info for Version (#352) go install does not apply goreleaser's -ldflags -X, so binaries installed that way reported 'dev'. Read debug.BuildInfo.Main.Version when the ldflag is unset. * docs(debian): trim comments on the named-repositories change The motivation for separate archives was repeated across six files and the reserved-name rationale across five. Keep each in one place -- the reference docs -- and leave the code comments to what an informed reader cannot get from the code: why an unknown first segment is not a 404 as it is for APK, and why the main archive keeps its legacy cache identities. Comments and prose only; no behaviour change. * docs(debian): drop a stray space in a test comment * Bump github.com/aws/aws-sdk-go-v2/service/ecr from 1.64.0 to 1.65.0 (#357) Bumps [github.com/aws/aws-sdk-go-v2/service/ecr](https://github.com/aws/aws-sdk-go-v2) from 1.64.0 to 1.65.0. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.64.0...service/s3/v1.65.0) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/service/ecr dependency-version: 1.65.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump github.com/git-pkgs/spdx from 0.3.1 to 0.3.2 (#355) Bumps [github.com/git-pkgs/spdx](https://github.com/git-pkgs/spdx) from 0.3.1 to 0.3.2. - [Commits](https://github.com/git-pkgs/spdx/compare/v0.3.1...v0.3.2) --- updated-dependencies: - dependency-name: github.com/git-pkgs/spdx dependency-version: 0.3.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump zizmorcore/zizmor-action from 0.6.3 to 0.6.4 (#354) Bumps [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) from 0.6.3 to 0.6.4. - [Release notes](https://github.com/zizmorcore/zizmor-action/releases) - [Commits](https://github.com/zizmorcore/zizmor-action/compare/70fb788f84895a7701f5643d103d587e460b5c99...cc914d7f3750a2d13d75c7f184a1060aa0e9d482) --- updated-dependencies: - dependency-name: zizmorcore/zizmor-action dependency-version: 0.6.4 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump github.com/aws/aws-sdk-go-v2/config from 1.33.2 to 1.33.4 (#356) Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.33.2 to 1.33.4. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.33.2...config/v1.33.4) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/config dependency-version: 1.33.4 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat(debian): report a misspelled repository name A request to /debian/{name}/ for a name that is not configured fell through to the main archive, so a typo surfaced as the upstream's own HTML 404 with no mention of the repository. The reply now names the unknown repository and lists the configured ones. Scoped to {name}/dists/ and {name}/pool/. The main archive serves README, indices/ and project/ at its root, so an unknown first segment stays a main-archive path everywhere else. apt prints only the status line, so this is visible via curl or the proxy log rather than in apt output. * fix(helm): support OCI references in HTTP repository indexes * feat: add exact-version package denylist * Exit non-zero when mirror denylist is invalid * Omit malformed Helm chart releases instead of failing the index A single bad release in a third-party index made the whole repository unusable through the proxy. * End an eviction pass that cannot evict anything A failed delete leaves the record eligible, so the same batch came back on the next iteration while the recorded size never dropped. Against a backend refusing deletes the pass retried forever, burning CPU and flooding the log. A canceled context made it faster: every delete fails at once, so the loop spun as quickly as the database could answer. The pass now ends when a batch clears no record, and returns as soon as the context is canceled. Progress counts records rather than bytes, because a cached record can have no size recorded. A pass can therefore end with the cache still above its limit, leaving the records that failed for the next pass to retry. * Fix the handler test build The denylist tests call NewDebianHandler with three arguments, but the additional APT archives change gave it a fourth. Each passed CI on its own branch, and together they leave internal/handler without a test binary that compiles, so go vet and go test fail on main. * docs: explain APT denylist limitations and version pinning * Assert omitted Helm releases are not downloadable * build(deps): bump go.opentelemetry.io/otel/sdk from 1.44.0 to 1.45.0 Bumps [go.opentelemetry.io/otel/sdk](https://github.com/open-telemetry/opentelemetry-go) from 1.44.0 to 1.45.0. - [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md) - [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.44.0...v1.45.0) --- updated-dependencies: - dependency-name: go.opentelemetry.io/otel/sdk dependency-version: 1.45.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * Widen wall-clock bound in flaky probe timeout test Loaded CI runners exceeded the 500ms bound; 5s still proves the probe timeout fired instead of blocking on Store. * Add and align CI security and dependency automation * fix(mirror): match cached artifacts by resolved filename * Standardize the README license footer * Test that GCS Delete of a missing object returns nil * Give each fetch its own storage path Every fetch of an artifact wrote one path, so fetches with different URLs or digests, which do not share a coalescing key, could overwrite each other's bytes, serve the wrong ones, or delete them on a digest mismatch while the other was about to open them. storeArtifact now writes {ecosystem}/{name}/{version}/{fetch id}/{filename}, with a random id per fetch. Most ecosystems learn the digest only after the fetch, and storage has no rename, so the id is the one rule that fits all of them. Existing records keep their paths and stay readable. On a file:// bucket, Delete now removes an emptied fetch directory, which fileblob leaves behind, and no other directory, since another fetch may be creating its own inside it. A path a record stops pointing at is queued in pending_deletes rather than deleted, since a request that read the record may still open it. A loop deletes queued paths after max(1h, direct_serve_ttl), whether or not max_size is set. UpsertArtifact and clears apply only if the record still holds the path the caller read, so racing commits each queue the path they replaced and a clear never orphans a newer commit. Eviction still deletes inline, and counts space as freed only when the record still pointed at what it deleted. * build(deps): bump docker/setup-buildx-action from 4.3.0 to 4.4.1 Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 4.3.0 to 4.4.1. - [Release notes](https://github.com/docker/setup-buildx-action/releases) - [Commits](https://github.com/docker/setup-buildx-action/compare/37fe631027851001ddb9b187196cc803df7f5f0e...f87e5991a6d7451dcb8d9637bfbc97413f497069) --- updated-dependencies: - dependency-name: docker/setup-buildx-action dependency-version: 4.4.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * build(deps): bump docker/setup-qemu-action from 4.3.0 to 4.4.0 Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 4.3.0 to 4.4.0. - [Release notes](https://github.com/docker/setup-qemu-action/releases) - [Commits](https://github.com/docker/setup-qemu-action/compare/1f40c72289eff860ee54a304f1438e3cff362e0a...99012661954931238ded8c8b007157a8430204e1) --- updated-dependencies: - dependency-name: docker/setup-qemu-action dependency-version: 4.4.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * build(deps): bump docker/build-push-action from 7.3.0 to 7.4.0 Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 7.3.0 to 7.4.0. - [Release notes](https://github.com/docker/build-push-action/releases) - [Commits](https://github.com/docker/build-push-action/compare/53b7df96c91f9c12dcc8a07bcb9ccacbed38856a...c3c9e263c25d99ce0380d002d59b67737d91b0dc) --- updated-dependencies: - dependency-name: docker/build-push-action dependency-version: 7.4.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * build(deps): bump github.com/aws/aws-sdk-go-v2/service/ecr Bumps [github.com/aws/aws-sdk-go-v2/service/ecr](https://github.com/aws/aws-sdk-go-v2) from 1.65.0 to 1.66.0. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.65.0...service/s3/v1.66.0) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/service/ecr dependency-version: 1.66.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * build(deps): bump golang.org/x/net from 0.58.0 to 0.59.0 Bumps [golang.org/x/net](https://github.com/golang/net) from 0.58.0 to 0.59.0. - [Commits](https://github.com/golang/net/compare/v0.58.0...v0.59.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-version: 0.59.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * build(deps): bump github.com/git-pkgs/registries from 0.9.1 to 0.9.2 Bumps [github.com/git-pkgs/registries](https://github.com/git-pkgs/registries) from 0.9.1 to 0.9.2. - [Release notes](https://github.com/git-pkgs/registries/releases) - [Commits](https://github.com/git-pkgs/registries/compare/v0.9.1...v0.9.2) --- updated-dependencies: - dependency-name: github.com/git-pkgs/registries dependency-version: 0.9.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * build(deps): bump github.com/git-pkgs/magic from 0.3.1 to 0.4.0 Bumps [github.com/git-pkgs/magic](https://github.com/git-pkgs/magic) from 0.3.1 to 0.4.0. - [Release notes](https://github.com/git-pkgs/magic/releases) - [Commits](https://github.com/git-pkgs/magic/compare/v0.3.1...v0.4.0) --- updated-dependencies: - dependency-name: github.com/git-pkgs/magic dependency-version: 0.4.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * build(deps): bump modernc.org/sqlite from 1.58.0 to 1.59.0 Bumps [modernc.org/sqlite](https://gitlab.com/cznic/sqlite) from 1.58.0 to 1.59.0. - [Changelog](https://gitlab.com/cznic/sqlite/blob/master/CHANGELOG.md) - [Commits](https://gitlab.com/cznic/sqlite/compare/v1.58.0...v1.59.0) --- updated-dependencies: - dependency-name: modernc.org/sqlite dependency-version: 1.59.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * build(deps): bump github.com/git-pkgs/vers from 0.7.0 to 0.7.1 Bumps [github.com/git-pkgs/vers](https://github.com/git-pkgs/vers) from 0.7.0 to 0.7.1. - [Commits](https://github.com/git-pkgs/vers/compare/v0.7.0...v0.7.1) --- updated-dependencies: - dependency-name: github.com/git-pkgs/vers dependency-version: 0.7.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * build(deps): bump github.com/aws/aws-sdk-go-v2/config Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.33.4 to 1.33.5. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.33.4...config/v1.33.5) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/config dependency-version: 1.33.5 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * build(deps): bump github.com/git-pkgs/archives from 0.7.0 to 0.7.1 Bumps [github.com/git-pkgs/archives](https://github.com/git-pkgs/archives) from 0.7.0 to 0.7.1. - [Commits](https://github.com/git-pkgs/archives/compare/v0.7.0...v0.7.1) --- updated-dependencies: - dependency-name: github.com/git-pkgs/archives dependency-version: 0.7.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * fix(conda): preserve gzip for large repodata indexes * fix: centralize pass-through response relaying * Clear before deleting on eviction, and requeue failed reclaims Eviction deleted an object before checking its record still pointed at it. If a refetch had moved the record, that object was queued and a request that read the record earlier could still be opening it. Eviction now clears first, skips the delete if the record moved, and queues the object when the delete fails after a clear. Reclaim queues a failed delete again, which moves it behind the rest, so objects the backend keeps refusing cannot fill every batch. Also drop the clearArtifactCache passthrough and ClearCachedArtifact's unused context, note that only tests call ArtifactPath, and document that max_size does not count objects waiting to be reclaimed. * Log and count requests aborted mid-relay A truncated relay panics with http.ErrAbortHandler, which skipped the request log line, the request metrics and the access log entry. * Batch cache hit writes Every cached download wrote its hit in its own UPDATE. SQLite runs on one connection, so each download was a write transaction the rest queued behind, holding a warm npm cache near 1,000 requests a second with the CPU two thirds idle. Hits are now counted in memory and written in one transaction every database.hit_flush_interval, "1s" by default, "0" for a write per hit. A failed write keeps the hits for the next flush, and Close writes what is pending. The same test then served about 2,200 requests a second. Closes #322. * fix pnpm audit * apply review changes * Name every mounted route in requestEcosystem - Map /apk to alpine, matching what the handler writes to the database - Report /helm, /homebrew, /generic and /swift under their own names - Test every one of the 23 mounts, so the next route added cannot quietly fall through to "other" Part 1 of 6 splitting #381 up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(handler): stop asserting on an upstream connection the proxy abandons TestRelayRoutes fails intermittently on CI with "write: connection reset by peer" at the upstream handler's final Flush, on /hex/packages/demo and /gem/info/demo. Those two routes are the only ones in the table that fan out a second upstream request: with cooldown enabled, the hex and gem handlers fetch version timestamps concurrently with the artifact. The test server answers that sidecar request with the same 64 KiB chunked 502, but the handler returns as soon as it sees a non-200 and closes the body unread, so the proxy drops the connection while the test server is still writing to it. Whether the reset lands before or after the Flush is a scheduling race, which is why it only surfaces on loaded runners. The flush error says nothing about the response under test, which is asserted on the downstream side, so stop reporting it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Per-ecosystem cache and download statistics - Add GetEcosystemStats, aggregating packages, versions, artifacts, cache size, downloads and downloaded bytes per ecosystem - Publish six proxy_ecosystem_* gauges from it on the existing one-minute cache-stats tick - Set and selectively delete rather than Reset, so no scrape lands on a half-populated vector - Report artifacts with no package row under "unattributed" rather than dropping them, so the figures still add up Part 2 of 6 splitting #381 up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Add the /ui/analytics page - Ring of download size by ecosystem, cache figures, per-ecosystem table, vulnerability overview and a Runtime card - Add metrics.Gather so the page can render counters that were never in the database - Add proxy_response_bytes_total and response-writer byte counting - Serve a retained snapshot behind a staleness banner when the aggregation fails, rather than rendering old figures as current - Extract the security overview into a shared component Part 3 of 6 splitting #381 up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Report the per-ecosystem breakdown from GET /stats - Add downloaded_bytes, downloads and an ecosystems array, served from the snapshot the gauges and the page already share - Add stats_unavailable so a failed aggregation is distinguishable from an idle proxy - Regenerate the OpenAPI spec Part 4 of 6 splitting #381 up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Add a Grafana dashboard - 33 panels, no hardcoded data source UID - Three ecosystem variables, because the label means three different things across /metrics: the package record, the request path, and the handler's own name Part 5 of 6 splitting #381 up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Attribute requests to a caller and a client tool - Track per-caller request and byte counts in a bounded table - Export proxy_client_requests_total and proxy_client_response_bytes_total over a closed label set; addresses are never labels - Gate the sources card behind ui_request_sources, off by default, since /ui carries no authentication of its own - Move access_log.trust_forwarded_for to a top-level key - Keep remote as the TCP peer and add remote_ip alongside it Part 6 of 6 splitting #381 up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Update enrichment to v0.7.2 (#394) * Update ecosystems-go to v0.5.0 * Update enrichment to v0.7.2 * Update git-pkgs dependencies (#396) * Keep hit timestamps from moving backwards With proxies sharing a database, a batch can flush after a newer hit is already written, and overwriting last_accessed_at and updated_at with its older time could reorder LRU eviction. Both now keep the later of the stored and incoming times. The count still adds up. * build(deps): bump alpine from 3.24.1 to 3.24.2 (#397) Bumps alpine from 3.24.1 to 3.24.2. --- updated-dependencies: - dependency-name: alpine dependency-version: 3.24.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Install development tools directly in CI (#398) * Add storage.cache_artifacts to serve artifacts without storing them (#387) * Add storage.passthrough to serve artifacts without storing them With storage.passthrough enabled the proxy streams every artifact from upstream to the client and never writes it to storage or the cache database. Metadata filtering, cooldown and the denylist work as before, so the proxy can sit behind another cache (e.g. an Artifactory remote) purely as a policy layer without holding a second copy of every package. Artifacts whose digest is known up front (OCI, Swift, Helm) are verified while streaming. Their responses are sent chunked and the connection is aborted on a mismatch, so a client never receives a tampered artifact as a complete response. Passthrough is rejected together with scanning, direct_serve and mirror_api, and the mirror command refuses to run with it, since all of them depend on stored artifacts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Address review: abort truncated streams, guard every cache read - serveArtifact aborts the response on any body read error, and when fewer bytes than the declared size were written, not only on a digest mismatch. A failed or short upstream body in passthrough mode no longer reaches the client as a complete 200. - Streamed upstream read failures are logged and counted as stream_failed upstream errors before the response is aborted. - checkCache reports a miss when artifact caching is off, so every cache read path, including Swift archive HEAD requests, ignores entries stored before the mode was enabled. Denylisted versions are still rejected. - Rename storage.passthrough to storage.cache_artifacts (default true, PROXY_STORAGE_CACHE_ARTIFACTS), which names what actually changes. Tests cover an OCI blob shorter than its Content-Length and an npm tarball whose chunked response ends without the final chunk, both served through the HTTP handlers, plus a Swift HEAD request against a cached archive. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Rename Proxy.Passthrough to StreamArtifacts Match the storage.cache_artifacts option: the field says what changes (artifacts are streamed instead of stored) and its zero value keeps the usual caching behaviour for every Proxy built without the server config, such as the mirror command and tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> * Show the by-client table regardless of ui_request_sources The flag exists to withhold caller addresses. The by-client table carries none, and the same figures are already public at /metrics, so gating it left two metrics with no tile on the page in the default config. * Drop unused registry from TestUpdateEcosystemStatsSumsAliasedRows * Cache rewritten npm and Composer metadata (#402) The npm and Composer handlers rewrite every metadata document they serve so that download URLs point at the proxy: decode the whole document into generic maps, change the URLs, encode it again, and for Composer expand the minified format first. That ran on every request, cached metadata included. With metadata caching on and upstream out of the picture, a cached request still cost 7 ms for @babel/core, 17 ms for symfony/console and 155 ms and 134 MB of allocations for typescript, and an 8 vCPU VM serving cached npm packuments to 20 clients ran the proxy at 450% CPU for about 1,100 requests a second. Rewritten documents are now kept in memory, keyed by the ecosystem, proxy URL, package and a SHA-256 of the raw document, so new bytes from upstream are rewritten again and nothing is served stale. Requests that arrive while a document is being rewritten wait for that rewrite rather than running their own; a waiter leaves when its client does, and the rewrite still completes and is cached. The denylist is fixed at startup, so it needs no place in the key. Cooldown filtering depends on the current time, so with cooldown on the cache is bypassed. metadata_rewrite_cache_size bounds the cache (default "256MB", least recently used out first, "0" to rewrite on every request). NewProxy callers keep the old behaviour unless they set it. Whole cached requests, measured locally: @babel/core 7.1 ms -> 0.38 ms, 54,489 -> 119 allocations typescript 155 ms -> 10.5 ms, 1.37M -> 133 allocations symfony/console 17.4 ms -> 0.48 ms, 165,909 -> 123 allocations What remains is reading the raw document from storage and hashing it. * Copy inherited Composer fields shallowly when expanding (#401) expandMinifiedVersions deep-copied every inherited field into every version (b68184c) so that rewriting one version's dist URL in place could not change the versions that inherited it. For a package with a long history that means recursively copying require, autoload and the rest hundreds of times per document, on every metadata request. dist is the only field the proxy changes after expansion, so copy just that: rewriteDistURL now gives the version its own dist map before setting the URL, and expansion shares the other inherited values. TestComposerExpandMinifiedSharedDistReferences still guards the original bug, and fails if the copy in rewriteDistURL is removed. On the symfony/console metadata from Packagist the rewritten output is byte-identical, and the rewrite drops from about 20.7 ms to 13.8 ms, with allocations down from 13.6 MB to 9 MB. * Support HTTP byte-range requests for local file cached artifact downloads (#404) * feat(storage): expose efficient seek for local files Open file-backed artifacts with os.Open so local readers support efficient seeking. Leave cloud storage readers unchanged. Keep the legacy sidecar mapping test focused on fileblob's reader: Blob.Open now reads local files directly and no longer consults the sidecar. Continue checking sidecar cleanup and post-store reads. * feat(handler): preserve efficient seek through integrity checks Keep the cache result's single Reader field and preserve io.Seeker when the underlying storage reader supports efficient seeking. Sequential reads continue through whole-object integrity verification. A successful seek switches subsequent reads to the source for range responses, where full-object verification is not possible. Test both modes and verify the source is closed once. * feat(handler): add request-aware byte range responses Add a request-aware artifact serving helper for bounded, open-ended, and suffix byte ranges, including If-Range handling, 206 responses, and 416 responses for valid unsatisfiable ranges. Ignore malformed and multi-range requests by serving the full artifact. Advertise byte-range support only for seekable readers, preserving HEAD, redirect, and non-seekable behavior. Clarify that net/http recovers ErrAbortHandler per request and keeps the server running. * feat(handler): enable byte ranges across artifact downloads Forward each artifact request to the shared range-aware response helper, including cached generic release assets and OCI blobs. This lets the handler honor ranges only when its reader supports efficient seeking, without changing direct-storage redirect behavior. Add warm-cache endpoint tests that assert partial response headers and bytes without an upstream refetch. Document supported range behavior, unsupported reader fallback, and the integrity limitation of partial reads. * Coalesce concurrent metadata fetches (#400) * Coalesce concurrent metadata fetches Concurrent cache misses for one artifact already share a single upstream fetch (#329), but metadata misses did not: every request for a package's metadata went to the registry, even when the same document was already being fetched. In CI that is the common case. Composer resolves versions on every job, since MediaWiki core and its extensions commit no composer.lock, and each package is one /p2/ document, so a burst of jobs fetched each document once per job. Metadata misses now follow the same pattern as artifacts: the first caller fetches, the rest wait for its result, and the caller that takes the key rechecks the cache first so a fetch that just finished is not repeated. The key includes the Accept and Accept-Encoding headers, which change the bytes upstream returns (npm's abbreviated and full documents), and whether the caller validates the response. One difference from artifacts: the shared fetch is detached from the first caller's cancellation. Artifact fetches keep it for scanning and mirroring, neither of which applies to metadata, and otherwise one job disconnecting would fail every job waiting on the same document. runScan detaches the same way, and the HTTP client's timeout still bounds the fetch. This covers every ecosystem that goes through fetchOrCacheMetadata (Composer, npm, PyPI, Maven, Cargo, Swift, Pub, Helm, NuGet), with metadata caching on or off. Streamed metadata with caching off is not coalesced. On an 8 vCPU test instance with caching off, 10 concurrent clients requesting the npm metadata of 1,263 packages went from 12,630 upstream requests to 1,898. With cache_metadata on, 20 clients requesting 1,538 packages made exactly 1,538. * Run each caller's validate on a shared metadata fetch A caller that joined another's metadata fetch got the shared bytes without running its own validate. NuGet's cooldown path uses validate to decode the document for the request, so a joined download found no publication date and was allowed, bypassing the cooldown. coalesceMetadata now reports whether the caller joined, and coalescedMetadataMiss runs that caller's validate on the shared bytes. Validation before caching is unchanged: the first caller still runs it inside the fetch. TestNuGetCooldownConcurrentDownloads reproduces the bypass through NuGetHandler.Routes() with two overlapping downloads of a version inside its cooldown. * Support npm content-addressed tarball URLs * Add cooldown package pattern overrides (#407) * Add cooldown package pattern overrides * Expose full cooldown decisions through pattern policy * Reject unsafe cooldown pattern syntax * Fix cached package ecosystem filters * Update supportedEcosystems comment for filter pills --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: pinguinfuss <pinguinfuss@users.noreply.github.com> Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com> Co-authored-by: montehurd <montehurd@users.noreply.github.com> Co-authored-by: Andrew Rich <andrew.rich@beacon.bio> Co-authored-by: Abhinav Gautam <abgautam1017@gmail.com> Co-authored-by: montehurd <mhurd@wikimedia.org> Co-authored-by: wickedOne <wicliff.wolda@gmail.com> Co-authored-by: Wicliff Wolda <wicliff.wolda@quentic.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Daniil Skripchenko <90247855+DANIILSKRIPCHENKO@users.noreply.github.com> Co-authored-by: PhantomPhoton <120879831+PhantomPhoton@users.noreply.github.com>
Bump dependencies ahead of v0.7.0 (#235) - github.com/git-pkgs/archives v0.4.0 -> v0.5.0 - github.com/git-pkgs/magic v0.1.0 -> v0.2.0 - github.com/git-pkgs/spdx v0.1.4 -> v0.3.0 - github.com/prometheus/client_golang v1.24.0 -> v1.24.1 - modernc.org/sqlite v1.55.0 -> v1.56.0 - golang.org/x/text v0.38.0 -> v0.40.0 (fixes GO-2026-5970)
Bump google.golang.org/grpc from 1.81.1 to 1.82.1 (#216) Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.81.1 to 1.82.1. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.81.1...v1.82.1) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-version: 1.82.1 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Polish UI: lucide icons, sticky footer, hamburger nav, clickable logo (… …#162) - Move third-party JS into static/vendor/ and add lucide for icons; refine .gitignore so embedded vendor dirs aren't caught by the Go vendor rule. - Replace folder/file emojis in the source browser with lucide icons. - Wrap the header logo and title in a single anchor so the icon is clickable. - Drop the redundant "Powered by git-pkgs" footer block; add a GitHub repo link to the About column and bump the ecosystem count from 16+ to 17+. - Sticky footer pattern: body is min-h-full flex column with main growing to fill, so the footer sits at the bottom of short pages. - Hamburger menu under md: search and nav links collapse into a drawer toggled by a menu button; theme toggle stays visible at both sizes.
PreviousNext