Stop /api/export from starving the server, and stop timers dying silently - #8

What was happening

The origin was very likely never crashing. /api/export was holding the event loop long enough for the container HEALTHCHECK to fail its full retry budget, and whatever sits in front of the container then stopped routing to it — 502 from nginx/Caddy, 404 from Traefik. It recovers on its own, which is why it reads as "the project restarted on a fatal error".

The export was built on a generator specifically so it would not materialise the whole history in memory, but the route consumed that generator inside ReadableStream's start(). start() runs to completion before anything reads the stream, so the laziness bought nothing: every event was encoded into the stream's internal queue — which applies no backpressure of its own — while the loop was held.

Measured

300k events (357 MB database), served to a deliberately slow 2 MB/s client, next start in production mode:

Three consecutive 5 s timeouts is exactly HEALTHCHECK --timeout=5s --retries=3.

Changes

app/api/export/route.ts — consume the generator from pull() instead of start(), so a batch is only produced once the consumer has drained the previous one, and cancel() stops querying when the client hangs up.

lib/server/export.ts — pull() alone was not enough. stmt.iterate() holds a cursor open on the connection, and better-sqlite3 rejects every write on a connection with an open cursor (This database connection is busy executing a query) — verified directly. Since the connection is the process-wide singleton from runtime.ts, consuming that cursor lazily would have made the collectors, the run log and every admin POST fail for the whole duration of an export. So the generator now reads fixed-size batches by keyset — each one a complete, closed query, with (ts_utc, id) as the cursor so events sharing a timestamp are neither skipped nor duplicated.

lib/server/scheduler.ts — three synchronous SQLite calls sat outside any try in the timer paths, where a throw is unrecoverable:

• likedDue() / backupDue() are evaluated in the timer callback itself, so a throw escapes the scheduler and no further tick is ever scheduled;
• in the playback ticker's finally, the hourly summary and the sample purge run before the setTimeout that re-arms the loop, so a throw there ends playback collection permanently.

Neither is what takes the UI down, but both fail silently — the only symptom is collection quietly stopping. They now log and carry on.

Verification

• Export correctness against the same 300k rows: 300006 lines, all ids distinct, both sort orders monotonic, type= and from= filters matching the database's own counts, X-Event-Count consistent.
• CLI scripts/export.ts (which shares the generator) still emits the same 300006 lines.
• npm run typecheck, npm run lint (only the two pre-existing warnings), npm run build all clean.

Not addressed

raw_spotify has no retention policy — insertRaw stores the full response payload on every collector call and nothing ever purges it. That is what makes the database large enough for the above to bite, and on a home lab it grows toward a full disk. Worth a follow-up, but it is a separate change.

Generated by Claude Code