A stop that outlives the graceful-stop budget, or a SIGKILL, wipes the whole dagql cache on the next start #14956

What happens

When an engine with a large dagql cache stops, the "close dagql cache" phase persists the in-memory state and only then writes clean_shutdown=1 in the close path of dagql/cache.go. If that phase does not finish, the marker stays 0. On the next start the engine logs dagql persistence store marked unclean; wiping and cold-starting and local cache state invalid; resetting worker and dagql persistence state reason=dagql_unclean_shutdown, and the cache starts empty, with the worker's local cache state reset too.

The persist does not finish in two cases:

1. The default stop budget, defaultGracefulStopTimeout = 5 * time.Minute in cmd/engine/stop_budget.go, runs out partway through the persist. One engine logged failed to persist dagql cache during close err="insert snapshot_content_hash (HHO5L4B2C4XMXAKSFJAI6RFCFY): context deadline exceeded" after close dagql cache had run 4m54s. Its graceful stop ended at 5m32s with that error, and the next start wiped the cache.
2. The stop never reaches the persist. A second engine sat in wait for garbage collection for 30 minutes, until docker stop -t 1800 killed it with exit 137, and the next start wiped the cache.

Measured on dagger main 8afd98b plus a small local patch, self-hosted engines on 64- and 128-vCPU machines with local NVMe, each restarted idle with no client connected. Engines are labelled by letter:

An empty engine's first run of this workspace takes about 12 minutes, against 1-6 minutes for the same change classes on a warm one.

Why this matters for long-running engines

In-memory dagql state grows over hours, and checks on an engine three or more hours old ran slower: one dependency-bump run took 16m27s against 5m46s on a fresher pool. A periodic restart is the natural fix for that, but the engines whose caches are largest, and so most worth keeping, are the ones whose stop outlives the default budget.

Asks

• Persist incrementally, or checkpoint while running, so a cut-off close loses only what changed since the last checkpoint and does not leave the store marked unclean.
• If a wipe is still needed, scope it to what the unclean close could have left inconsistent, not the whole store.
• Bound wait for garbage collection within the stop budget, so the stop reaches the persist.
• Document --graceful-stop-timeout / shutdown.gracefulStopTimeout next to the stop-timeout guidance for self-hosted engines, and that it must stay below the container runtime's stop timeout.

Related: #14926 (Address.container resolving per call) and #14903 (git-remote workspace cache keys), from the same self-hosted CI setup.