Visitar URL original
feat(webapp,run-engine): queue metrics and health dashboard (#4131) · triggerdotdev/trigger.dev@4eb9292 · GitHub
Skip to content

Commit 4eb9292

Browse files
ericallamkathiekiwisamejr
authored
feat(webapp,run-engine): queue metrics and health dashboard (#4131)
## Summary Three related changes, each independently gated: **Queue metrics and health.** Per-queue depth, throughput (enqueued, started, completed), concurrency, whether a queue is throttled, and scheduling delay (how long a run waits between becoming eligible and actually starting), plus a per concurrency-key breakdown for keyed queues. Collected from inside the run queue itself, stored in ClickHouse, and surfaced on the Queues list, a new per-queue detail page, the task pages, and the run inspector. The question it answers is "does this queue have enough concurrency to keep up, and if not, which key or which limit is the constraint". **Percent-based queue concurrency limits.** A queue's concurrency override can now be expressed as a percentage of the environment limit, stored as the source of truth and re-materialized whenever the environment limit changes. Absolute overrides above the environment limit are now **rejected with a 400** instead of being silently capped, which is a behavior change on `POST /api/v1/queues/:queue/concurrency/override`. **The `health` report.** A server-computed verdict on whether work is flowing, whether the runs that do start are healthy, and whether telemetry is fresh, rendered as text with sparklines. Available as `GET /api/v1/reports/:key`, `trigger report`, and the `get_report` MCP tool (plus a `report` MCP prompt, which shows up as a slash command in hosts that support prompts). With the flags off, the Queues page renders the pre-metrics component verbatim, nothing is emitted, and nothing is written to ClickHouse. ## Configuration Two independent gates, on purpose. Emission is global so data accrues for everyone before anyone can look at it; the view is per organization so it can be turned on for one org at a time without a deploy. **Runtime flags (no restart)** | Flag | Store | Gates | | --- | --- | --- | | `queue_metrics:enabled` | run-queue Redis key (`"1"`/`"0"`, off by default) | All emission, gauges and counters. Cached in-process for 10s with stale-while-revalidate, warmed eagerly at boot so the first op after a deploy is not dropped. | | `queue_metrics:gauge_sample_rate` | run-queue Redis key, `0..1` | Fraction of queue ops that emit a gauge. Counters are never sampled, so throughput stays exact at any rate. | | `queueMetricsUiEnabled` | feature-flag catalog: global `FeatureFlag` row, per-org `Organization.featureFlags` override wins | Whether an org sees the metrics view at all: the Queues list variant, the queue detail route, the built-in Queues dashboard, the concurrency-keys endpoint, and the metrics blocks on task pages and the run inspector. Off by default; a gated org gets a 404 on the detail route rather than an empty page. | Both Redis keys are readable and writable from `/admin/queue-metrics` (super-admin UI, with a live per-shard stream-health table) and `GET`/`POST /admin/api/v1/queue-metrics` (admin PAT). The admin surface uses its own Redis client, so it works on any instance regardless of whether that instance runs the emitter or the consumer. **Environment variables (boot time)** | Variable | Default | Notes | | --- | --- | --- | | `QUEUE_METRICS_EMIT_ENABLED` | `0` | Constructs the emitter and injects it into the run engine. Without it the run queue has no emitter at all. | | `QUEUE_METRICS_CONSUMER_ENABLED` | `0` | Boots the stream consumer on this instance. Independent of emission, so consumers can be sized separately from the API. | | `QUEUE_METRICS_STREAM_SHARD_COUNT` | `4` | Stream shards, hashed per queue. | | `QUEUE_METRICS_CONSUMER_BATCH_SIZE` | `1000` | Poll batch equals insert batch, so an ack can never outrun a write. | | `QUEUE_METRICS_REDIS_{HOST,PORT,USERNAME,PASSWORD,TLS_DISABLED}` | falls back to the run-queue Redis | Set `HOST` to move the metrics stream onto a dedicated instance so a metrics backlog cannot compete with the run queue for memory. Self-hosters can leave it unset and get a single-Redis deployment. | | `QUEUE_METRICS_COUNTER_STREAM_MAXLEN` | `2000000` shared, `8000000` dedicated | Bound on how much a stalled consumer can hold. The default is deliberately lower when the stream shares the queue-critical Redis. | | `QUEUE_METRICS_COUNTER_ODOMETER_TTL_SECONDS` | `604800` | TTL on the per-queue cumulative counter key, refreshed on every write, so only queues idle for the whole window are purged. | | `QUEUE_METRICS_MAX_QUEUE_NAMES_PER_ENV` | `1000` | Distinct queue names tracked per environment; overflow collapses into `__overflow__`. | | `QUEUE_METRICS_MAX_CONCURRENCY_KEYS_PER_QUEUE` | `10000` | Same idea one level down, per queue. | | `QUEUE_METRICS_GAUGE_SAMPLE_RATE` | `1` | Default for the live sample-rate key above. | | `QUEUE_METRICS_QUERY_TABLES_VISIBLE` | `0` | Lists the queue-metrics tables in the Query page, its schema docs, the schema API and the AI query context. Off keeps them unlisted while the feature is dark; a query naming them still runs either way. | | `QUEUE_METRICS_CLICKHOUSE_URL` | falls back to the shared wiring | Runs queue metrics on their own ClickHouse service: the consumer's inserts and every queue-metrics read go through it, so a metrics-heavy chart refresh never competes with runs-list or trace reads. Unset reproduces the previous split exactly (inserts on `CLICKHOUSE_URL`, reads on the query pool). | | `QUEUE_METRICS_CLICKHOUSE_READER_URL` | the write URL | Reader split, so the consumer's inserts can never land on a read endpoint. | | `QUEUE_METRICS_CLICKHOUSE_{KEEP_ALIVE_ENABLED,KEEP_ALIVE_IDLE_SOCKET_TTL_MS,MAX_OPEN_CONNECTIONS,LOG_LEVEL,COMPRESSION_REQUEST}` | `1`, unset, `10`, `info`, `1` | Pool tuning, matching the other per-workload ClickHouse clients. | Migrations to apply: ClickHouse `036_create_queue_metrics_v1.sql`, and a Postgres migration adding the nullable `TaskQueue.concurrencyLimitOverridePercent`. Both are additive. ## How collection works Queue operations produce two kinds of signal, and they have opposite failure modes, so they are handled differently. **Gauges** (queued, running, queue limit, env queued, env running, env limit, throttled, plus keys-with-backlog and worst-key wait on keyed queues) are read *inside* the same Redis script that performs the enqueue or dequeue, so the reading is atomic with the operation it describes rather than a racy follow-up read. The script returns them on its reply and the app forwards them to the stream. Gauges are sampled and drop-tolerant: they are aggregated with `max`, so a lost reading costs resolution, never correctness. **Counters** (enqueued, started, completed, plus nack and dead-lettered) are cumulative odometers. Each event increments a per-queue key on the metrics Redis and emits the absolute total, and ClickHouse takes the difference across buckets at read time. This is the important property of the design: a summed-delta counter undercounts permanently on any lost event, while a cumulative one self-heals, because the next surviving reading restates the whole total. Only bucket granularity can be lost, never the total. A queue returning after its odometer TTL expired restarts at 1 and reset detection handles it, which is safe precisely because expiry only spans a window with no activity. Both land on one sharded Redis stream. A consumer reads it with a consumer group, reclaims stale pending entries on a 15s interval rather than on every poll, maps one entry to one or two ClickHouse rows (whole-queue and, for keyed queues, per-key), and acks only after the insert lands. Each batch carries a dedup token derived from its stream-entry ids, and the target tables set `non_replicated_deduplication_window`, so a retried batch cannot double-count either the raw rows or the aggregates that hang off them. Consumer and emitter both emit OTel metrics (`queue_metrics.emitter.emitted`, `queue_metrics.consumer.{entries,rows_inserted,insert_errors,insert_duration,stream_depth,group_lag,pending,lag_unknown}`); stream depth and group lag are the two worth alerting on, and `lag_unknown` exists because Redis can report a null lag after a trim, which must not be read as zero. ## Storage and read path `queue_metrics_raw_v1` is a short landing table with a 6 hour TTL. Four aggregate tiers are materialized straight from raw, never cascaded off each other, each with a 30 day TTL: - `queue_metrics_v1`, 10 second buckets per queue, the default read path - `queue_metrics_5m_v1`, 5 minute buckets per queue, for wide ranges and cross-queue ranking - `env_metrics_v1`, 10 second buckets per environment, queue-independent so it stays cheap at any range - `queue_metrics_ck_v1`, 10 second buckets per concurrency key Every tier is an MV from raw because the counter states do not survive a cascade: their merge is order sensitive, so a `-MergeState` chain off the 10s table inflates the result, and the same property means an aggregate state may only be merged inside one queue. That constraint is now enforced by the query engine rather than by reviewer discipline: a column can declare a `mergeGroupKey`, and any query that references it without grouping by, or pinning to a single value of, every named key fails to compile with an actionable message. On the read side, TRQL gains three tables (`queue_metrics`, `env_metrics`, and a `queue_metrics_by_key` that is hidden from the editor, schema docs and schema API but still queryable, so per-key rows can never silently merge into a plain per-queue query), plus `deltaSumTimestampMerge` and `quantilesTDigestMerge`. Two schema-level optimizations ride along: a table can declare coarser rollups, so a query whose bucket interval is 5 minutes or wider is routed to the 5m table with no change to the query itself, and it can opt into the ClickHouse query cache with time bounds floored to a fixed grid, so the auto-refreshing dashboards actually share cache entries instead of missing on every tick. Both are caller-side substitutions, so the printer stays unaware of physical layout. All of this can also live on its own ClickHouse service. A table declares the pool its reads run on, the three queue-metrics tables name the dedicated one, and the ingestion consumer writes through the same client, so both directions move together with one env var and nothing else routes differently. The other engine change is opt-in gap filling: charts can request rows for empty buckets, where counters zero-fill and gauges carry forward. Grouped gauge series are densified per group and carried inside a partition, so a quiet queue's line holds its last value without bleeding another queue's value into it. ## Queue concurrency limits `concurrencyLimitOverridePercent` on `TaskQueue` is the source of truth when an override is set as a percentage; the absolute `concurrencyLimit` is materialized from it (floored, clamped to at least 1 so a percentage can never act as a pause, and never above the environment limit). Every path that changes an environment limit now recalculates the environment's percent-based overrides afterwards, outside the transaction, and pushes changed limits to the engine. The push is attempted even when the stored value did not change, so a previously failed sync self-heals rather than leaving the database and the engine diverged; paused queues are skipped so a recalculation cannot effectively unpause one. The API accepts exactly one of `concurrencyLimit` or `percent`, and the reject-instead-of-clamp change above means a request asking for more than the environment allows now fails loudly. The percent bound (greater than 0, at most 100) is defined once and shared by the zod schema, the dashboard mutation handler and the service, so the three cannot drift. The concurrency-keys table on a queue is now paginated against the ClickHouse per-key tier, ranked by peak backlog with the total on every row from a single scan, and only the keys on the current page are enriched with live counts from Redis. That replaces a hard top-50 cap with something whose cost is a function of page size rather than key cardinality. ## The health report `GET /api/v1/reports/:key?period=&format=markdown|ansi|json`. The verdict is computed on the server and is deterministic, not model-generated. Three independent analyzers run over one input snapshot: flow (is work moving, and if not, is the cause a limit, throttling, one bad queue, or dead-lettering), execution (are the runs that start succeeding, and at what latency), and liveness (how fresh is the telemetry). When telemetry is genuinely stale, the first two are forced to unknown and every actionable field is stripped, so no surface ever advises action off stale data. Authorization is per query table rather than a blanket query grant: a JWT must be scoped to every table the report reads (`runs`, `env_metrics`, `queue_metrics`), so a narrowly scoped token cannot pull a report that reads more than it was granted. `period` is validated as a shorthand with a 90 day ceiling at the edge. The report catalog is a registry of `{ load, interpret }` entries, so the next report is a new entry and no change to the route, the view model, the renderers, the CLI or the MCP tool. `trigger mcp` no longer launches the install wizard when stdout is a TTY, which fixed a real failure: hosts spawn the server over a PTY, so the wizard would open and the client would time out waiting for a server that never started. The wizard now needs `trigger mcp --install`. ## The part that is live regardless of every flag The enqueue and dequeue scripts now return a 2-tuple so a gauge reading can ride back on the reply. Every return site in the eight affected scripts is wrapped, and a `nil` original is converted to `false` on the way out, because a raw `nil` in the first slot would make Lua truncate the multi-bulk reply and silently drop the gauge on the throttled and empty-queue paths. The reply shape and the destructuring on the app side are exercised on every queue operation whether or not metrics are enabled, so that is the part of `run-engine` worth the closest review. One behavior fix in the same area: the scheduling-delay anchor is set only on a run's first entry into the queue. Anchoring it to trigger time on re-enqueues made waitpoint and checkpoint resumes report the entire wait as scheduling delay. Queue ordering is untouched, so a re-enqueued run keeps its position, and nacks deliberately keep the original anchor because a rolled-back dequeue is the same continuous wait. A pending-version promotion still anchors to trigger time, on purpose: that promotion is the run's first real entry into the queue, since the trigger deliberately held it back waiting for a worker version, and the TTL is armed at the same point for the same reason. The consequence is worth naming, because it is a judgement call: a run that waits on a deployment reports that wait as scheduling delay on its queue, which is time unrelated to queue capacity. ## Verification Unit and integration suites across the new package, the run queue, the mapping layer, the query engine and ClickHouse (including a test that applies migration 036 through the same splitter CI uses, and a regression test that inserts the same batch three times to prove the aggregates do not inflate). Beyond that, the whole path was driven end to end against a live stack with real runs: emitter to Redis stream to consumer to ClickHouse to the dashboards, for both the local dev path and the deployed path where a supervisor drives the dequeue, with assertions on exact counter reconstruction per queue and per concurrency key, throttling, environment saturation, scheduling delay, and a deliberate mid-stream reading drop to confirm the cumulative counters still reconstruct the correct total. The gated-off state was checked on every touched surface. The dedicated ClickHouse service was verified against a second, separately-schema'd instance: with it configured, the driven counters reconstruct exactly on the dedicated instance, the shared instance gains no rows for that window, a read through the query API returns the value that exists only on the dedicated instance, and a `runs` query still succeeds (it would fail outright if it were mis-routed to a service without that table). With the variable unset, the full suite passes unchanged. --------- Co-authored-by: Katia Bulatova <katia@trigger.dev> Co-authored-by: Katia Bulatova <katherine.bulatova@gmail.com> Co-authored-by: James Ritchie <james@trigger.dev>
1 parent 639eaf6 commit 4eb9292

152 files changed

Lines changed: 18359 additions & 1124 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.changeset/mcp-server-starts-without-wizard.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
"trigger.dev": patch
3+
---
4+
5+
`trigger mcp` now always starts the MCP server, and the interactive install wizard has moved behind `trigger mcp --install`. Previously the wizard opened whenever stdout was a terminal, so any MCP host that spawns the command over a pseudo-terminal waited on a server that never started and eventually timed out.

‎.changeset/report-health.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
---
2+
"@trigger.dev/core": patch
3+
"trigger.dev": patch
4+
---
5+
6+
Ask whether an environment is healthy and get an answer instead of a wall of charts. `trigger report health` returns a verdict on three questions: is work flowing, are the runs that start succeeding, and is the telemetry fresh enough to trust either answer. When something looks wrong it names the most likely cause and a next action.
7+
8+
```bash
9+
npx trigger.dev@latest report health --env prod --period 24h
10+
```
11+
12+
The verdict is computed server side, so the CLI, the new `get_report` MCP tool, and `GET /api/v1/reports/health` all return the same text with the same sparklines. In MCP hosts that support prompts, `report` is also available as a slash command.

‎.server-changes/agent-detail-metrics-layout.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
area: webapp
3+
type: improvement
4+
---
5+
6+
The agent detail page now uses the same layout as the other metrics pages: filters stay pinned at the top, the activity charts sit in a row beneath them, and the tabs and table scroll with the page, with the agent config in a resizable panel on the right.

‎.server-changes/paginate-concurrency-keys-table.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
area: webapp
3+
type: improvement
4+
---
5+
6+
A queue's concurrency keys are now paged and searchable, so a queue with thousands of keys shows all of them instead of only the busiest 50.

‎.server-changes/queue-concurrency-percent-limits.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
area: webapp
3+
type: feature
4+
---
5+
6+
A queue's concurrency limit can now be set as a percentage of your environment limit, from the dashboard or the API. Percentage limits track the environment limit, so raising or lowering it re-divides capacity without you editing every queue.

‎.server-changes/queue-metrics-dashboard.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
area: webapp
3+
type: feature
4+
---
5+
6+
See how every queue in an environment is doing: backlog, throughput, concurrency against its limit, when it was throttled, and how long runs waited before starting. Each queue also gets its own page, which breaks the same numbers down per concurrency key so you can tell whether one key is starving the rest.

‎.server-changes/reject-queue-limit-above-env-limit.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
area: webapp
3+
type: breaking
4+
---
5+
6+
Setting a queue concurrency limit higher than your environment limit is now rejected with an error instead of being silently reduced to the environment limit, so a queue never looks like it has more capacity than it can get. Limits already saved are unchanged.

0 commit comments

Comments
 (0)