Repository navigation
feat(metrics): measure sink CPU usage - #27097
Draft
gwenaskell wants to merge 5 commits into
Draft
gwenaskell wants to merge 5 commits into
gwenaskell wants to merge 5 commits into
Conversation
gwenaskell
force-pushed
the
yoenn.burban/sink-cpu-usage
branch
from
November 7, 2026 16:04
0c60f93 to
4fea25c
Compare
The CPU counter of the component task being polled is kept in a thread-local, and spawn_timed reads it at spawn time instead of taking it as an argument. Transforms build() runs under a non-measuring scope so that tasks spawned at build time inherit the counter. Also document that CPU usage is a rough estimate on Windows.
gwenaskell
force-pushed
the
yoenn.burban/sink-cpu-usage
branch
from
November 9, 2026 16:30
b992d72 to
d62ed9e
Compare
gwenaskell
changed the base branch from
master
to
yoenn.burban/cpu-timer-inheritance
November 9, 2026 16:30
Co-authored-by: Eva Parish <eva.parish@datadoghq.com>
1 of 2 tasks
Add `measure_cpu_usage` to sinks. Sink subtasks (request building, hyper connections, tower buffer worker, per-sink background tasks) are spawned with the CPU-timed spawn helpers so that they are attributed to the sink. Hyper connection tasks are spawned through `vector_http::CpuTimedExecutor`.
gwenaskell
force-pushed
the
yoenn.burban/sink-cpu-usage
branch
from
November 12, 2026 09:37
d62ed9e to
77b3478
Compare
Base automatically changed from
yoenn.burban/cpu-timer-inheritance
to
master
November 13, 2026 10:44
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extends the per-component
component_cpu_usage_ns_totalmetric (#25317) to sinks, including the tasks they spawn.Stacked on #27104 (counter inheritance through a thread-local,
spawn_timed/spawn_timed_without_span): review that one first.Most sink CPU does not run in the sink task: encoding and compression happen in
ConcurrentMaptasks (request_builder), and TLS encryption and socket writes in hyper's per-connection tasks. Timing only the sink task would miss most of it.measure_cpu_usageoption; the sink task is wrapped withcpu_timedand itsbuild()runs undercpu_scoped.ConcurrentMap(request_builder/concurrent_map)vector_http::CpuTimedExecutoron theHttpClientpool and thevectorsink'sHttp2Clientcrate::http::serve(used byprometheus_exporter; sources have no counter, so they are unaffected)BatchSinkservice callsBufferworker of distributed services (Buffer::pairinstead ofBufferLayer)Windows accuracy: the
measure_cpu_usagedoc of sinks carries the same Windows caveat as transforms (see #27104).Not covered
kafkasink compression and network I/O)spawn_blocking: the closure runs on a blocking-pool thread, outside any timed poll. Our own call sites (dns.rs, kafka healthcheck) could be wrapped in a follow-up, but two indirect uses cannot be covered, because thespawn_blockingcall happens inside the dependency with no hook:tokio::fswrites in thefilesink (the write syscall and buffer copy; compression stays in the sink task and is counted)References
Related: #25317
Depends on: #27104
Vector configuration
How did you test this PR?
component_cpu_usage_emitted_sink: the mock completion sink burns CPU in aspawn_timedsubtask; the existing no-measure test also checks the sink.cargo testoncpu_metrics,topology::test,sinks::util,http,vector-httpand the touched sinks/transforms; clippy on the touched feature set.Does this PR include user facing changes?
no-changeloglabel to this PR.Contributor Guidelines
@vectordotdev/vectorto reach out to us regarding this PR.git merge origin masterandgit push.