Visitar URL original
feat(processor): report dest_transformer diff rows for multiplexing destination transformations by mihir20 · Pull Request #7446 · rudderlabs/rudder-server · GitHub
Skip to content

feat(processor): report dest_transformer diff rows for multiplexing destination transformations - #7446

Merged
mihir20 merged 1 commit into
masterfrom
pipeline-inspector-gaps/04-dest-transformer-diff-metrics
Oct 8, 2026
Merged

mihir20 merged 1 commit into
masterfrom
pipeline-inspector-gaps/04-dest-transformer-diff-metrics

Conversation

@mihir20

@mihir20 mihir20 commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Description

Destination transformation can multiplex, returning more or fewer events than it was given, so succeeded + filtered + aborted for the dest_transformer stage does not equal its input. #6485 removed the stage's diff rows on DB-load grounds, reasoning that the explicit rows make them redundant; that does not hold under multiplexing. This brings them back behind Reporting.destTransformerDiffMetrics.enabled (default false), computed with the existing getDiffMetrics helper from the event-filter output counts.

The diff rows carry in_pu='' while the sibling dest_transformer rows carry in_pu='event_filter'; a reconciliation that filters the stage by in_pu will not see them. The succeeded rows are unchanged: the count maps they now populate feed only the diff.

Linear Ticket

PIPE-3454

Security

  • The code changed/added as part of this pull request won't create any security issues with how the software is being used.

@codecov

codecov Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.00%. Comparing base (f424257) to head (9d65563).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7446      +/-   ##
==========================================
+ Coverage   84.50%   85.00%   +0.50%     
==========================================
  Files         623      623              
  Lines       62050    62086      +36     
==========================================
+ Hits        52434    52776     +342     
+ Misses       9613     9307     -306     
  Partials        3        3              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@atzoum atzoum left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the description of a PR is longer than the change itself, then something is wrong with the description!
#7435 (review)

Comment thread processor/processor.go
// inCountMap and inCountMetadataMap are the event-filter output counts (the dest_transformer
// stage's input counts), threaded through so destTransform can compute DEST_TRANSFORMER diff
// metrics without recomputing them.
inCountMap map[string]int64

@atzoum atzoum Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These count maps are using a string which doesn't provide much type information, when they are actually keyed by sourceID, destinationID, sourceJobRunID, eventName, eventType.
A comparable struct key would would be much more appropriate, we could drop the strings.Join allocation for every event, and remove the split:

type metricCountKey struct {
	sourceID, destinationID, sourceJobRunID, eventName, eventType string
}

inCountMap and inCountMetadataMap always share the same key, so they could become a single map[metricCountKey]*metricCount{count int64; metadata MetricMetadata}. Failed and filtered have no metadata map, so they can stay map[metricCountKey]int64. Can you please create a followup ticket for this?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, a comparable key is the right shape. Tracked as PIPE-3466 (https://linear.app/rudderstack/issue/PIPE-3466): introduce metricCountKey{sourceID, destinationID, sourceJobRunID, eventName, eventType}, merge inCountMap and inCountMetadataMap into one map[metricCountKey]*metricCount, keep failed and filtered as map[metricCountKey]int64, and drop the strings.Join / strings.Split on the hot path.

Keeping it out of this PR on purpose: the key is built and split in the existing code and flows through updateMetricMaps, getTransformationMetrics, getNonSuccessfulMetrics and getDiffMetrics, so the change touches every call site including the user_transformer one. Planned as a standalone PR against master once this set has merged.

@mihir20
mihir20 force-pushed the pipeline-inspector-gaps/03-dest-transformer-passthrough branch from d068aef to 5c39494 Compare October 5, 2026 07:40
@mihir20
mihir20 force-pushed the pipeline-inspector-gaps/04-dest-transformer-diff-metrics branch from 27d6efe to d6d0f33 Compare October 5, 2026 07:40
@mihir20
mihir20 force-pushed the pipeline-inspector-gaps/03-dest-transformer-passthrough branch from 5c39494 to 8759288 Compare October 5, 2026 09:59
@mihir20
mihir20 force-pushed the pipeline-inspector-gaps/04-dest-transformer-diff-metrics branch from d6d0f33 to f8ed75b Compare October 5, 2026 09:59
@mihir20
mihir20 requested a review from atzoum October 5, 2026 10:57
@mihir20
mihir20 force-pushed the pipeline-inspector-gaps/03-dest-transformer-passthrough branch from 8759288 to 43e15a9 Compare October 6, 2026 06:45
@mihir20
mihir20 force-pushed the pipeline-inspector-gaps/04-dest-transformer-diff-metrics branch from f8ed75b to 4417ca9 Compare October 6, 2026 06:45
@mihir20
mihir20 force-pushed the pipeline-inspector-gaps/03-dest-transformer-passthrough branch from 43e15a9 to e4d2415 Compare October 6, 2026 07:15
@mihir20
mihir20 force-pushed the pipeline-inspector-gaps/04-dest-transformer-diff-metrics branch from 4417ca9 to 10c26ba Compare October 6, 2026 07:15

@atzoum atzoum left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's get rid of inPus altogether, have them empty everywhere. They are too much of a trouble

@mihir20
mihir20 force-pushed the pipeline-inspector-gaps/03-dest-transformer-passthrough branch from e4d2415 to 1ab6710 Compare October 6, 2026 09:55
@mihir20
mihir20 force-pushed the pipeline-inspector-gaps/04-dest-transformer-diff-metrics branch from 10c26ba to b4f31c7 Compare October 6, 2026 09:55
@ktgowtham
ktgowtham requested a review from itsmihir October 6, 2026 11:08
@mihir20
mihir20 force-pushed the pipeline-inspector-gaps/03-dest-transformer-passthrough branch from 1ab6710 to 2b40828 Compare October 7, 2026 09:14
@mihir20
mihir20 force-pushed the pipeline-inspector-gaps/04-dest-transformer-diff-metrics branch from b4f31c7 to 88be095 Compare October 7, 2026 09:14
@mihir20
mihir20 force-pushed the pipeline-inspector-gaps/03-dest-transformer-passthrough branch from 2b40828 to 2b4b9d2 Compare October 7, 2026 09:39
@mihir20
mihir20 force-pushed the pipeline-inspector-gaps/04-dest-transformer-diff-metrics branch from 88be095 to 8e14d5e Compare October 7, 2026 09:39
@mihir20
mihir20 force-pushed the pipeline-inspector-gaps/03-dest-transformer-passthrough branch from 2b4b9d2 to a26b6bc Compare October 7, 2026 11:21
@mihir20
mihir20 force-pushed the pipeline-inspector-gaps/04-dest-transformer-diff-metrics branch from 8e14d5e to 64c39af Compare October 7, 2026 11:21
@mihir20
mihir20 force-pushed the pipeline-inspector-gaps/03-dest-transformer-passthrough branch from a26b6bc to 6026d67 Compare October 7, 2026 12:12
@mihir20
mihir20 force-pushed the pipeline-inspector-gaps/04-dest-transformer-diff-metrics branch from 64c39af to ad5275d Compare October 7, 2026 12:12
Base automatically changed from pipeline-inspector-gaps/03-dest-transformer-passthrough to master October 7, 2026 16:14
…estination transformations

Destination transformation can return more or fewer events than it was
given, so succeeded + filtered + aborted for the dest_transformer stage does
not equal its input and the pipeline funnel cannot reconcile the node. Behind
the new reloadable flag Reporting.destTransformerDiffMetrics.enabled (default
false) the processor now computes a diff row per source, destination, event
name and event type with the existing getDiffMetrics helper, using the
event-filter output counts as the stage input and the real transformer
outcomes as the stage output. The row carries an empty inPU.

The succeeded rows are unchanged: the count maps they now populate feed only
the diff computation. The user_transformer diff call site, event_filter,
destination_filter, the transformAt none pass-through path and router
transform handling are untouched.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

🔒 Scanned for secrets using gitleaks 8.28.0
@mihir20
mihir20 force-pushed the pipeline-inspector-gaps/04-dest-transformer-diff-metrics branch from ad5275d to 9d65563 Compare October 7, 2026 16:23
@mihir20

mihir20 commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

Done as its own PR on top of this one, #7458: CreatePUDetails loses the inPU parameter so no producer can set one, and every processor, router, batch router and warehouse row now carries an empty in_reported_by. The column stays on the wire and in the reports table; nothing reads it.

@mihir20
mihir20 requested a review from atzoum October 8, 2026 08:26
@mihir20
mihir20 enabled auto-merge (squash) October 8, 2026 13:51
@mihir20
mihir20 disabled auto-merge October 8, 2026 13:52
@mihir20
mihir20 merged commit cd95c32 into master Oct 8, 2026
127 of 130 checks passed
@mihir20
mihir20 deleted the pipeline-inspector-gaps/04-dest-transformer-diff-metrics branch October 8, 2026 14:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants