Visitar URL original
Improve span link encoding performance by PerfectSlayer · Pull Request #12768 · DataDog/dd-trace-java · GitHub
Skip to content

Improve span link encoding performance - #12768

Open
PerfectSlayer wants to merge 6 commits into
masterfrom
bbujon/span-links
Open

PerfectSlayer wants to merge 6 commits into
masterfrom
bbujon/span-links

Conversation

@PerfectSlayer

@PerfectSlayer PerfectSlayer commented Oct 7, 2026 •

Copy link
Copy Markdown
Collaborator

What Does This Do

This PR addresses my feedback from #11855:

  • a single event collection per span instead of one in OtelSpan and one in DDSpan;
  • no v0 encoding logic in the OTel classes;
  • no core implementation details in internal-api.

About implementation details:

  • Stores span events as DDSpanEvent objects on DDSpan, created in core through two new AgentSpan methods: addEvent(name, attributes) and addEvent(name, attributes, timestamp, unit). Previously the OTel shim JSON-encoded them into the events tag when the span finished. internal-api no longer has an event type.
  • Encodes the events tag in core (DDSpanContext) at serialization time, next to the span links tag. This covers the tag-based protocols and writers: v0.4/v0.5, OTLP, CI Visibility, LLM Obs and file-based.
  • Events are appended to a list under the span lock, so each add is cheap even for spans with many events. The list is frozen when the span finishes: later events are ignored, as the OTel shim already did after end(), and serialization uses the list without copying it. A copy is only taken when a span is serialized while still running, such as long-running spans.
  • Writes structured events directly in v1 payloads (span field 12) so TraceMapperV1 no longer parses JSON.
  • The v0.x tag keeps the previous StringBuilder encoding and escaping rules. It now uses one builder per tag instead of one per event and per attribute set. I didn't use datadog.json.JsonWriter: on JDK 17 and earlier its OutputStreamWriter allocates an 8 KB buffer per call, and it encodes one character at a time, which is costly for exception stack traces.
  • Events without an explicit time are now timestamped with the tracer clock, the same one used for span start and end. The OTel shim previously used a static SystemTimeSource, which has only millisecond precision.
  • All JSON encoding code is removed from the OTel shim.

Motivation

Events were JSON-encoded on the application thread for every protocol. v1 then parsed that JSON back to write native events: encode → decode → re-encode.
This also moves work off the application thread: attributes are copied into a map instead of being JSON-encoded when an event is added, and no tag is built when the span finishes.

Additional Notes

Intended behaviour changes:

  • The events tag is only added when the span is serialized, so trace interceptors no longer see it. This already applies to _dd.span_links.

Fixes:

  • Event names are now escaped in the events tag. A name containing ", \ or a control character used to produce invalid JSON, or to be read as a different name. Names without those characters are encoded exactly as before.
  • Span events recorded without a timestamp could get a time up to 1 ms before the span start, because they didn't use the span's clock. They now use the tracer clock.

Existing known limitations, to address separately:

  • There is still no limit on the number of events per span, as before. The OTel SDK caps them at 128 by default.

Contributor Checklist

Jira ticket: [PROJ-IDENT]

🤖 Generated with Claude Code

This would avoid encoding span tags as JSON, to then decode them as Java objects in order then to encode them as msgstruct
@PerfectSlayer PerfectSlayer added comp: core Tracer core tag: performance Performance related changes type: refactoring inst: opentelemetry OpenTelemetry instrumentation tag: ai generated Largely based on code generated by an AI or LLM and removed inst: opentelemetry OpenTelemetry instrumentation labels Oct 7, 2026
@datadog-prod-us1-5

This comment has been minimized.

@dd-octo-sts

dd-octo-sts Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 13.98 s 14.03 s [-1.2%; +0.5%] (no difference)
startup:insecure-bank:tracing:Agent 13.00 s 13.01 s [-0.7%; +0.5%] (no difference)
startup:petclinic:appsec:Agent 17.24 s 17.01 s [+0.4%; +2.3%] (maybe worse)
startup:petclinic:iast:Agent 17.01 s 16.95 s [-0.5%; +1.2%] (no difference)
startup:petclinic:profiling:Agent 16.65 s 17.03 s [-3.6%; -0.8%] (maybe better)
startup:petclinic:sca:Agent 17.10 s 16.53 s [-1.1%; +8.0%] (no difference)
startup:petclinic:tracing:Agent 16.25 s 16.17 s [-0.5%; +1.4%] (no difference)

Commit: 1efcf72d · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

No longer use hardcoded system time source but the tracer one.
@PerfectSlayer

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-08T12:39:13.254312Z 1efcf72 Draft marked ready
🔒 Security Review ✅ Completed 2026-10-08T12:39:58.178815Z 1efcf72 Draft marked ready

Security findings

Advisory findings (1)

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛡️ Codex Security Review · Automatically triggered

Here are some automated security review suggestions for this pull request.

Reviewed commit: 8f00694578

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

Comment thread dd-trace-core/src/main/java/datadog/trace/core/DDSpanContext.java

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8f00694578

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread dd-trace-core/src/main/java/datadog/trace/core/DDSpan.java Outdated
Comment thread dd-trace-core/src/main/java/datadog/trace/core/DDSpan.java Outdated
Comment thread dd-trace-core/src/main/java/datadog/trace/core/DDSpan.java Outdated
@PerfectSlayer

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 1efcf72d2e

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1efcf72d2e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread dd-trace-core/src/main/java/datadog/trace/core/DDSpan.java
Comment thread dd-trace-core/src/main/java/datadog/trace/core/DDSpanContext.java
@PerfectSlayer
PerfectSlayer marked this pull request as ready for review October 8, 2026 12:34
@PerfectSlayer
PerfectSlayer requested review from a team as code owners October 8, 2026 12:34
@PerfectSlayer
PerfectSlayer requested review from jordan-wong and mcculls and removed request for a team October 8, 2026 12:34
@PerfectSlayer
PerfectSlayer requested review from amarziali and removed request for a team October 8, 2026 12:34
@PerfectSlayer PerfectSlayer changed the title Keep span events structured until serialization Improve span link encoding performance Oct 8, 2026

@datadog-prod-us1-5 datadog-prod-us1-5 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

LoggingWriter and PrintingWriter lose recorded span events. When a tag-based writer serializes a span before v1, the retained events tag also duplicates the native event payload.

Open Bits AI session

🤖 Bits Code Review · Commit 1efcf72 · @DataDog review to ask questions

Comment thread dd-trace-core/src/main/java/datadog/trace/core/DDSpanContext.java
@PerfectSlayer
PerfectSlayer added this pull request to stack #12809 October 9, 2026 08:51
* @param attributes The OpenTelemetry attributes to convert.
* @return The attributes keyed by attribute name.
*/
static Map<String, ?> eventAttributes(Attributes attributes) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it worth moving this to OtelConventions and just dropping this file?

public List<? extends AgentSpanLink> getLinks() {
return this.links;
List<AgentSpanLink> links = this.links;
return links.isEmpty() ? links : unmodifiableList(links);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could use a similar links == NO_LINK trick as getEvents

boolean writeTopLevel = meta.topLevel();
int tagCount = 0;
for (TagMap.EntryReader entry : tags) {
if (!DDTags.SPAN_EVENTS.equals(entry.tag())) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider keeping this for the situation where there are multiple writers configured - a preceding writer in the chain could serialize the events into the SPAN_EVENTS tag, which would then get written out here despite the same data being written out separately.

@mcculls mcculls left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good improvement, just some minor suggestions

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: core Tracer core tag: ai generated Largely based on code generated by an AI or LLM tag: performance Performance related changes type: refactoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants