Visitar URL original
[mysql] Bound how long SHOW BINARY LOGS can block commits by ggilder · Pull Request #25991 · DataDog/integrations-core · GitHub
Skip to content

[mysql] Bound how long SHOW BINARY LOGS can block commits - #25991

Open
ggilder wants to merge 6 commits into
DataDog:masterfrom
ggilder:mysql-binlog-net-write-timeout
Open

ggilder wants to merge 6 commits into
DataDog:masterfrom
ggilder:mysql-binlog-net-write-timeout

Conversation

@ggilder

@ggilder ggilder commented Nov 10, 2026 •

Copy link
Copy Markdown

What does this PR do?

Runs the SHOW BINARY LOGS query used for mysql.binlog.disk_use with a session net_write_timeout of 2s, as the last statement of the check run.

If the server hits the timeout, it drops the connection. The check logs a warning and skips the binary log metric for that run. Each run opens a new connection, and nothing else runs on it after this query, so the rest of the run is not affected. If the query ran earlier, the dropped connection would fail every later query in the run, lose the run's status metrics, and report mysql.can_connect as CRITICAL.

A dedicated connection would also avoid that, but it needs an extra slot beyond the documented MAX_USER_CONNECTIONS for Database Monitoring.

I checked the timeout behavior against the MySQL 8.0.36 test environment. With net_write_timeout = 1 and a client that stops reading a large result, the server dropped the connection while the client was not reading, and the client got error 2013 (CR_SERVER_LOST), which is the case the warning handles.

Motivation

SHOW BINARY LOGS holds the binary log index lock (LOCK_index) while it sends one row per binary log file (show_binlogs() in sql/rpl_source.cc). A second SHOW BINARY LOGS, or a binary log rotation, takes the binary log lock (LOCK_log) and then waits for LOCK_index. Every commit needs LOCK_log, so all commits on the server stall until the first statement finishes sending.

The send only finishes as fast as the agent reads. We saw this on an Aurora MySQL 3 writer with long binary log retention and tens of thousands of binary log files. The cluster check runner was CPU-starved, so it read the result slowly. The writer had repeated server-wide commit freezes of up to about 20 seconds. Performance Insights showed one SHOW BINARY LOGS from the datadog user holding LOCK_index while a second one held LOCK_log. We worked around it by setting binlog_size_metrics: false.

The server default net_write_timeout is 60s, so it did not bound these stalls. A low per-query value caps how long a stalled reader can block one write, and hitting it costs one missed binary log metric point. The limit applies to each blocked write, not to the whole statement, so a reader that is slow but still making progress can hold the lock for longer in total.

The value is a fixed constant rather than a config option, to keep the change small. I am happy to make it configurable if you prefer.

Checklist before requesting review

  • I self-reviewed the full diff, and this description matches the final change
  • Tests cover the change, or this description explains why none are needed
  • Automated review comments, including Codex review, were addressed or answered
  • I, the human author, understand this change, can explain it to reviewers, and believe it meets the repository standards

SHOW BINARY LOGS holds the binary log index lock while it sends one row
per file. A second SHOW BINARY LOGS or a log rotation waits for that lock
while holding the binary log lock, which every commit needs. When the
agent reads the result slowly, commits on the server stall until
net_write_timeout (60s by default) fires.

Run the query with a 2s session net_write_timeout and restore the
previous value afterward.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dd-octo-sts

dd-octo-sts Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

✅ Dispatcher tests: passed

Dispatcher beta: informational only
Existing CI remains the merge signal.

  30/30 jobs

✅ 30 passed · nothing failed

Batches · ✅ batch-01 30/30

Dispatcher finished on cb282ce — GitHub Run · Dispatcher Logs.

@datadog-official

datadog-official Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Pipelines  Code Coverage

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 92.12% (+2.35%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: a20138f | Docs | Give us feedback!

ggilder and others added 2 commits November 12, 2026 09:07
When net_write_timeout fires, the server drops the connection. On the
shared check connection, that made later queries in the same run fail,
dropped the run's status metrics, and reported mysql.can_connect as
CRITICAL. A dedicated connection limits a timeout to the binary log
metric.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Assert the dedicated connection gets the instance connection settings,
assert which warning each error logs, and add a test that a timeout does
not touch the main check connection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ggilder
ggilder marked this pull request as ready for review November 12, 2026 18:19
@ggilder
ggilder requested review from a team as code owners November 12, 2026 18:19
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-12T18:41:03.851302Z b2dd116 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@datadog-official datadog-official Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

The dedicated binary-log connection preserves instance settings and isolates handled query failures from the main check connection. The two-second timeout bounds individual blocked writes rather than total query duration.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 436ca23

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 436ca23b85

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread mysql/datadog_checks/mysql/mysql.py Resolved
Comment thread mysql/tests/test_binlog_metrics.py Resolved
Feedback from @chatgpt-codex-connector:
- A dedicated connection needs a sixth slot, over the documented MAX_USER_CONNECTIONS 5 for DBM. Run the query on the main connection as the last statement of the check run instead, so a timeout only drops the binary log metric.
- Add type hints to the new test helpers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ggilder

ggilder commented Nov 12, 2026

Copy link
Copy Markdown
Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep them coming!

Reviewed commit: b2dd1163ef

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant