Visitar URL original
Add session-backed Postgres remote query execution by nubtron · Pull Request #26011 · DataDog/integrations-core · GitHub
Skip to content

Add session-backed Postgres remote query execution - #26011

Draft
nubtron wants to merge 3 commits into
masterfrom
nubtron/remote-queries-postgres-spec-20261112
Draft

Add session-backed Postgres remote query execution#26011
nubtron wants to merge 3 commits into
masterfrom
nubtron/remote-queries-postgres-spec-20261112

Conversation

@nubtron

@nubtron nubtron commented Nov 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

What does this PR do?

Adds session-backed remote queries to a loaded Postgres check: resolve/execute/cancel methods, isolated read-only connections, describe-before-COPY, typed descriptors, bounded CSV framing/pages, sequential intake uploads, retries/splitting, finalize/abort, tracing, and remote_queries.timeout_ms.

Reuses loaded connection settings, database identifiers, autodiscovery snapshots, and the framework's proxy bypass matching. No new dependency or base-check changes.

Local verification (all commands prefixed with HATCH_DATA_DIR=/instance_storage/remote-queries-spec-ai/caches/hatch):

ddev --no-interactive test postgres:py3.13-17.0-UTF8 -- tests/test_remote_queries_unit.py tests/test_remote_queries_integration.py
ddev --no-interactive test postgres:py3.13-17.0-UTF8 -- tests/test_unit.py tests/test_discovery.py -k 'cancel or database_identifier or autodiscovery_simple or autodiscovery_max_databases or autodiscovery_refresh'
ddev --no-interactive test --lint postgres
ddev validate config -s postgres
ddev validate models -s postgres

Results: 109 passed; 1 opt-in median-of-five quality test skipped. Existing tests: 10 passed, 59 deselected. Lint passes; 2 configs valid; 5 model files synchronized. Quality thresholds and deployed-Agent behavior are not claimed.

Provisional choices: 503 unavailable is the capacity class; other retryable failures have five attempts. Cancellation by run ID cancels duplicate-ID calls together. Legacy libpq cancellation waits at most 5 s on a dedicated daemon; an abandoned cancel thread can outlive the run. Dedicated event loops do not wait for abandoned DNS resolver threads. Embedded Agent libpq is 18.3; local tests use 16.15.

Blocking / unverified: this is scoped WIP, not a complete-spec or production-readiness claim. Libpq can receive an entire oversized COPY block before the framer rejects it, so strict early-read/memory bounds remain unresolved. AWS/Azure token fetching remains synchronous; deadline/cancel and independence on that path are unverified. Idle-machine quality measurements, deployed verification, and paired Agent/AP contract alignment remain outstanding.

Motivation

Fresh implementation of the session-backed Postgres specification, pinned at planning commit 1e3384369dc05ca2bec6e4eb7cd82d8ef0fe0b0e. Paired with the core-Agent change: DataDog/datadog-agent#60311.

Checklist before requesting review

  • I self-reviewed the full diff, and this description matches the final change
  • Tests cover the change, or this description explains why none are needed
  • Automated review comments, including Codex review, were addressed or answered
  • I, the human author, understand this change, can explain it to reviewers, and believe it meets the repository standards

@dd-octo-sts

dd-octo-sts Bot commented Nov 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

❌ Dispatcher tests: failed

Dispatcher beta: informational only
Existing CI remains the merge signal.

Caution

1 integration failed. 38 of 38 jobs failed.

  38/38 jobs

❌ 38 failed

Batches · ❌ batch-01 38/38

❌ postgres: 38 failed targets

Tests failed in every target:

  • postgres.tests.test_config_defaults::test_all_config_defaults

Dispatcher finished on 702eaa6 — GitHub Run · Dispatcher Logs.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Nov 12, 2026 •
edited
Loading

Copy link
Copy Markdown

evalya-impact-summary

Evalya CI impact

Impact analysis: 0 selected, 0 skipped (of 0 test tasks)
Publish tasks:   4 (always emitted)
Changed files:   14

Full impact report
Learn more about CI impact filtering

@datadog-official

datadog-official Bot commented Nov 12, 2026 •
edited by datadog-prod-us1-4 Bot
Loading

Copy link
Copy Markdown
Contributor

Pipelines  Tests

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 40 Pipeline jobs failed

PR | test / test (linux, ubuntu-22.04, postgres, Postgres (py3.13-10.0-C), py3.13-10.0-C) / Postgres (py3.13-10.0-C)-py3.13-10.0-C — 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

PR | test / test (linux, ubuntu-22.04, postgres, Postgres (py3.13-10.0-UTF8), py3.13-10.0-UTF8) / Postgres (py3.13-10.0-UTF8)-py3.13-10.0-UTF8 — 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

PR | test / test (linux, ubuntu-22.04, postgres, Postgres (py3.13-11.0-C), py3.13-11.0-C) / Postgres (py3.13-11.0-C)-py3.13-11.0-C — 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

View all 40 failed jobs.

⚠️ Warnings

🧪 1 Test failed in 1 job

PR | run — ❌ 1 test failed

View more details · View in GitHub Actions

❌ test_all_config_defaults from test_config_defaults.py   View in Datadog 


================================================================================
MISSING EXPECTED DEFAULTS!
================================================================================

The following fields exist in InstanceConfig but are missing from EXPECTED_DEFAULTS:
  ['remote_queries']

To fix this:
...

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

🔄 Datadog retried 1 test - 0 passed on retry View in Datadog

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 2bc9707 | Docs | View more details | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Nov 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Disk usage change

Commit 2bc9707 compared against 6365aac.

Uncompressed

linux-aarch64 linux-x86_64 macos-aarch64 macos-x86_64 windows-x86_64
🔺 +38.11 KiB 🔺 +38.11 KiB 🔺 +38.11 KiB 🔺 +38.11 KiB 🔺 +38.11 KiB
Details
Name Version Type linux-aarch64 linux-x86_64 macos-aarch64 macos-x86_64 windows-x86_64
postgres 23.15.0 Integration 🔺 +38.11 KiB 🔺 +38.11 KiB 🔺 +38.11 KiB 🔺 +38.11 KiB 🔺 +38.11 KiB

Compressed

linux-aarch64 linux-x86_64 macos-aarch64 macos-x86_64 windows-x86_64
🔺 +11.17 KiB 🔺 +11.17 KiB 🔺 +11.17 KiB 🔺 +11.17 KiB 🔺 +11.17 KiB
Details
Name Version Type linux-aarch64 linux-x86_64 macos-aarch64 macos-x86_64 windows-x86_64
postgres 23.15.0 Integration 🔺 +11.17 KiB 🔺 +11.17 KiB 🔺 +11.17 KiB 🔺 +11.17 KiB 🔺 +11.17 KiB

@nubtron nubtron mentioned this pull request Nov 12, 2026
Draft
@dd-octo-sts

dd-octo-sts Bot commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Validation Report

Validation Description Status
pr-description Validate the pull request checklist and visible description length

To fix: Tick every item in .github/PULL_REQUEST_TEMPLATE.md and keep the description under 3,000 visible characters.		 ❌
qa-label Validate the pull request declares whether it needs QA for the next Agent release

To fix: Choose exactly one QA label:
• qa/required if the PR needs QA validation.
• qa/skip-qa if the PR does not need QA validation.		 ❌

Run ddev validate all changed --fix to attempt to auto-fix supported validations.

Passed validations (20)
Validation Description Status
agent-reqs Verify check versions match the Agent requirements file ✅
ci Validate CI configuration and code coverage settings ✅
codeowners Validate every integration has a CODEOWNERS entry ✅
config Validate default configuration files against spec.yaml ✅
dep Verify dependency pins are consistent and Agent-compatible ✅
http Validate integrations use the HTTP wrapper correctly ✅
imports Validate check imports do not use deprecated modules ✅
integration-style Validate check code style conventions ✅
jmx-metrics Validate JMX metrics definition files and config ✅
labeler Validate PR labeler config matches integration directories ✅
legacy-signature Validate no integration uses the legacy Agent check signature ✅
license-headers Validate Python files have proper license headers ✅
licenses Validate third-party license attribution list ✅
metadata Validate metadata.csv metric definitions ✅
models Validate configuration data models match spec.yaml ✅
openmetrics Validate OpenMetrics integrations disable the metric limit ✅
package Validate Python package metadata, naming, and wheel contents ✅
readmes Validate README files have required sections ✅
saved-views Validate saved view JSON file structure and fields ✅
version Validate version consistency between package and changelog ✅

View full run

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

documentation integration/postgres internal Identify a non-fork PR

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@nubtron