Document external collectors for RDI proof of concept projects

Readers can currently select type: external without an input contract or operating guidance. This change adds Use an external collector under Data pipelines → Prepare source databases, with navigation weight 2. It explains the recommended PoC scope, production downsides, integration options, stream/event contract, configuration, transformation job, and a complete Python writer. It also explains the differences in source controls and visibility and links the guide from current source guidance, existing mentions, and relevant CLI references.

The recommendation does not declare external collectors unsupported. The same pipeline configuration and deployment command apply to self-managed VM and Kubernetes installations. Cloud setup and the 1.19.1 snapshot are unchanged.

Publication dependency: RDSC-6142 / RDI PR #2875 fixes deployment validation for a connectionless external source. The released RDI 2.0.0 API fails the documented example with Unsupported database type: None, despite passing common schema/job validation. The guide now warns about this issue. Keep this docs PR in draft until the fix is available in a release and the version guidance can name it.

Validation

• The exact inline configuration and job pass the RDI 2.0.0 configuration and job schemas and the released API's common validation. The inline Python writer matches the script used in runtime tests.
• A disposable Redis Enterprise instance and the released redis/rdi-flink-processor:2.0.0 image passed read, create, update, and delete operations, transformation/key routing, the resulting hash after a repeated create, malformed JSON and missing key/value rejection to dead-letter queues, schema/payload wrappers with schema: null, and discovery/routing of another source stream. These are processor data-path checks, not a complete installation/deployment qualification or general delivery-guarantee tests.
• Stopping the test processor left the external writer able to add events while target results stayed unchanged. After processor restart, buffered events and new updates were processed.
• Follow-up validation reproduced the deployment regression in the full released validate_config service path with the documented job, both with and without table validation. Disabling table validation does not bypass the issue.
• The released redis/rdi-api:2.0.0 service functions reject per-source external start, stop, and reset with HTTP 400, and metadata discovery with HTTP 423. Kubernetes access was mocked; these are service checks, not an end-to-end CLI/control-plane qualification. Whole-pipeline stop/reset remained accepted at that layer.
• The focused RDI Hugo build passes with the CI Hugo version 0.143.1. Navigation order, added rendered references/anchors, and code blocks were checked. Browser inspection covered the lead recommendation, navigation, input contract, and example layout. git diff --check passes.

Limits and review notes

• A complete Kubernetes installation and whole-pipeline CLI stop/start/reset/removal were not exercised. The installed host CLI could not run on this architecture. Runtime checks used the released processor directly, with checkpointing and properties aligned to the released chart. No runtime or schema changes are included.
• The writer's runtime checks used the isolated database's non-TLS endpoint. TLS and mutual TLS configuration branches are included but were not tested end to end.
• The full site build is blocked by missing generated client example files (jupyter-example) outside this change. The focused RDI build passes.
• The configuration schema currently marks external with x-internal-values: ["external"]. This change leaves the marker, generated schema visibility, and runtime behavior unchanged. Review this deliberate public guidance against that marker before publication.
• Input stream ownership was checked separately: an external source has no collector-owned input stream manifest. The guide therefore does not promise that reset or source removal deletes externally written streams.

Parallel qualification evidence

Reviewed RDSC-6142 and RDI PR #2875 at 928f605ac3a0647087f94f7bbfad8e1cf665d84f. Its Helm/Flink/single-source restrictions are smoke-runner constraints, not new product restrictions. The ticket and PR report a fresh-machine Helm installation with Redis TLS/mTLS, connectionless deployment, no managed source collector, exact target checks for all four operations, and actual whole-pipeline stop/start with queued delivery. That qualification used a development build with the API fix applied, not the exact PR commit or the released 2.0.0 images. Exact-commit smoke CI was pending at review time; the implementation PR is open. This is reference evidence and was not rerun in this docs task.

The public guide now explains manual VM configuration after installation, separates writer credentials from target secrets, and creates the pipeline before setting target secrets on installations without a default pipeline. It does not link private implementation evidence from the public pages. Full VM qualification remains unconfirmed.