feat: add native Content Understanding support - #2815

AB#5837660

## Summary
Resolve the native Content Understanding review findings across task
cancellation, bounded FIFO DNS admission, Python parameter lifecycle, Spark
schema planning and operation diagnostics. Preserve the full six-stage review,
all five targeted correction passes, original findings and their resolution
history in reviews/pr-2815.

## Prompting Intent
The engineer requested focused fixes, regression coverage and an iterative
targeted review-fix loop, rather than restarting six stages. They explicitly
selected GPT/high, Grok/low and Opus/high and emphasized avoiding new defects
while correcting the findings. Keep the native contribution and detailed draft
PR scope intact; do not claim remote, live-service or merge readiness.

## Linked Sources
- Draft PR: microsoft#2815
- User-provided CU scope document and subsequent scope/review decisions in the
  authoring session; the private input document is not included in this commit.
- Connector contract: docs/Explore Algorithms/AI Services/Content Understanding.md
- Complete review and driver history:
  reviews/pr-2815/task-5837660-attempt-3-stage-attempt-2-review-1-gpt-6.1-sol.md
- Independent final targeted reports:
  reviews/pr-2815/task-5837660-attempt-4-stage-attempt-5-review-4-gpt-6.1-sol.md
  reviews/pr-2815/task-5837660-attempt-4-stage-attempt-5-review-4-grok-4.7.md
  reviews/pr-2815/task-5837660-attempt-4-stage-attempt-5-review-4-claude-opus-5.5.md
- Repository guidance: AGENTS.md and applicable .github/skills/
- Verified integration target: d3f3af6

## Rationale
Keep service behavior in Scala and reuse existing parameter setters, Spark type
coercion and bounded diagnostic merging instead of adding independent policies.
Carry and restore owning task context so queued/asynchronous calls can observe
cancellation; keep native DNS workers bounded and retain FIFO position and
permit ownership through waits and interruption. Cancellation remains local
best effort, not remote rollback.

Validate service parameters before mutation while preserving ordinary None
no-ops. Restore ordinary JVM defaults on Python transfer and convert sequences
without sending raw bytes through pickle. Plan HTTP schemas explicitly without
session/RDD inference, matching resolver, coercion and nullability behavior.

Preserve accepted creation state on unavailable lookup responses and retain
specific body/transport errors and arbitrary service-error shapes. Discarded
creation handles report uncertainty and recovery without following unsafe URLs
or replaying submissions. Normalize the post-admission DNS interruption without
changing native-worker or permit cleanup behavior.

All three final targeted reviewers returned CLEAN and passed independent
transport/scope audits on the frozen source. Historical dirty, false-positive
and invalid-invocation evidence is retained rather than rewritten. Final local
validation passed 174 CU Scala tests and four affected Python paths after style
and compilation. Earlier complete 12-test shared HTTP, 60-method Python and
code-generation gates retain their original snapshot attribution. Live CU,
Azure CI, packaged-install and representative performance evidence remain
outstanding; the PR stays draft.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 653e4257-0554-408f-9e0e-b23750249edb

AB#5837660

## Summary
Remove 50 superseded review reports from the current PR file diff. Retain the
three final source-review reports unchanged and add a concise index linking
the complete published history. Include the artifact-only cleanup review.
No implementation, test, build or configuration file changes.

## Prompting Intent
The engineer said "we shouldn't keep all of them" after comparing committed
review artifacts in other SynapseML PRs. Reduce reviewer-facing clutter while
preserving original findings, dispositions, failed invocations and provenance.

## Linked Sources
- Current PR: microsoft#2815
- Published 53-report archive: https://github.com/BoraElkin/SynapseML/tree/1bab6fb4aef3d00b616f09b023a03062abc9695e/reviews/pr-2815
- Review decisions and resolution history: https://github.com/BoraElkin/SynapseML/blob/1bab6fb4aef3d00b616f09b023a03062abc9695e/reviews/pr-2815/task-5837660-attempt-3-stage-attempt-2-review-1-gpt-6.1-sol.md
- Comparable merged evidence-pruning precedent: microsoft#2775

## Rationale
Keep current source signoffs visible without retaining every intermediate
report in the final tree. Verified all 53 hosted paths, blob hashes and modes
against the published ancestor before pruning. Immutable links retain the
full history without rewriting commits or relabeling historical outcomes.
The index distinguishes targeted final review from the earlier six-stage
review and preserves outstanding external-validation limits.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 653e4257-0554-408f-9e0e-b23750249edb

AB#5837660

## Summary
Use the existing parameter-aware whole-scalar converter in getOrDefault and
extractParamMap, matching generated typed getters for JVM-backed ServiceParams.
Remove the redundant recursive converter and preserve the raw-byte guard.
Add a regression for empty and nonempty case-class sequences in configured and
restored wrappers, and record the scoped follow-up review in the review index.

## Prompting Intent
The engineer requested fixing Copilot's confirmed parameter-conversion finding
and continuing validation until the PR is ready for human review, without
introducing unrelated changes or losing existing review history.

## Linked Sources
- PR: microsoft#2815
- Original Copilot finding: microsoft#2815 (comment)
- Repeated current-head finding: microsoft#2815 (comment)
- Prior source and review archive: https://github.com/BoraElkin/SynapseML/tree/1bab6fb4aef3d00b616f09b023a03062abc9695e/reviews/pr-2815

## Rationale
Reuse the converter that already knows the ServiceParam's JSON format rather
than extending a separate recursive converter without parameter type metadata.
The regression failed in four expected subcases before the change. Core
code generation, all 35 selected Python compatibility tests and pinned Black
passed afterward, including map, sequence, column, default and raw/pickle-byte
paths. No Scala API, persisted format, dependency or pipeline change is needed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 653e4257-0554-408f-9e0e-b23750249edb

AB#5837660

## Summary
Update both Codecov notification upload counts from 54 to 55 and document the
41 UnitTests legs. Include the focused clean review evidence.

## Prompting Intent
The engineer requested making the native Content Understanding PR ready for
reviewer handoff, authorized validation, and reported the failed CI Helpers job.
Correct the directly coupled failure without removing tests or weakening gates.

## Linked Sources
- Feature PR: microsoft#2815
- Failed validation: https://dev.azure.com/msdata/b9b2accc-2d1c-45b3-9d24-0eb5d78cc47f/_build/results?buildId=245266944
- Existing regression: tools/ci/tests/test_e2e_impact.py::test_selection_preserves_all_expected_coverage_uploads
- Focused review: reviews/pr-2815/task-5837660-attempt-8-review-0-harness.md

## Rationale
The added CU unit-test matrix leg produces one additional coverage upload.
Both Codecov consumers must wait for that report. The existing assertion
correctly caught the mismatch, so retain it and the full test matrix unchanged.
The affected helper module now passes all 121 tests locally. No coverage
percentage threshold, pipeline execution, dependency pin, or product code changes.
Full Azure validation is still required on the corrected merge revision.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 653e4257-0554-408f-9e0e-b23750249edb

AB#5837660

## Summary
Use the established Testable naming convention for two serialization fixtures
and align the OpenAI schema-setter regression with JVM-visible scalar state.
Include the focused clean review evidence.

## Prompting Intent
The engineer requested following the reviewer-handoff plan: fix CI failures
caused by the CU PR, validate compatibility, publish corrections, and verify
full current-head CI without weakening tests or expanding feature scope.

## Linked Sources
- Feature PR: microsoft#2815
- Failed validation: https://dev.azure.com/msdata/A365/_build/results?buildId=245268757
- Existing fixture discovery: src/test/scala/com/microsoft/azure/synapse/ml/core/test/fuzzing/FuzzingTest.scala
- Existing schema-setter contract: cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/openai/HasOpenAIResponseSchema.scala
- Focused review: reviews/pr-2815/task-5837660-attempt-9-review-0-harness.md
- Separate anomaly baseline evidence: https://dev.azure.com/msdata/A365/_build/results?buildId=245261452

## Rationale
Test-only persistence fixtures must not be discovered as exported stages.
The existing Testable convention handles this without adding exemptions or
changing global stage checks; both persistence variants retain their assertions.
The OpenAI test must distinguish stale pending Python state from a deliberately
configured JVM scalar. Check pending-map removal directly, retain the schema
assertion after transfer, and require public isSet to observe the JVM value.

Core style and compilation passed, as did 10 serializer tests, 10 global
discovery tests, and 36 OpenAI/ServiceParam Python tests. The one pre-existing
ignored fuzzing check is unchanged. Pinned Black passed. The original Azure
run supplies failing-before evidence for both corrections.

No product implementation or pipeline changes. The independent anomaly
failure also occurs on exact master before a built-in successful retry; the
VW R failure happened during TLS setup before tests. Neither is silently
waived or repaired here. A fresh exact-revision Azure run remains required.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 653e4257-0554-408f-9e0e-b23750249edb

AB#5837660

## Summary
Resolve the native Content Understanding review findings across task
cancellation, bounded FIFO DNS admission, Python parameter lifecycle, Spark
schema planning and operation diagnostics. Preserve the full six-stage review,
all five targeted correction passes, original findings and their resolution
history in reviews/pr-2815.

## Prompting Intent
The engineer requested focused fixes, regression coverage and an iterative
targeted review-fix loop, rather than restarting six stages. They explicitly
selected GPT/high, Grok/low and Opus/high and emphasized avoiding new defects
while correcting the findings. Keep the native contribution and detailed draft
PR scope intact; do not claim remote, live-service or merge readiness.

## Linked Sources
- Draft PR: microsoft#2815
- User-provided CU scope document and subsequent scope/review decisions in the
  authoring session; the private input document is not included in this commit.
- Connector contract: docs/Explore Algorithms/AI Services/Content Understanding.md
- Complete review and driver history:
  reviews/pr-2815/task-5837660-attempt-3-stage-attempt-2-review-1-gpt-6.1-sol.md
- Independent final targeted reports:
  reviews/pr-2815/task-5837660-attempt-4-stage-attempt-5-review-4-gpt-6.1-sol.md
  reviews/pr-2815/task-5837660-attempt-4-stage-attempt-5-review-4-grok-4.7.md
  reviews/pr-2815/task-5837660-attempt-4-stage-attempt-5-review-4-claude-opus-5.5.md
- Repository guidance: AGENTS.md and applicable .github/skills/
- Verified integration target: d3f3af6

## Rationale
Keep service behavior in Scala and reuse existing parameter setters, Spark type
coercion and bounded diagnostic merging instead of adding independent policies.
Carry and restore owning task context so queued/asynchronous calls can observe
cancellation; keep native DNS workers bounded and retain FIFO position and
permit ownership through waits and interruption. Cancellation remains local
best effort, not remote rollback.

Validate service parameters before mutation while preserving ordinary None
no-ops. Restore ordinary JVM defaults on Python transfer and convert sequences
without sending raw bytes through pickle. Plan HTTP schemas explicitly without
session/RDD inference, matching resolver, coercion and nullability behavior.

Preserve accepted creation state on unavailable lookup responses and retain
specific body/transport errors and arbitrary service-error shapes. Discarded
creation handles report uncertainty and recovery without following unsafe URLs
or replaying submissions. Normalize the post-admission DNS interruption without
changing native-worker or permit cleanup behavior.

All three final targeted reviewers returned CLEAN and passed independent
transport/scope audits on the frozen source. Historical dirty, false-positive
and invalid-invocation evidence is retained rather than rewritten. Final local
validation passed 174 CU Scala tests and four affected Python paths after style
and compilation. Earlier complete 12-test shared HTTP, 60-method Python and
code-generation gates retain their original snapshot attribution. Live CU,
Azure CI, packaged-install and representative performance evidence remain
outstanding; the PR stays draft.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 653e4257-0554-408f-9e0e-b23750249edb

AB#5837660

## Summary
Remove 50 superseded review reports from the current PR file diff. Retain the
three final source-review reports unchanged and add a concise index linking
the complete published history. Include the artifact-only cleanup review.
No implementation, test, build or configuration file changes.

## Prompting Intent
The engineer said "we shouldn't keep all of them" after comparing committed
review artifacts in other SynapseML PRs. Reduce reviewer-facing clutter while
preserving original findings, dispositions, failed invocations and provenance.

## Linked Sources
- Current PR: microsoft#2815
- Published 53-report archive: https://github.com/BoraElkin/SynapseML/tree/1bab6fb4aef3d00b616f09b023a03062abc9695e/reviews/pr-2815
- Review decisions and resolution history: https://github.com/BoraElkin/SynapseML/blob/1bab6fb4aef3d00b616f09b023a03062abc9695e/reviews/pr-2815/task-5837660-attempt-3-stage-attempt-2-review-1-gpt-6.1-sol.md
- Comparable merged evidence-pruning precedent: microsoft#2775

## Rationale
Keep current source signoffs visible without retaining every intermediate
report in the final tree. Verified all 53 hosted paths, blob hashes and modes
against the published ancestor before pruning. Immutable links retain the
full history without rewriting commits or relabeling historical outcomes.
The index distinguishes targeted final review from the earlier six-stage
review and preserves outstanding external-validation limits.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 653e4257-0554-408f-9e0e-b23750249edb

## Summary
Retry definite Content Understanding 429 responses indefinitely by default,
honor Retry-After, and preserve the existing no-replay rules for accepted or
uncertain submissions. Add optional preview input truncation and a runnable
invoice notebook that reads saved results rather than repeating submissions.
Restore the generated binary getter after the rebase changed shared service
parameter conversion.

## Prompting Intent
Apply the concrete findings from the review of microsoft#2815 to
the engineer's existing microsoft#2698. Keep direct REST calls,
follow SynapseML HTTP conventions, cover concurrency and parameter behavior,
and make the documentation and examples clear and safe to rerun.
No Azure DevOps task ID was supplied.

## Linked Sources
- Existing PR: microsoft#2698
- Related review: microsoft#2815 (review)
- REST analyze API: https://learn.microsoft.com/rest/api/contentunderstanding/content-analyzers/analyze?view=rest-contentunderstanding-2026-06-01-preview
- Python SDK API: https://learn.microsoft.com/python/api/azure-ai-contentunderstanding/azure.ai.contentunderstanding.contentunderstandingclient
- Invoice schema: https://github.com/Azure/content-understanding-toolkit/blob/main/prebuilt-schema/2025-11-01/procurement/invoice.md
- Model setup: https://learn.microsoft.com/azure/ai-services/content-understanding/concepts/models-deployments
- Service limits: https://learn.microsoft.com/azure/ai-services/content-understanding/service-limits

## Rationale
A definite 429 without an operation handle can be retried without treating it
as uncertain acceptance. Keep other POST failures and all handle-bearing
submissions out of that retry path. Preserve an explicit opt-out for callers
that need bounded retries. Do not spend the ordinary poll budget on throttling.

Use the shared HTTP scheduler rather than a detached CU worker. Carry the
owning Spark task context into asynchronous requests so cancellation and task
completion terminate their retry loops. Convert InterruptedException into a
nonfatal cancellation failure because Scala Future otherwise leaves its
promise unfinished. Do not claim that a Python-only interrupt cancels a
driver-side JVM request.

Keep omitted, false, and null-column truncation behavior distinct. Include the
analysis option in request identity but exclude the execution retry policy.
The notebook generates deterministic synthetic input, journals the operation
before polling, and extracts invoice fields from storage. Preserve the
single-writer and acceptance-to-journal crash-window limitations.

The generated document-bytes getter bypasses the shared scalar converter:
Py4J already returns raw bytes, while PySpark's generic converter treats a
bytearray as a pickled return value. The existing save/load test reproduced
the failure against the newly generated wrappers. Keep the fix local to CU.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 51d494ca-e7e5-44a3-9bd6-353038a5c877

## Summary
Preserve validated analyzer-management operation handles when a response fails,
exclude inactive poll-only columns from durable writes, and reject explicit
input-truncation settings on API versions older than 2026-06-01-preview.
Add public-path regressions and document the version requirement. Format the
invoice notebook with CI's Black Jupyter support.

## Prompting Intent
Improve the existing Content Understanding PR using the integration review
findings. Keep direct REST calls, safe submission recovery, generated Python
support, and normal SynapseML parameter behavior. Address concrete review
findings without adding SDK dependencies or changing other connectors.

## Linked Sources
- User PR: microsoft#2698
- Related review: microsoft#2815
- Management handle: microsoft#2698 (comment)
- Preview validation: microsoft#2698 (comment)
- Writer projection: microsoft#2698 (comment)
- SDK validation: https://github.com/Azure/azure-sdk-for-python/blob/a35469f4df2fe3cbfed42efbc240b270f0b26dda/sdk/contentunderstanding/azure-ai-contentunderstanding/tests/test_allow_input_truncation_gating.py
- REST analyze: https://learn.microsoft.com/rest/api/contentunderstanding/content-analyzers/analyze?view=rest-contentunderstanding-2026-06-01-preview
- Notebook style failure: https://dev.azure.com/msdata/A365/_build/results?buildId=245646399

## Rationale
Keep management failures visible without replaying a potentially accepted PUT.
Validate the handle before attaching it to an exception, so unsafe URLs remain
unusable. Derive writer columns from the transformer's existing active-parameter
map instead of maintaining a second list.

Validate the effective API version before each submission, including mixed
scalar/column bindings and explicit false. Null still omits the option. A
minimum date avoids rejecting future versions merely because SynapseML does
not yet name them. Four new regressions failed against the previous code.
No ADO task was supplied for this GitHub follow-up.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 51d494ca-e7e5-44a3-9bd6-353038a5c877

## Summary
Use the shared generated-setter helper for Content Understanding setParams.
Reject conflicting scalar and column arguments before mutating the stage.
Preserve ignored None arguments and changes made through separate setter calls.
Add Python regressions and document the behavior.

## Prompting Intent
Improve the user's existing Content Understanding PR using the integration
review findings. Keep it consistent with other SynapseML callers and address
concrete findings from the review of the pushed head.

## Linked Sources
- User PR: microsoft#2698
- Current-head review: microsoft#2698 (review)
- Related integration review: microsoft#2815 (review)
- Shared setter validation: https://github.com/microsoft/SynapseML/blob/6c2699defcfb24d5bff5ba55f2252ba2d8cbea24/core/src/main/python/synapse/ml/core/schema/Utils.py

## Rationale
The custom setter loop silently accepted conflicting arguments and chose a
binding by insertion order. Reuse the same helper as the generated constructor
instead of duplicating its validation. Keep skip_none=True so existing CU calls
continue to ignore None, and retain the CU-specific byte and persistence
overrides. The regression reproduces the failure with string, byte, map, and
false boolean values in both argument orders.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 51d494ca-e7e5-44a3-9bd6-353038a5c877

## Summary
Configure Content Understanding from the Fabric runtime when users omit an
endpoint and credentials. Preserve its AI gateway path through submission,
polling, and durable recovery. Add scoped implicit authentication, bounded
token refresh, regression tests, and a keyless Fabric OCR notebook.

## Prompting Intent
The engineer asked to include Fabric wiring in the existing Content
Understanding PR, following the OpenAIPrompt setup while preserving BYO
Foundry support, direct REST calls, polling, retries, and concurrency.

## Linked Sources
- User PR: microsoft#2698
- Related integration review: microsoft#2815
- Fabric client: https://github.com/microsoft/SynapseML/blob/37948d7d03ef0fa7f1aa24c54c2dd4d8a29c8dc0/core/src/main/scala/com/microsoft/azure/synapse/ml/fabric/FabricClient.scala
- Shared authentication: https://github.com/microsoft/SynapseML/blob/37948d7d03ef0fa7f1aa24c54c2dd4d8a29c8dc0/cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/CognitiveServiceBase.scala
- REST analyze: https://learn.microsoft.com/rest/api/contentunderstanding/content-analyzers/analyze?view=rest-contentunderstanding-2025-11-01
- Python API: https://learn.microsoft.com/python/api/azure-ai-contentunderstanding/azure.ai.contentunderstanding.contentunderstandingclient

## Rationale
Reuse FabricClient and shared credential precedence, but retain the CU
protocol's admission-aware retries. CU uses the AI workload route rather
than OpenAI's route. Attach implicit tokens only to the exact runtime-derived
endpoint and restrict operation handles to the same gateway scope.

Never refresh explicit credentials or replay a submission that returned a
handle. Keep token acquisition failures during polling resumable instead of
losing the accepted job or recording a terminal service failure.

The managed gateway does not support analyzer administration. Reject those
calls locally and keep BYO Foundry as the documented option for custom
analyzers and LLM capabilities unavailable through the managed service.
No ADO task was supplied for this GitHub change.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 51d494ca-e7e5-44a3-9bd6-353038a5c877