fix: honor the assigned Spark session for complex Params - #2809

AB#5837660

## Summary
Add a combined persistence regression after integrating the latest master fixes.
Verify that an explicitly assigned Spark session is used without promoting
default scalar parameters into explicit values. Preserve the historical review
archive through rebased links and include the focused integration review.

## Prompting Intent
The engineer approved resolving the newly introduced merge conflicts without
losing either CU behavior or upstream fixes, validating and reviewing the
result, publishing with an explicit force-with-lease, and rerunning CI.

## Linked Sources
- Feature PR: microsoft#2815
- Python method extraction: microsoft#2810
- Assigned Spark sessions: microsoft#2809
- Response cleanup: microsoft#2808
- Attachment cancellation: microsoft#2807
- Global parameter synchronization: microsoft#2806
- Integration target: 6c2699d
- Focused review: reviews/pr-2815/task-5837660-attempt-10-review-0-harness.md

## Rationale
Keep both sides of the code-generation and persistence conflicts rather than
choosing one wholesale. A combined public save/load test checks the interaction
that independent upstream and CU tests do not explicitly cover. Rebase preserves
the six original changes and their messages; all 53 archived report blobs and
modes are unchanged. Local validation passed 310 Scala and 73 Python tests,
with one existing ignored global-fuzzing test, plus style, compilation, codegen
and pinned formatting. New-head CI remains a separate post-publication gate.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 653e4257-0554-408f-9e0e-b23750249edb

## Summary
Add direct coverage for the existing no-argument and null-session serializer
fallback, including active/default session preservation, and retain the
independent review evidence. No production behavior changes in this commit.

## Prompting Intent
Complete validation, independent reviews, CI and comment reconciliation for
the remaining SynapseML pull requests after the prerequisites were merged.
Preserve legacy behavior and public signatures; do not merge any pull request.

## Linked Sources
- microsoft#2818
- microsoft#2809
- microsoft#2803
- reviews/pr-2818/task-unknown-attempt-11-stage-attempt-1-review-3-grok-4.7-slice1.md
- reviews/pr-2818/task-unknown-attempt-11-stage-attempt-1-review-6-grok-4.7-slice1.md

## Rationale
Keep fallback behavior covered at the public serializer boundary rather than
changing it in response to an incorrect baseline assumption. The baseline and
current runtime agree in 18 absent-session cases; review rebuttals preserve
the original findings and document this evidence.

## Validation
sbt "set core / Test / fork := false" "core/compile" "core/Test/compile" "core/testOnly com.microsoft.azure.synapse.ml.core.serialize.ValidateComplexParamSerializer com.microsoft.azure.synapse.ml.stages.VerifyUDFTransformerPersistence" "core/scalastyle" "core/Test/scalastyle"
Passed: 31 tests, 0 skipped, production/test compilation and both style gates.
Six independent three-family review rounds completed with explicit dispositions.
New-head CI, remote-comment reconciliation and the final CI-qualified review
remain separate gates; the pull request remains a draft.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733

Summary:
- Propagate assigned sessions through individual and array pipeline serializers.
- Restore the previous active session on success and failure; preserve the default session and existing no-argument/null-session behavior.
- Retain the merged legacy-overload tests and add nested persistence and policy regressions without changing serialized formats.

Prompting intent:
Preserve reviewed follow-up fixes after the prerequisite PRs were merged, and complete current-source validation and review.

Linked sources:
Follow-up to microsoft#2809 and microsoft#2803.

Rationale:
Nested pipeline parameters can fall back to another active Spark session instead of the session assigned to their reader or writer. That can select the wrong configuration or legacy-deserialization policy. This follow-up carries the assigned session through nested stages and restores the caller's active session.

Local validation:
sbt "set core / Test / fork := false" "core/compile" "core/Test/compile" "core/testOnly com.microsoft.azure.synapse.ml.core.serialize.ValidateComplexParamSerializer com.microsoft.azure.synapse.ml.stages.VerifyUDFTransformerPersistence" "core/scalastyle" "core/Test/scalastyle"
Passed: 29 tests, 0 skipped, compile/Test compile, both scalastyle gates.

Publication:
Explicitly authorized draft-first bootstrap; formal reviews and current-head CI remain pending.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733

## Summary
Add direct coverage for the existing no-argument and null-session serializer
fallback, including active/default session preservation, and retain the
independent review evidence. No production behavior changes in this commit.

## Prompting Intent
Complete validation, independent reviews, CI and comment reconciliation for
the remaining SynapseML pull requests after the prerequisites were merged.
Preserve legacy behavior and public signatures; do not merge any pull request.

## Linked Sources
- microsoft#2818
- microsoft#2809
- microsoft#2803
- reviews/pr-2818/task-unknown-attempt-11-stage-attempt-1-review-3-grok-4.7-slice1.md
- reviews/pr-2818/task-unknown-attempt-11-stage-attempt-1-review-6-grok-4.7-slice1.md

## Rationale
Keep fallback behavior covered at the public serializer boundary rather than
changing it in response to an incorrect baseline assumption. The baseline and
current runtime agree in 18 absent-session cases; review rebuttals preserve
the original findings and document this evidence.

## Validation
sbt "set core / Test / fork := false" "core/compile" "core/Test/compile" "core/testOnly com.microsoft.azure.synapse.ml.core.serialize.ValidateComplexParamSerializer com.microsoft.azure.synapse.ml.stages.VerifyUDFTransformerPersistence" "core/scalastyle" "core/Test/scalastyle"
Passed: 31 tests, 0 skipped, production/test compilation and both style gates.
Six independent three-family review rounds completed with explicit dispositions.
New-head CI, remote-comment reconciliation and the final CI-qualified review
remain separate gates; the pull request remains a draft.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733

Add AIPrompt on the existing OpenAIPrompt interface for explicit chat,
Responses, embeddings and native decision-model APIs. Preserve request
options, typed results, errors and raw responses through Scala and generated
Python, copy, persistence and pipelines. Add the AIPrompt notebook and guide.

Keep the contribution focused on AIPrompt and small native specializations.
Separate independent legacy defects and optional Delta recovery into their
own issues and PRs. Exercise supported notebook examples on packaged
artifacts, but leave OpenAIDecisions examples commented because Azure
OpenAI/Foundry does not support that API.

- Native inference proposal: microsoft#2759
- Deferred recovery: microsoft#2800
- Deferred live Decisions qualification: microsoft#2760
- Global parameter prerequisite: microsoft#2806
- Attachment cancellation prerequisite: microsoft#2807
- Response cleanup prerequisite: microsoft#2808
- Assigned-session prerequisite: microsoft#2809
- Python stub prerequisite: microsoft#2810
- Existing tutorial: docs/Explore Algorithms/OpenAI/OpenAI.ipynb

Reuse existing authentication and HTTP infrastructure rather than add a
second inference transport. Keep public behavior in Scala and generate
Python conveniences without changing legacy public JVM signatures or
serialized parameter shapes. Explicit API selection and lossless JSON avoid
silent option loss; bounded cancellation handles interrupted Spark work.
Keep ordinary inference lazy and replayable, without Delta, an automatic
tool loop, response-size configuration or an exactly-once claim.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733

Add AIPrompt on the existing OpenAIPrompt interface for explicit chat,
Responses, embeddings and native decision-model APIs. Preserve request
options, typed results, errors and raw responses through Scala and generated
Python, copy, persistence and pipelines. Add the AIPrompt notebook and guide.

Keep the contribution focused on AIPrompt and small native specializations.
Separate independent legacy defects and optional Delta recovery into their
own issues and PRs. Exercise supported notebook examples on packaged
artifacts, but leave OpenAIDecisions examples commented because Azure
OpenAI/Foundry does not support that API.

- Native inference proposal: microsoft#2759
- Deferred recovery: microsoft#2800
- Deferred live Decisions qualification: microsoft#2760
- Global parameter prerequisite: microsoft#2806
- Attachment cancellation prerequisite: microsoft#2807
- Response cleanup prerequisite: microsoft#2808
- Assigned-session prerequisite: microsoft#2809
- Python stub prerequisite: microsoft#2810
- Existing tutorial: docs/Explore Algorithms/OpenAI/OpenAI.ipynb

Reuse existing authentication and HTTP infrastructure rather than add a
second inference transport. Keep public behavior in Scala and generate
Python conveniences without changing legacy public JVM signatures or
serialized parameter shapes. Explicit API selection and lossless JSON avoid
silent option loss; bounded cancellation handles interrupted Spark work.
Keep ordinary inference lazy and replayable, without Delta, an automatic
tool loop, response-size configuration or an exactly-once claim.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733

Add AIPrompt on the existing OpenAIPrompt interface for explicit chat,
Responses, embeddings and native decision-model APIs. Preserve request
options, typed results, errors and raw responses through Scala and generated
Python, copy, persistence and pipelines. Add the AIPrompt notebook and guide.

Keep the contribution focused on AIPrompt and small native specializations.
Separate independent legacy defects and optional Delta recovery into their
own issues and PRs. Exercise supported notebook examples on packaged
artifacts, but leave OpenAIDecisions examples commented because Azure
OpenAI/Foundry does not support that API.

- Native inference proposal: microsoft#2759
- Deferred recovery: microsoft#2800
- Deferred live Decisions qualification: microsoft#2760
- Global parameter prerequisite: microsoft#2806
- Attachment cancellation prerequisite: microsoft#2807
- Response cleanup prerequisite: microsoft#2808
- Assigned-session prerequisite: microsoft#2809
- Python stub prerequisite: microsoft#2810
- Existing tutorial: docs/Explore Algorithms/OpenAI/OpenAI.ipynb

Reuse existing authentication and HTTP infrastructure rather than add a
second inference transport. Keep public behavior in Scala and generate
Python conveniences without changing legacy public JVM signatures or
serialized parameter shapes. Explicit API selection and lossless JSON avoid
silent option loss; bounded cancellation handles interrupted Spark work.
Keep ordinary inference lazy and replayable, without Delta, an automatic
tool loop, response-size configuration or an exactly-once claim.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733

Add AIPrompt on the existing OpenAIPrompt interface for explicit chat,
Responses, embeddings and native decision-model APIs. Preserve request
options, typed results, errors and raw responses through Scala and generated
Python, copy, persistence and pipelines. Add the AIPrompt notebook and guide.

Keep the contribution focused on AIPrompt and small native specializations.
Separate independent legacy defects and optional Delta recovery into their
own issues and PRs. Exercise supported notebook examples on packaged
artifacts, but leave OpenAIDecisions examples commented because Azure
OpenAI/Foundry does not support that API.

- Native inference proposal: microsoft#2759
- Deferred recovery: microsoft#2800
- Deferred live Decisions qualification: microsoft#2760
- Global parameter prerequisite: microsoft#2806
- Attachment cancellation prerequisite: microsoft#2807
- Response cleanup prerequisite: microsoft#2808
- Assigned-session prerequisite: microsoft#2809
- Python stub prerequisite: microsoft#2810
- Existing tutorial: docs/Explore Algorithms/OpenAI/OpenAI.ipynb

Reuse existing authentication and HTTP infrastructure rather than add a
second inference transport. Keep public behavior in Scala and generate
Python conveniences without changing legacy public JVM signatures or
serialized parameter shapes. Explicit API selection and lossless JSON avoid
silent option loss; bounded cancellation handles interrupted Spark work.
Keep ordinary inference lazy and replayable, without Delta, an automatic
tool loop, response-size configuration or an exactly-once claim.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733

* feat: add unified AI prompting and native decision model support

Add AIPrompt on the existing OpenAIPrompt interface for explicit chat,
Responses, embeddings and native decision-model APIs. Preserve request
options, typed results, errors and raw responses through Scala and generated
Python, copy, persistence and pipelines. Add the AIPrompt notebook and guide.

Keep the contribution focused on AIPrompt and small native specializations.
Separate independent legacy defects and optional Delta recovery into their
own issues and PRs. Exercise supported notebook examples on packaged
artifacts, but leave OpenAIDecisions examples commented because Azure
OpenAI/Foundry does not support that API.

- Native inference proposal: #2759
- Deferred recovery: #2800
- Deferred live Decisions qualification: #2760
- Global parameter prerequisite: #2806
- Attachment cancellation prerequisite: #2807
- Response cleanup prerequisite: #2808
- Assigned-session prerequisite: #2809
- Python stub prerequisite: #2810
- Existing tutorial: docs/Explore Algorithms/OpenAI/OpenAI.ipynb

Reuse existing authentication and HTTP infrastructure rather than add a
second inference transport. Keep public behavior in Scala and generate
Python conveniences without changing legacy public JVM signatures or
serialized parameter shapes. Explicit API selection and lossless JSON avoid
silent option loss; bounded cancellation handles interrupted Spark work.
Keep ordinary inference lazy and replayable, without Delta, an automatic
tool loop, response-size configuration or an exactly-once claim.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733

* style: format AIPrompt Python regression tests

Summary
- Apply the pinned Black formatter to the AIPrompt regression suite and its
  Fabric validation helper.

Prompting Intent
- Keep the focused AIPrompt pull request limited to required feature work
  and repairs to its own validation gates.

Linked Sources
- #2811
- https://github.com/microsoft/SynapseML/actions/runs/42014432464

Rationale
- The Python style gate identified formatting in two new test files.
- The changes preserve both files' Python ASTs, and the repository's
  tracked Python files pass Black 22.3.0.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733

* fix: align AIPrompt coverage count and notebook formatting

## Summary
Account for the additional AI test lane in Codecov's expected upload count
and apply the pinned notebook formatter to the new AIPrompt tutorial.

## Prompting Intent
Repair the deterministic checks introduced by the focused AIPrompt change
without changing dependency pins, workflows, release tooling or unrelated
notebooks, and preserve the already-executed tutorial behavior.

## Linked Sources
- #2759
- #2811

## Rationale
The authorized AI matrix lane raises expected uploads from 54 to 55.
Keep the existing coverage-count assertion unchanged and fix its configuration.
Black 22.3.0 with Jupyter support changes formatting only: all 18 code-cell
ASTs, metadata and markdown remain identical, outputs stay cleared and
OpenAIDecisions remains commented. The full CI Black command and the targeted
coverage-upload selection test pass locally.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733

* fix: harden AIPrompt cancellation and compatibility boundaries

Integrate reviewed session-persistence, codegen and attachment fixes; retain
task kill reasons, transport cancellation, value-codec regressions and the
32 supported notebook examples.

Freeze scope, finish running reviews, address blockers, publish and run
current-head CI instead of starting additional full review matrices.

- #2811
- #2818
- #2819
- #2820

Ensure the integrated public AIPrompt path preserves the same cancellation,
persistence and generated-wrapper contracts as the prerequisite fixes.
Keep unsupported Decisions examples commented out and existing limitations
visible. Do not expand scope to unrelated codegen or connector behavior.

Source-scoped qualification covers 511 cognitive and 189 core cases with zero
skips, six notebook cases and 33 installed Python cases. Compile, style,
codegen, packaging and exact installed artifact provenance are retained.
The latest test-only changes passed targeted regressions without altering
the qualified runtime artifacts. Existing reviews and dispositions are
retained; no additional full matrix is claimed. Fresh CI is still required.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733

* docs: explain AIPrompt API contracts with a diagram

## Summary
Add a shared API-routing diagram to the AIPrompt getting-started notebook
and usage guide. Clarify supported parameter mappings, intentional legacy
differences, native results and provider availability without changing code.

## Prompting Intent
Explain whether AIPrompt supports all five API types and preserves the
semantics and parameters of OpenAIPrompt and the native decision stages.
Generate an understandable diagram and include it in the PR description
and the getting-started notebook.

## Linked Sources
- #2811
- #2759
- #2760
- docs/Explore Algorithms/OpenAI/Unified AI prompting.md
- cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/ai/AIRequestAdapter.scala
- cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/ai/AIPrompt.scala
- https://github.com/user-attachments/assets/bfcb52f9-b019-455b-9e23-598a44fae229

## Rationale
Use one GitHub-hosted diagram in the PR, notebook and guide, following
the tutorials' hosted-image convention. Combine a generated background
with deterministic labels and connectors for precise technical content.
Keep the existing runnable examples unchanged and explicitly distinguish
Foundry SystemOne from the unsupported Azure OpenAI/Foundry Decisions path.
Describe per-API support instead of claiming universal legacy parity.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733

* fix: route AIPrompt API versions without a version allowlist

Summary:
- Select the v1 Chat Completions and Responses route with apiVersion.
- Forward other version strings in api-version, including for Decisions.
- Default OpenAIDecisions to v1 while preserving complete URL overrides.
- Cover scalar, per-row and persisted versions and document migration.

Prompting Intent:
Let callers use v1, preview, dated and future API versions through the
existing parameter without library-maintained version restrictions.

Linked Sources:
- User request to select the v1 URI and forward all other version strings.
- #2759
- #2811

Rationale:
API version selection belongs to the provider. Reuse the existing
ServiceParam and preserve public signatures, legacy stages and URL
override behavior instead of introducing a separate routing parameter.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733

* fix: default AIPrompt chat and responses to v1

Summary:
- Default Chat Completions and Responses requests to the v1 route.
- Preserve explicit and configured version overrides and embedding defaults.
- Cover default routing through copy, save/load and clearing an override.
- Update the usage guide and notebook migration notes.

Prompting Intent:
Keep v1 as the default API version for Chat Completions and Responses,
alongside the existing OpenAIDecisions v1 default.

Linked Sources:
- User request: keep v1 as the default for Chat Completions and Responses.
- #2811
- #2759

Rationale:
Use the v1 route without requiring a redundant setter. Change only the
generative fallback so configured scalar and per-row versions, complete
URL overrides, embeddings and legacy OpenAI stages retain their behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733

* fix: support Foundry Decisions inference

Foundry exposes Decisions for supported gpt-6-luna deployments, but the
new stages rejected Azure hosts before HTTP and selected bearer-key
authentication for every Decisions endpoint.

Route Azure resource roots through /openai/v1/decisions, retain api-key
authentication for Azure keys, and preserve Entra, proxy, version and
complete-URL behavior. Replace obsolete rejection tests with routing and
authentication regressions. Add opt-in Chat and Responses live tests,
enable the notebook's Foundry Decisions examples, and correct the guide
and shared diagram.

User intent: correct Foundry support and live-test all four inference APIs.
Sources: #2759, #2760,
https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/decisions

Validation: 215 Scala cases, 33 installed-package Python cases, six
notebook tests and four real service tests passed with zero skips.
The live tests exercised Chat, Responses, Decisions and SystemOne using
six valid stage invocations on existing Sweden Central deployments.
Compilation, Scala style, Python formatting, codegen and packaging passed.
Loaded JARs and generated Python sources matched the validated build.
No new deployments or dependency changes; no full notebook rerun or
current-head remote CI pass is claimed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba39157-c8dd-4adb-90e0-ff963a1e6733

AB#5837660

## Summary
Add a combined persistence regression after integrating the latest master fixes.
Verify that an explicitly assigned Spark session is used without promoting
default scalar parameters into explicit values. Preserve the historical review
archive through rebased links and include the focused integration review.

## Prompting Intent
The engineer approved resolving the newly introduced merge conflicts without
losing either CU behavior or upstream fixes, validating and reviewing the
result, publishing with an explicit force-with-lease, and rerunning CI.

## Linked Sources
- Feature PR: microsoft#2815
- Python method extraction: microsoft#2810
- Assigned Spark sessions: microsoft#2809
- Response cleanup: microsoft#2808
- Attachment cancellation: microsoft#2807
- Global parameter synchronization: microsoft#2806
- Integration target: 6c2699d
- Focused review: reviews/pr-2815/task-5837660-attempt-10-review-0-harness.md

## Rationale
Keep both sides of the code-generation and persistence conflicts rather than
choosing one wholesale. A combined public save/load test checks the interaction
that independent upstream and CU tests do not explicitly cover. Rebase preserves
the six original changes and their messages; all 53 archived report blobs and
modes are unchanged. Local validation passed 310 Scala and 73 Python tests,
with one existing ignored global-fuzzing test, plus style, compilation, codegen
and pinned formatting. New-head CI remains a separate post-publication gate.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 653e4257-0554-408f-9e0e-b23750249edb