feat: add opt-in recovery for AIPrompt results - #2813

An interrupted AI inference job can leave users repeating expensive model calls. This PR lets AIPrompt save its inputs and completed results in Delta, so users can reopen saved results or continue unfinished work instead of restarting the whole job.

This is optional saved progress and recovery, not a new model API or a guarantee of exactly-once billing.

Related Issues/PRs

• feat: add recoverable Delta results for AIPrompt #2800 tracks recovery; Add AIPrompt with API parameter passthrough and durable Delta results #2759 tracks the broader AI feature.
• feat: add unified AI prompting and native decision model support #2811 is now merged. This PR is rebased onto master 53eaecb293 and adds recovery on top of the shipped core API; it no longer includes an unmerged prerequisite.
• The rebase preserves the model-parameter inheritance added by feat: add OpenAI prompt caching with GPT-6 Luna defaults #2781. Added regressions cover cache-extension recovery, changed cache identity on resume, and early rejection of unsupported typed cache controls.

The other prerequisite fixes (#2818, #2819 and #2820) are already merged and integrated. This PR does not close the broader umbrella or the separate live Decisions qualification in #2760.

What changes are proposed in this pull request?

For jobs such as classifying support tickets or extracting information from documents:

1. Save progress: setting outputPath snapshots the input and saves row results/errors in bounded Delta batches.
2. Read without calling the model again: loadOutput returns saved outcomes; getOutputStatus reports progress.
3. Continue after interruption: resume uses the saved input and processes only rows whose outcome has not yet been saved. Saved row errors are final outcomes, not automatically retried.

Example: assume prompt is a configured AIPrompt, tickets is the input DataFrame, and spark has Delta support.

from synapse.ml.services.ai import AIPrompt

path = "Files/aiprompt/ticket-run-001"  # Use a fresh path for each new job.
durable = prompt.copy({}).setOutputPath(path).setOutputBatchSize(1000)
results = durable.transform(tickets)  # Runs inference and saves results immediately.
results.show()                       # Reads saved results; no new model calls.

saved = AIPrompt.loadOutput(spark, path)
status = AIPrompt.getOutputStatus(spark, path)

After an interruption, recreate the matching stage/configuration and output path, supply credentials, and call durable.resume(spark). No input DataFrame is needed: recovery uses the stored snapshot. Stop the previous Spark application's work first. If it left ownership recorded, follow the explicit owner-release procedure in the recovery guide before resuming.

Important boundaries:

• Without outputPath, ordinary lazy inference is unchanged and does not require Delta.
• Durable execution is a standalone, one-run job, not a reusable stage inside a Spark ML pipeline. A new job needs a new path.
• Resume requires matching configuration, SynapseML build and Spark identity. This is not a cross-version migration feature.
• Requests whose outcomes were not saved can run again and incur another charge.
• Saving progress adds storage work. Initial execution was 18.5-22.4x slower in the prior fast synthetic benchmark; this is not a live-model performance measurement.

How is this patch tested?

Validated predecessor ae322b24ae: 60 focused Scala tests and two installed-wheel Python tests passed, with zero failures, skips or cancellations. All-module production/test compilation and Scala style, core/cognitive packaging, and Black also passed. Current head 72fd2ebeaf integrates the latest master; all seven patch commits are unchanged by git range-diff, and the core, cognitive, build, dependency and pipeline trees are identical. Fresh targeted/package validation and Azure CI are in progress; the completed runs below are not relabeled as current-head executions.

The Scala runs include three cache/preflight cases, 52 endpoint/API-version/recovery/input cases, and five independent-driver process cases with real Delta explicitly required. These verify committed-batch and lost-acknowledgement recovery without replay, permitted replay of paid responses lost before commit, interruption while waiting for the control lock, driver death at each initialization write, and a single inferencing winner during concurrent first creation/later acquisition.

The Python tests use newly packaged JARs and installed wheels with real Delta, verifying recovery without replaying committed rows and explicit confirmation before administrative owner release. The harness verified all 451 packaged Python files against generated sources, JVM class origins and artifact hashes. Its recorded head is ae322b24aef1a5f6d4a6cf22f6abe1c53e0b3257, with an empty source diff.

Predecessor Azure build 246489507 succeeded, including all 66 jobs and Databricks GPU E2E. It is a pullRequest build of refs/pull/2813/merge with source head ae322b24ae, not a manual branch-only build. The review of that head has no active or suppressed findings; its request for maintainer review of ownership, ambiguous commits and paid-request replay still applies. Separate five-API usability validation and the shared parser fix are in #2834.

Remaining readiness boundary: recurring Delta/process recovery coverage and maintainer durability/replay review remain required. The predecessor ordinary Azure AI job reports 522 passed and 188 canceled; its green status is not evidence that the optional Delta/process scenarios executed. The focused local results above explicitly enabled those dependencies and all five requested process scenarios and had zero cancellations, but do not replace recurring coverage for the full recovery suite. Broader Fabric qualification is prior-head evidence. A new Fabric smoke attempt failed with HTTP 500 while creating its scratch lakehouse, before Spark started; it provides no runtime coverage. See the prior assessment.

• I have written tests (not required for typo or doc fix) and confirmed the proposed feature/bug-fix/change works.

Does this PR change any dependencies?

• No. You can skip this section.
• Yes. Make sure the dependencies are resolved correctly, and list changes here.

No dependency manifests or pins change. Only users enabling durable execution need to supply a compatible Delta runtime.

Does this PR add a new feature? If so, have you added samples on website?

• No. You can skip this section.
• Yes. Make sure you have added samples following below steps.

Added Durable AI results.md and Quickstart - Durable AI results.ipynb. Notebook execution is disabled by default; the recovery notebook itself has not been executed as a qualification test.

1. Find the corresponding markdown file for your new feature in website/docs/documentation folder.
   Make sure you choose the correct class estimators/transformers and namespace.
2. Follow the pattern in markdown file and add another section for your new API, including pyspark, scala (and .NET potentially) samples.
3. Make sure the DocTable points to correct API link.
4. Navigate to website folder, and run npm start to make sure the website renders correctly.
5. Don't forget to add <!--pytest-codeblocks:cont--> before each python code blocks to enable auto-tests for python samples.
6. Make sure the WebsiteSamplesTests job pass in the pipeline.