[MEDI] Don't stop document processing on enricher error - #7005

* introduce EnricherOptions option bag

* implement batching

* don't validate results returned by IChatClient

* don't expose FileInfo as source via IngestionResult, as it could be Stream in the future. Just expose the document id

* Enricher failures should not fail the whole ingestion pipeline, as they are best-effort enhancements

The following PRs are included in this backport:

- [MEDI] start producing NuGet packages (dotnet/extensions/dotnet#7016)
- Update version numbers in AI changelogs (dotnet/extensions/dotnet#7008)
- [MEDI] Don't stop document processing on enricher error (dotnet/extensions/dotnet#7005)
- [MEDI] add PackageTags (dotnet/extensions/dotnet#7022)
- Add MarkItDownMcpReader for MCP server support (dotnet/extensions/dotnet#7025)
- Image generation tool (dotnet/extensions/dotnet#6749)
- Make MEAI packages use 10.0 runtime packages (dotnet/extensions/dotnet#7028)

----
#### AI description  (iteration 1)
#### PR Classification
This pull request backports multiple MEAI library updates, including new image generation features, refactoring of data ingestion enrichers, removal of legacy exporter code, and updated OpenTelemetry instrumentation.

#### PR Summary
The changes integrate new image generation tool support into chat clients with corresponding types and integration tests, refactor data ingestion enrichers to use a unified `EnricherOptions` abstraction with batching, and remove outdated JSON schema exporter and nullability helper files while updating OpenTelemetry metrics and project metadata.
- `src/Libraries/Microsoft.Extensions.AI`: Added new types (`HostedImageGenerationTool.cs`, `ImageGenerationToolCallContent.cs`, `ImageGenerationToolResultContent.cs`) and integration tests to enable hosted image generation across AI providers.
- `src/Libraries/Microsoft.Extensions.DataIngestion`: Refactored enrichers (Sentiment, Keyword, Classification, Summary) to use the new `EnricherOptions` and batching via the `Batching.cs` utility, with updated tests.
- Removed legacy schema exporter files (e.g. files under `src/Shared/JsonSchemaExporter/` and `NullabilityInfoContext/`) to clean up unused functionality.
- Updated OpenTelemetry instrumentation in OpenAI, Azure AI, Embedding, and SpeechToText clients to align with the latest semantic conventions.
- Revised project and package configuration files with updated metadata, preview stage tags, and code quality settings.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->

* Merged PR 54952: Getting ready for the 10.0 stable release. Flowing .NET Servicing

#### AI description  (iteration 1)
#### PR Classification
This PR updates dependency versions and build pipeline configurations to prepare for the .NET 10.0 stable release.

#### PR Summary
The changes update dependency and LTS versions (upgrading many from 9.0.10 to 9.0.11), enable release-specific flags, and streamline the build pipelines for servicing.
- **`eng/Version.Details.xml` and `eng/Versions.props`**: Upgraded various dependency versions and LTS numbers and set stabilization flags (e.g., `StabilizePackageVersion` to true, `DotNetFinalVersionKind` to release).
- **`azure-pipelines.yml`**: Removed the code coverage stage to simplify the CI pipeline.
- **`eng/pipelines/templates/BuildAndTest.yml`**: Added tasks to set up private feed credentials and commented out integration tests that require authentication.
- **`NuGet.config`**: Revised package source configuration by removing package source mapping and adding new internal feed URLs.
- **`Directory.Build.props`**: Suppressed NU1507 warnings to accommodate internal branch configuration.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->

* [MEDI] start producing NuGet packages (#7016)

* remove IsPackable=false, provide all mandatory properties for each package we want to ship

* add basic READMEs

* Update version numbers in AI changelogs (#7008)

* [MEDI] Don't stop document processing on enricher error (#7005)

* introduce EnricherOptions option bag

* implement batching

* don't validate results returned by IChatClient

* don't expose FileInfo as source via IngestionResult, as it could be Stream in the future. Just expose the document id

* Enricher failures should not fail the whole ingestion pipeline, as they are best-effort enhancements

* [MEDI] add PackageTags (#7022)

* Add MarkItDownMcpReader for MCP server support (#7025)

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Adam Sitnik <adam.sitnik@gmail.com>

* Image generation tool (#6749)

* Prototype of using ImageGenerationTool

* Handle DataContent returned from ImageGen

* React to rename and improve metadata

* Handle image_generation tool content from streaming

* Add handling for combining updates with images

* Add tests for new ChatResponseUpdateExtensions

* Rename ImageGenerationTool to HostedImageGenerationTool

* Remove ChatResponseUpdateCoalescingOptions

* Add ImageGeneratingChatClient

* Fix namespace of tool

* Replace traces of function calling

* More namepsace fix

* Enable editing

* Update to preview OpenAI with image tool support

* Temporary OpenAI feed

* Fix tests

* Add integration tests for ImageGeneratingChatClient

* Remove ChatRole.Tool -> Assistant workaround

* Remove use of private reflection for Image results

* Add ChatResponseUpdate.Clone

* Move all mutable state into RequestState object

* Adjust prompt to improve integration test reliability

* Refactor tool initialization

I verified that the tool creation is cached by ReflectionAIFunctionDescriptor

This change includes a small optimization to avoid additional allocation
around inserting tools into the options.

* Add integration tests for streaming

Fixes the removal of tool content - this was broken for streaming when
I changed removal to be based on callId.  We don't have the CallId yet
in the streaming case so we have to remove by name.

* React to changes and fix tests

* Address feedback

* Fix SkipTestException from ConditionalTheory

* Fix formatting

* Add back image replacement coalescing (removed in merge)

* Fix template tests and use new OpenAI

* Remove use of temporary staging nuget feed

* Address feedback

* Make ImageGeneratingChatClient use ImageGenerationTool*Content

* Remove ApplyUpdates and Coalesce ImageResults instead of DataContent.

* Workaround OpenAI issue where image data is not read for partial images.

openai/openai-dotnet#809

* Improved workaround

* Return ImageGenerationToolCallContent from OpenAI

* Add OpenAI image tool tests with representation of real traffic

* Correct the event sequence for streaming single image

* Fix some docs and refactor for clarity

* Make MEAI packages use 10.0 runtime packages (#7028)

* Make MEAI packages use 10.0 runtime packages

* Add back MEAI.Abstractions JsonSchemaExporter tests

* Address feedback

* Remove unneeded trimming suppression

* When using latest .NET packages, force System.Numerics.Tensors to 10.0 (for MEAI) (#7031)

* Add a new Microsoft.Agents.AI.Templates package with an aiagents-webapi project template (#7014)

* Initial Microsoft.Agents.AI.Templates structure

* Refine Microsoft.Agents.AI.Templates infrastructure

* Move project template infrastructure utilities into a shared folder

* Add the webapi-agents project template content with GitHub models

* Support parameterized AI Service Provider

* Rename to aiagents-webapi

* Support parameterized chatmodel and update docs with renames

* Add Snapshot tests

* Add aiagents-webapi snapshot tests

* Add aiagents-webapi execution tests (and component governance)

* Improve aiagents-webapi template parameters

* Apply suggestions from copilot code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Move shared ProjectTemplate infrastructure to not get included in Shared.Tests

* Fix the template sandbox / execution tests after moving infrastructure

* Ignore CA1716 warning about 'Shared' namespace in template tests

* Clean up template sandbox source/output

* Rename to "aiagent-webapi" and favor singular "Agent". Docs cleanup.

* Update templates dev doc to cover Microsoft.Agents.AI.Templates too

* Fix remaining template sandbox references with new paths

* Add a tool call in aiagent-webapi. Update workflow API usage for upcoming change.

Fix snapshots

* Exclude csproj.in file from template package

* Add a survey link to the aiagent-webapi template's generated readme

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Use `Microsoft.Extensions.DataIngestion` in AI Chat Web template (#7023)

* Add Markdown support

* Remove PDF support

* Revert "Remove PDF support"

This reverts commit e1d066034962c9686bf8150984b6adf0e25846c8.

* Add 'Example_GPS_Watch.md'

* Add MEDI dependencies

* Revert "[MEDI] Remove collection key type workaround (#7010)"

This reverts commit a369be9.

* MEDI integration into chat template

* Remove PdfPig dependency

* Fix citation + normalize identifier path

* Undo changes to `M.E.DI.csproj`

* Update snapshots

* Update DataIngestion unit tests to handle keys as either strings or guids

* Update SK and fix MEDI version

* Remove SK workaround

* Fix sandbox paths to allow running tests multiple times

* Reliable data ingestion

* Enable MEDI tracing

* Simplify log message

* Add `PdfPigReader` for non-Aspire template

* Invert PdfPigReader exclusion condition

* Use Markitdown MCP

* Update snapshots

* Undo changes to `IngestionPipelineTests.cs`

* Update src/ProjectTemplates/Microsoft.Extensions.AI.Templates/src/ChatWithCustomData/ChatWithCustomData-CSharp.Web/Services/Ingestion/DocumentReader.cs

Co-authored-by: Jeff Handley <jeffhandley@users.noreply.github.com>

* Update snapshots

* Improve template execution test failure output

* Support .NET 10 in aichatweb, using it by default

* Show a message when loading documents by loading docs as a separate tool

* disable the incremental ingestion

* map every PDF page to a single section

* drop SK dependency

* Add system prompt instructions for calling the LoadDocuments tool. Fix code formatting.

---------

Co-authored-by: Jeff Handley <jeffhandley@users.noreply.github.com>
Co-authored-by: Adam Sitnik <adam.sitnik@gmail.com>

* Add Agent Framework DevUI into the aiagent-webapi template (#7026)

* Integrate DevUI into the aiagent-webapi project template

* Improve aiagent-webapi Program.cs per feedback.

* Remove --no-devui. Fix OpenAI clients. Augment execution test sandbox ignores.

* Rename to Microsoft.Agents.AI.ProjectTemplates

* Set Microsoft.Agents.AI package versions

* Simplify the GitHub and OpenAI key config vars for aiagent-webapi

* Sort package references

* Fix troubleshooting section in READMEs

* Revert MEAI.Templates change. Make launchSettings .gitignore more specific.

---------

Co-authored-by: Mackinnon Buck <mackinnon.buck@gmail.com>

* Fix display of target frameworks in agents template. Hide the chat model textbox from the IDE template UI.

---------

Co-authored-by: Adam Sitnik <adam.sitnik@gmail.com>
Co-authored-by: Stephen Toub <stoub@microsoft.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
Co-authored-by: Jeff Handley <Jeff.Handley@microsoft.com>
Co-authored-by: Jeff Handley <jeffhandley@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Mackinnon Buck <mackinnon.buck@gmail.com>