Visitar URL original
Introduce Microsoft.Extensions.DataIngestion.Abstractions by adamsitnik · Pull Request #6949 · dotnet/extensions · GitHub
Skip to content

Introduce Microsoft.Extensions.DataIngestion.Abstractions - #6949

Merged
adamsitnik merged 7 commits into
dotnet:mainfrom
adamsitnik:dataIngestionAbstractions
Oct 23, 2025
Merged

Introduce Microsoft.Extensions.DataIngestion.Abstractions#6949
adamsitnik merged 7 commits into
dotnet:mainfrom
adamsitnik:dataIngestionAbstractions

Conversation

@adamsitnik

@adamsitnik adamsitnik commented Oct 22, 2025 •
edited by dotnet-policy-service Bot
Loading

Copy link
Copy Markdown
Member

The APIs got approved in #6893 (comment) and in #6895 (comment)

Microsoft Reviewers: Open in CodeFlow

@adamsitnik adamsitnik self-assigned this Oct 22, 2025
Copilot AI review requested due to automatic review settings October 22, 2025 14:23
Copilot AI reviewed Oct 22, 2025
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces the Microsoft.Extensions.DataIngestion.Abstractions library, implementing the APIs approved in the referenced GitHub issues. The library provides abstractions for processing documents from various formats into structured chunks suitable for data ingestion scenarios (e.g., RAG pipelines).

Key changes:

  • Core document representation classes (IngestionDocument, IngestionDocumentElement and its specialized types)
  • Processing pipeline abstractions (IngestionDocumentReader, IngestionDocumentProcessor, IngestionChunker, IngestionChunkProcessor, IngestionChunkWriter)
  • Support classes (IngestionChunk) for representing processed content chunks

Reviewed Changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated no comments.

Show a summary per file
File Description
Microsoft.Extensions.DataIngestion.Abstractions.csproj Project file defining multi-targeting (including netstandard2.0) and conditional package references
IngestionDocument.cs Core document container with section management and content enumeration
IngestionDocumentElement.cs Base class and specialized element types (Section, Paragraph, Header, Footer, Table, Image)
IngestionDocumentReader.cs Abstract reader with file/stream overloads and extensive media type mapping
IngestionDocumentProcessor.cs Abstract processor for document transformation pipeline
IngestionChunk.cs Generic chunk representation with metadata support and validation
IngestionChunker.cs Abstract chunker for splitting documents into chunks
IngestionChunkProcessor.cs Abstract processor for chunk transformation pipeline
IngestionChunkWriter.cs Abstract writer with disposable pattern for chunk output
Microsoft.Extensions.DataIngestion.Tests.csproj Test project configuration with analyzer suppressions
IngestionDocumentTests.cs Unit tests for document enumeration and validation

@SamMonoRT
SamMonoRT requested a review from roji October 22, 2025 15:14
adamsitnik
adamsitnik commented Oct 22, 2025
View reviewed changes
Comment thread src/Libraries/Microsoft.Extensions.DataIngestion.Abstractions/IngestionDocumentElement.cs Resolved
stephentoub
stephentoub reviewed Oct 22, 2025
View reviewed changes
Comment thread src/Libraries/Microsoft.Extensions.DataIngestion.Abstractions/IngestionChunk.cs Resolved
Comment thread src/Libraries/Microsoft.Extensions.DataIngestion.Abstractions/IngestionChunk.cs Resolved
Comment thread src/Libraries/Microsoft.Extensions.DataIngestion.Abstractions/IngestionChunk.cs Resolved
Comment thread src/Libraries/Microsoft.Extensions.DataIngestion.Abstractions/IngestionDocument.cs Resolved
Comment thread src/Libraries/Microsoft.Extensions.DataIngestion.Abstractions/IngestionDocumentElement.cs Resolved
Comment thread src/Libraries/Microsoft.Extensions.DataIngestion.Abstractions/IngestionDocumentReader.cs Resolved
Comment thread src/Libraries/Microsoft.Extensions.DataIngestion.Abstractions/IngestionDocumentReader.cs Resolved
Comment thread ...Extensions.DataIngestion.Abstractions/Microsoft.Extensions.DataIngestion.Abstractions.csproj Resolved
Comment thread ...Extensions.DataIngestion.Abstractions/Microsoft.Extensions.DataIngestion.Abstractions.csproj Resolved
Comment thread test/Libraries/Microsoft.Extensions.DataIngestion.Tests/IngestionDocumentTests.cs Resolved
stephentoub
stephentoub reviewed Oct 22, 2025
View reviewed changes
Comment thread src/Libraries/Microsoft.Extensions.DataIngestion.Abstractions/IngestionDocumentElement.cs Resolved
stephentoub
stephentoub reviewed Oct 22, 2025
View reviewed changes
Comment thread ...Extensions.DataIngestion.Abstractions/Microsoft.Extensions.DataIngestion.Abstractions.csproj Resolved
adamsitnik
adamsitnik commented Oct 23, 2025
View reviewed changes
Comment thread src/Libraries/Microsoft.Extensions.DataIngestion/Microsoft.Extensions.DataIngestion.csproj Resolved
@adamsitnik
adamsitnik force-pushed the dataIngestionAbstractions branch from db5d273 to 7bbb1ef Compare October 23, 2025 10:40
@adamsitnik
adamsitnik merged commit 72f930a into dotnet:main Oct 23, 2025
@adamsitnik
adamsitnik deleted the dataIngestionAbstractions branch October 23, 2025 11:22
This was referenced Nov 11, 2025
Closed
Merged
Merged
Merged
Merged
This was referenced Nov 19, 2025
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Merged
Closed
@github-actions github-actions Bot locked and limited conversation to collaborators Nov 23, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@stephentoub stephentoub stephentoub approved these changes
@roji roji Awaiting requested review from roji

Assignees

@adamsitnik adamsitnik

Labels

area-data-ingestion

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@adamsitnik @stephentoub