Visitar URL original
Introduce SectionChunker by KrystofS · Pull Request #7015 · dotnet/extensions · GitHub
Skip to content

Introduce SectionChunker - #7015

Merged
adamsitnik merged 5 commits into
dotnet:mainfrom
KrystofS:feature/Section-chunker
Nov 10, 2025
Merged

Introduce SectionChunker#7015
adamsitnik merged 5 commits into
dotnet:mainfrom
KrystofS:feature/Section-chunker

Conversation

@KrystofS

@KrystofS KrystofS commented Nov 5, 2025 •
edited by dotnet-policy-service Bot
Loading

Copy link
Copy Markdown
Contributor
Microsoft Reviewers: Open in CodeFlow

Copilot AI review requested due to automatic review settings November 5, 2025 17:02
@github-actions github-actions Bot added the area-ai Microsoft.Extensions.AI libraries label Nov 5, 2025
stephentoub
stephentoub reviewed Nov 5, 2025
View reviewed changes
Comment thread src/Libraries/Microsoft.Extensions.DataIngestion/Chunkers/SectionChunker.cs Resolved
Copilot AI reviewed Nov 5, 2025
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces a new SectionChunker class for the Microsoft.Extensions.DataIngestion library, which treats each IngestionDocumentSection as a separate entity for chunking. The implementation handles nested sections and maintains header context across chunks.

  • Adds SectionChunker class to support section-based document chunking
  • Creates comprehensive test coverage for the new chunker through SectionChunkerTests
  • Introduces a base test class DocumentChunkerTests for shared chunker test scenarios

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
src/Libraries/Microsoft.Extensions.DataIngestion/Chunkers/SectionChunker.cs Implements the new SectionChunker class with support for nested sections and header context preservation
test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Chunkers/SectionChunkerTests.cs Provides comprehensive test coverage including single/multiple sections, nested sections, size limits, and headers
test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Chunkers/DocumentChunkerTests.cs Defines abstract base test class with common test scenarios for all document chunkers

Comment thread src/Libraries/Microsoft.Extensions.DataIngestion/Chunkers/SectionChunker.cs Resolved
Comment thread test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Chunkers/SectionChunkerTests.cs Resolved
KrystofS and others added 2 commits November 5, 2025 18:16
@KrystofS
Update test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Chunke…
0be5cbc 
…rs/SectionChunkerTests.cs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@KrystofS
fix tests
be1e706
@KrystofS
KrystofS requested a review from stephentoub November 5, 2025 20:57
@adamsitnik adamsitnik added area-data-ingestion and removed area-ai Microsoft.Extensions.AI libraries labels Nov 10, 2025
@adamsitnik
adamsitnik requested a review from Copilot November 10, 2025 16:05
adamsitnik
adamsitnik approved these changes Nov 10, 2025
View reviewed changes

@adamsitnik adamsitnik left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you @KrystofS !

@adamsitnik
adamsitnik enabled auto-merge (squash) November 10, 2025 16:11
Copilot AI reviewed Nov 10, 2025
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

@adamsitnik
adamsitnik merged commit e742371 into dotnet:main Nov 10, 2025
jeffhandley pushed a commit to jeffhandley/extensions that referenced this pull request Nov 21, 2025
@KrystofS @jeffhandley
[MEDI] Introduce SectionChunker (dotnet#7015)
e7fe639
@jeffhandley jeffhandley mentioned this pull request Nov 21, 2025
Merged
@adamsitnik adamsitnik mentioned this pull request Nov 21, 2025
Closed
Copilot AI mentioned this pull request Nov 21, 2025
Merged
12 tasks done
jeffhandley pushed a commit that referenced this pull request Nov 21, 2025
@KrystofS @jeffhandley
[MEDI] Introduce SectionChunker (#7015)
e48d735
This was referenced Dec 11, 2025
Closed
Closed
Closed
Closed
Closed
Closed
Closed
This was referenced Dec 11, 2025
Closed
Closed
Closed
Closed
Closed
@github-actions github-actions Bot locked and limited conversation to collaborators Dec 11, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@adamsitnik adamsitnik adamsitnik approved these changes
@stephentoub stephentoub Awaiting requested review from stephentoub

Assignees

@KrystofS KrystofS

Labels

area-data-ingestion

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@KrystofS @stephentoub @adamsitnik