Visitar URL original
Introduce Markdown readers by adamsitnik · Pull Request #6969 · dotnet/extensions · GitHub
Skip to content

Introduce Markdown readers - #6969

Merged
adamsitnik merged 17 commits into
dotnet:mainfrom
adamsitnik:markdownReaders
Oct 31, 2025
Merged

Introduce Markdown readers#6969
adamsitnik merged 17 commits into
dotnet:mainfrom
adamsitnik:markdownReaders

Conversation

@adamsitnik

@adamsitnik adamsitnik commented Oct 27, 2025 •
edited by dotnet-policy-service Bot
Loading

Copy link
Copy Markdown
Member
Microsoft Reviewers: Open in CodeFlow

Copilot AI review requested due to automatic review settings October 27, 2025 16:43
@adamsitnik
adamsitnik requested a review from a team as a code owner October 27, 2025 16:43
Copilot AI reviewed Oct 27, 2025
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces Markdown reading capabilities to the data ingestion library through two new readers: MarkdownReader for native Markdown files and MarkItDownReader that leverages the external MarkItDown tool to convert various document formats to Markdown before parsing.

Key changes:

  • Adds MarkdownReader for parsing .md files using the Markdig library
  • Adds MarkItDownReader that wraps the MarkItDown CLI tool to convert documents (PDF, DOCX, etc.) to Markdown
  • Introduces shared MarkdownParser to parse Markdig AST into IngestionDocument model
  • Implements comprehensive test suite with conformance tests and format-specific test cases

Reviewed Changes

Copilot reviewed 12 out of 19 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
MarkdownReader.cs Implements reader for native Markdown files using Markdig parser
MarkdownParser.cs Core parsing logic converting Markdig AST to IngestionDocument model
MarkItDownReader.cs Wraps MarkItDown CLI tool to convert various document formats to Markdown
Microsoft.Extensions.DataIngestion.Markdown.csproj Project file for MarkdownReader with Markdig dependency
Microsoft.Extensions.DataIngestion.MarkItDown.csproj Project file for MarkItDownReader, shares MarkdownParser code
DocumentReaderConformanceTests.cs Base test class defining conformance tests for document readers
MarkdownReaderTests.cs Tests specific to MarkdownReader functionality
MarkItDownReaderTests.cs Tests specific to MarkItDownReader with CLI availability checks
ArrayUtils.cs Test utility for mapping 2D arrays used in table assertions
Microsoft.Extensions.DataIngestion.Tests.csproj Updated project file adding references and test file configuration
General.props Adds Markdig package reference
Versions.props Specifies Markdig version 0.42.0

Comment thread src/Libraries/Microsoft.Extensions.DataIngestion.Markdig/MarkdownParser.cs Resolved
Comment thread src/Libraries/Microsoft.Extensions.DataIngestion.MarkItDown/MarkItDownReader.cs Resolved
Comment thread ...ies/Microsoft.Extensions.DataIngestion.Tests/Microsoft.Extensions.DataIngestion.Tests.csproj Resolved
roji
roji reviewed Oct 28, 2025
View reviewed changes
Comment thread src/Libraries/Microsoft.Extensions.DataIngestion.Markdig/MarkdownParser.cs Resolved
@adamsitnik
adamsitnik requested review from cincuranet and roji October 29, 2025 12:54
ericstj
ericstj reviewed Oct 30, 2025
View reviewed changes
Comment thread test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Readers/MarkItDownReaderTests.cs Resolved
@ericstj
ericstj requested a review from Copilot October 30, 2025 01:15
ericstj
ericstj reviewed Oct 30, 2025
View reviewed changes
Comment thread src/Libraries/Microsoft.Extensions.DataIngestion.MarkItDown/MarkItDownReader.cs Resolved
Copilot AI reviewed Oct 30, 2025
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 13 out of 20 changed files in this pull request and generated 3 comments.

Comment thread src/Libraries/Microsoft.Extensions.DataIngestion.Markdig/MarkdownReader.cs Resolved
Comment thread src/Libraries/Microsoft.Extensions.DataIngestion.MarkItDown/MarkItDownReader.cs Resolved
Comment thread src/Libraries/Microsoft.Extensions.DataIngestion.Markdig/MarkdownParser.cs Resolved
ericstj
ericstj reviewed Oct 30, 2025
View reviewed changes
Comment thread src/Libraries/Microsoft.Extensions.DataIngestion.MarkItDown/MarkItDownReader.cs Resolved
cincuranet
cincuranet reviewed Oct 30, 2025
View reviewed changes
Comment thread src/Libraries/Microsoft.Extensions.DataIngestion.Markdig/MarkdownParser.cs Resolved
Comment thread src/Libraries/Microsoft.Extensions.DataIngestion.Markdig/MarkdownParser.cs Resolved
Comment thread src/Libraries/Microsoft.Extensions.DataIngestion.Markdig/MarkdownParser.cs Resolved
@adamsitnik
Merge remote-tracking branch 'upstream/main' into markdownReaders
c01c970 
# Conflicts:
#	test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Microsoft.Extensions.DataIngestion.Tests.csproj
@adamsitnik
address code review feedback:
19ece18 
- delete temporary file when .CopyToAsync fails
- handle all image types
@adamsitnik
adamsitnik requested a review from cincuranet October 30, 2025 15:44
@adamsitnik
adamsitnik requested a review from ericstj October 30, 2025 16:03
This was referenced Nov 26, 2025
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Merged
Closed
Closed
Closed
Merged
Closed
Closed
Closed
Closed
Closed
Closed
Merged
Closed
Closed
Merged
Closed
Closed
@github-actions github-actions Bot locked and limited conversation to collaborators Dec 1, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@stephentoub stephentoub stephentoub left review comments
Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@cincuranet cincuranet cincuranet approved these changes
@roji roji Awaiting requested review from roji
@ericstj ericstj Awaiting requested review from ericstj

Assignees

@adamsitnik adamsitnik

Labels

area-telemetry

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@adamsitnik @roji @stephentoub @cincuranet @ericstj