Visitar URL original
fix: honor pre-tokenization in reverse lookup by unsafePtr · Pull Request #7879 · dotnet/machinelearning · GitHub
Skip to content

fix: honor pre-tokenization in reverse lookup - #7879

Open
unsafePtr wants to merge 1 commit into
dotnet:mainfrom
unsafePtr:fix/tiktoken-reverse-pretokenization
Open

unsafePtr wants to merge 1 commit into
dotnet:mainfrom
unsafePtr:fix/tiktoken-reverse-pretokenization

Conversation

@unsafePtr

Copy link
Copy Markdown

The reverse token-count lookup passed the normalization setting in both argument positions, ignoring the pre-tokenization setting. Pass each setting to its corresponding parameter and cover the public API with a regression test.

@dotnet-policy-service dotnet-policy-service Bot added the community-contribution Pull requests submitted by community contributors label Nov 5, 2026
@codecov

codecov Bot commented Nov 5, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 70.79%. Comparing base (054ddeb) to head (da14179).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #7879      +/-   ##
==========================================
- Coverage   70.79%   70.79%   -0.01%     
==========================================
  Files        1419     1419              
  Lines      279977   279982       +5     
  Branches    27592    27592              
==========================================
- Hits       198216   198205      -11     
- Misses      73616    73639      +23     
+ Partials     8145     8138       -7     
Flag Coverage Δ
Debug 70.79% <100.00%> (-0.01%) ⬇️
production 64.24% <100.00%> (-0.01%) ⬇️
test 90.64% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs 81.20% <100.00%> (ø)
...est/Microsoft.ML.Tokenizers.Tests/TiktokenTests.cs 99.32% <100.00%> (+<0.01%) ⬆️

... and 10 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution Pull requests submitted by community contributors

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant