Visitar URL original
fix(knowledge): keep a bundle's document token totals on import by waleedlatif1 · Pull Request #9495 · simstudioai/sim · GitHub
Skip to content

fix(knowledge): keep a bundle's document token totals on import - #9495

Merged
waleedlatif1 merged 1 commit into
stagingfrom
fix/kb-import-token-count
Oct 31, 2026
Merged

waleedlatif1 merged 1 commit into
stagingfrom
fix/kb-import-token-count

Conversation

@waleedlatif1

Copy link
Copy Markdown
Collaborator

Summary

  • Imported documents now keep the token total the source document had. The import was summing the embedding tokenizer's per-chunk counts, which estimate differently from the chunker that produced the source total, so totals came out about 15% high. The manifest already carries the source total; the prepared index now passes it through.
  • Removed the tokenCount ?? sum-of-chunks fallback added in feat(knowledge): import knowledge base bundles through the API and sim knowledge import #9327: both processing branches now always supply a total, so the upload path behaves exactly as it did before feat(knowledge): import knowledge base bundles through the API and sim knowledge import #9327.
  • Manifest tokenCount and characterCount are bounded to the integer columns they're written to, so an out-of-range bundle is refused up front instead of failing each document at its final write.
  • Log "indexing N chunks with their bundled vectors" instead of "generating embeddings" when vectors come from the bundle (no embedding call is made on that path).

Type of Change

  • Bug fix

Testing

  • bundle-import.integration.ts (11 tests, real Postgres/Redis): the round-trip tests now assert imported token totals match the source; both fail on the pre-fix service.ts and pass with the fix
  • import-archive.test.ts (19 tests): new case refusing an out-of-range count; fails without the bound
  • Processing-path integration suites unchanged and passing: upload-read-provenance, embedding-processing-recovery, embedding-insert-batches, stored-document-recovery, processing-lock-scope, dormant-search-processing (71 tests)
  • documents/service.test.ts, biome on changed files; type-check and audits in CI

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing (new tests pass the test-audit authoring gate)
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

- Imported documents took their token total from the embedding tokenizer's
  per-chunk counts, which estimate differently from the chunker that produced
  the source total, so totals read about 15% high. The prepared index now
  carries the manifest's total, matching the source document.
- Bound manifest token and character counts to the integer columns they are
  written to, so an out-of-range bundle is refused before any work instead of
  failing each document at its final write.
- Log "indexing with bundled vectors" instead of "generating embeddings" when
  a document's vectors come from the bundle.
@vercel

vercel Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
docs Skipped Skipped Oct 31, 2026 3:51am UTC

Request Review

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 6 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Turn on auto-fix | Re-trigger cubic

@greptile-apps

greptile-apps Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge.

Summary

Preserves source document token totals when importing bundled chunks.

  • Imported documents keep the bundle's document token total.
  • Bundle imports reject counts above the database limit.
  • Bundle imports log when they index bundled vectors.

No actionable issues found. Tests were inspected, not run.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Bundle manifest] --> B[Validate count bounds]
  B --> C[Read bundled chunks]
  C --> D[Carry source token total]
  D --> E[Reuse vectors or create embeddings]
  E --> F[Save document with source total]
Loading

Reviews (1) · Last reviewed commit: "fix(knowledge): keep a bundle's document..." · Reviewed by Greptile

@waleedlatif1
waleedlatif1 merged commit e56bf3d into staging Oct 31, 2026
58 checks passed
@waleedlatif1
waleedlatif1 deleted the fix/kb-import-token-count branch October 31, 2026 03:57

This branch was previously deployed

1 inactive deployment
Preview — 93ea6615 Deployed Oct 31, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant