Visitar URL original
Studio: cover the unsigned 64-bit edge of a linked-folder identity by danielhanchen · Pull Request #10606 · unslothai/unsloth · GitHub
Skip to content

Studio: cover the unsigned 64-bit edge of a linked-folder identity - #10606

Merged
danielhanchen merged 1 commit into
mainfrom
fix-stdev-uint64-roundtrip
Sep 9, 2026
Merged

danielhanchen merged 1 commit into
mainfrom
fix-stdev-uint64-roundtrip

Conversation

@danielhanchen

Copy link
Copy Markdown
Member

What this is

Follow-up hardening for the intermittent Uploads (windows-latest) failure in rag-upload-compatibility.yml:

test_reauthorizing_same_path_refreshes_root_identity_and_retains_mappings
assert ('xac8c2fef8c2fb32e', 281474976725560) == (12433365377158722350, 281474976725560)

The assertion itself was already corrected in #10536. This adds the coverage that would have caught it, and the coverage that still would not catch the same class of regression in the two neighbouring code paths.

Diagnosis

linked_folders.root_device and linked_folder_files.device are declared INTEGER, and SQLite's INTEGER is signed 64-bit. os.stat identities are unsigned, so folder_sync._store_identity writes anything above (1 << 63) - 1 as f"x{value:x}" and folder_sync._load_identity decodes it back. That encoding has been there since the feature landed; the failure was a test reading the raw column instead of going through the loader.

Confirmed by exercising it rather than by reading:

  • sqlite3 raises OverflowError: Python int too large to convert to SQLite INTEGER for 1 << 63, (1 << 64) - 1 and 12433365377158722350, and stores (1 << 63) - 1 fine.
  • _store_identity((12433365377158722350, 281474976725560)) returns exactly ('xac8c2fef8c2fb32e', 281474976725560), byte for byte the value in the CI failure. 0xac8c2fef8c2fb32e == 12433365377158722350.

Why it is intermittent

The workflow pins Python 3.12. Since python/cpython#102149, Windows os.stat reads FILE_ID_INFO, so st_dev is the full 64-bit VolumeSerialNumber and st_ino is the 128-bit FileId. A volume serial is effectively random, so its top bit is set roughly half the time. When it is set, st_dev > (1 << 63) - 1, the column holds hex text and the raw comparison fails; when it is clear, the value fits, the column holds an integer and the same test passes. Both observed instances have the top bit set: 0xac8c2fef8c2fb32e here and 0xb2f8ffc4f8ff84bf in #10536. On Linux and macOS st_dev is small, so the encoding is never reached and the case is invisible. On NTFS the 128-bit file id keeps its index in the low 64 bits, which is why the inode half of the tuple matched while the device half did not.

Who was wrong

The test, not the production code. The stored form is a deliberate, documented encoding whose round trip through _load_identity is lossless across the whole unsigned 64-bit range and beyond, and the column value is an opaque OS token that nothing outside folder_sync interprets. Asserting the raw column against st_dev over-specified the storage representation rather than the contract, which is that the identity read back equals the identity written and still compares equal on the next scan. Changing production code to keep the raw integer readable would mean widening the column to TEXT and rewriting every existing row, which is strictly worse than the flake for a persisted store.

That verdict is only sound because the round trip really is lossless, so this branch proves that instead of assuming it.

What this adds

  • test_reauthorizing_encodes_a_root_identity_above_the_sqlite_maximum forces _root_identity to return (0xAC8C2FEF8C2FB32E, 1 << 127) and drives the reauthorize path, which writes root_device through its own UPDATE. It then reconciles once more, because the reconciler reloads that row as the identity to expect. Neither of those two steps had a test that reached the encoding at all on a host whose st_dev is small.
  • test_identities_round_trip_across_the_sqlite_integer_boundary parametrises 0, 1, 2**63 - 2, 2**63 - 1, 2**63, 2**63 + 1, 2**64 - 1 and 1 << 127 through a real INTEGER column, and asserts that values SQLite can hold stay integers so rows written by older builds keep loading.
  • Assertions now name the half that diverged and what the column actually held, instead of printing a two-element tuple and leaving the reader to spot which end moved.
  • The sibling exposure, st_ino, is covered by the same parametrisation and by the existing file-identity case; a 128-bit ReFS file id is included.

Before and after

Mutation differential, whole test_rag_linked_folders.py, control worktree pinned to f452d415edce3dc937b4da1f450ed878151813af:

Mutation main this branch
none 91 passed 100 passed
_reauthorize_folder writes *identity instead of *_store_identity(identity) 91 passed, survives 1 failed, killed by test_reauthorizing_encodes_a_root_identity_above_the_sqlite_maximum
reconciler uses the raw columns instead of _load_identity 91 passed, survives 1 failed, killed by the same test
_store_identity boundary <= becomes < 91 passed, survives 1 failed, killed by ...boundary[at-max]
_load_identity drops the hex branch 2 failed 7 failed

Three mutations that main does not notice are now caught, and the fourth is caught much more loudly.

Full RAG suite differential, run sequentially against the same pinned control:

  • control: 7 failed, 794 passed, 38 skipped
  • branch: 7 failed, 803 passed, 38 skipped

The seven are the same test_rag_captioning.py cases on both sides, a missing optional module on this host, unrelated to this change. test_rag_ocr_fallback.py fails to collect identically on both sides for the same reason and was excluded from both runs.

Caveat

This host is Linux. A genuine Windows st_dev cannot be produced here, so the out-of-range identity is simulated by substituting _root_identity, using the exact value from the CI failure. What was actually observed here is the sqlite3 overflow, the encoded form matching the CI string byte for byte, the lossless round trip through a real INTEGER column, and the mutation differential above. What only Windows CI can confirm is that a real runner's volume serial takes the same path end to end and that the previously flaking case is now stable across runs whose serial has its top bit set.

A Windows volume serial number fills all 64 bits, so about half of them land above
SQLite's signed maximum and _store_identity writes them as hex text. Only the insert
half of that encoding was covered, so the reauthorize UPDATE and the reconciler's
decode were both free to drop it without any test noticing on Linux or macOS.

Add the missing reauthorize case with a forced out-of-range identity, parametrise the
round trip just below, at and above 2**63 including a 128-bit ReFS file id, and name
the half that diverged instead of dumping a raw tuple.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 9, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-09T11:36:50.841051Z 98cacf7 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@unslothai unslothai deleted a comment from chatgpt-codex-connector Bot Sep 9, 2026
@danielhanchen

Copy link
Copy Markdown
Member Author

@codex review

@danielhanchen

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Swish!

Reviewed commit: 98cacf7689

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@unslothai unslothai deleted a comment from chatgpt-codex-connector Bot Sep 9, 2026
@danielhanchen
danielhanchen merged commit 6d6ca80 into main Sep 9, 2026
38 of 41 checks passed
@danielhanchen
danielhanchen deleted the fix-stdev-uint64-roundtrip branch September 9, 2026 15:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant