Visitar URL original
Free-threading tests by crusaderky · Pull Request #1136 · google/sentencepiece · GitHub
Skip to content

Free-threading tests - #1136

Merged
taku910 merged 1 commit into
google:masterfrom
crusaderky:pytest-run-parallel
Aug 12, 2025
Merged

taku910 merged 1 commit into
google:masterfrom
crusaderky:pytest-run-parallel

Conversation

@crusaderky

@crusaderky crusaderky commented Aug 11, 2025 •

Copy link
Copy Markdown
Contributor

Run the whole test suite with pytest-run-parallel.

  • The new tests prove that the pattern
    spm.SentencePieceTrainer.train(...)
    sp = spm.SentencePieceProcessor()
    sp.Load(...)

is thread safe as long as multiple calls don't share the same objects; and that

  • everything else - e.g. SentencePieceProcessor.Normalize() - is safe even while sharing the same SentencePieceProcessor object (this is because pytest-run-parallel does not parallelise setUp, so all threads use the same shared state).

Notable exclusion is the logstream= parameter, which is NOT thread safe. However it is undocumented and untested, which makes it fall out of scope of this PR.

Note how the stress testing performed by pytest-run-parallel is stochastic, and that on 4 threads it is unlikely to catch the more uncommon race conditions. However, I've extensively ran it on 32 threads locally and did not incur in any failures.

@taku910
taku910 merged commit 31646a4 into google:master Aug 12, 2025
@crusaderky
crusaderky deleted the pytest-run-parallel branch August 12, 2025 09:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants