Tiny Bilingual Retriever

Write-up: I Distilled a 568M Multilingual Model Into a 37M Japanese-English Encoder — Here's What Survived

Distill BAAI/bge-m3 (568M) into sbintuitions/modernbert-ja-30m (the "30M" model; 36.7M parameters including embeddings) for English-Japanese cross-lingual retrieval on CPU.

Tokyo companies with global customers need English queries to find Japanese documents. A 568M model is too big for CPU serving. Can a ~37M student match the teacher's cross-lingual quality?

1. Distill bge-m3 into modernbert-ja-30m using EN-JA sentence pairs
2. Compress with Matryoshka truncation, int8/binary quantisation, ONNX Runtime
3. Fuse with MeCab-segmented BM25 via RRF
4. Measure quality kept vs index MB vs CPU ms

All values are in results/baselines.json (a tracked copy of data/results/baselines.json, which is git-ignored; re-run 2026-10-05; the nDCG values reproduced the numbers first reported here exactly). Search p50 is the brute-force cosine search over the 5,000 pre-encoded passages for one query (JA-JA run, CPU, single run), not query-encoding time, so it mostly tracks embedding size (1,024-d bge-m3 vs 256-d for the 30M models). The latency column of the first run (3.6 to 9.7 ms) differed by up to ~5x from the re-run on the same machine, so read these as order-of-magnitude.

The 71% headline is the first row. Until 2026-10-05 only the second row was on the Hub, so its card now says so.

Train/eval overlap check. The 50k distillation pairs and the EN-JA eval are both seed-0 samples of the OPUS-100 en-ja train split; 27 of the 500 eval queries (5.4%, chance level) were also training pairs. Scoring only the 473 unseen queries (scripts/leakage_check.py, results/leakage_check.json): distilled 0.4838 (vs 0.4809 overall), Matryoshka 0.4139 (vs 0.4158), untrained base 0.0371 (vs 0.0373). The overlap does not drive the result.

Key finding: The distilled student captures 71% of the teacher's EN-JA quality with 15× fewer parameters (36.7M vs 568M) and a 4× smaller index (4.9 vs 19.5 MB). Matryoshka truncation lets you choose the quality/size trade-off at serving time: dim=64 keeps 87% of the dim-256 Matryoshka model's score at 1/4 its index. Two caveats on that number: the Matryoshka fine-tune itself cost quality at full width (EN-JA nDCG 0.4809 distilled → 0.4158 at dim 256), and measured against the teacher, dim 64 is 54% (0.3613 vs 0.6742).

Fusion finding: RRF hurts when one method dominates. The student is strong at EN-JA but weak at JA-JA; BM25 is the reverse. Fusing a strong method with a weak one drags the strong method down. Hybrid only helps when both methods are reasonably good on the same query type.

Trade-off: JA-JA training improves monolingual retrieval (0.15→0.43) but degrades cross-lingual (0.42→0.26). The student becomes more balanced but loses its EN-JA specialty.

int8 is nearly free — 99.6% quality at 1/4 the index size. Binary is catastrophic — the sign of each element loses too much information.

# Baselines
PYTHONPATH=src python scripts/run_baselines.py

# Train student
PYTHONPATH=src python scripts/train_student.py

# Compress
PYTHONPATH=src python scripts/compress.py

# Hybrid fusion
PYTHONPATH=src python scripts/run_fusion.py

• EN-JA eval is synthetic (opus-100 pairs), not a standard benchmark
• Absolute nDCG inflated by subsampling; only relative comparisons meaningful
• Student starts with no retrieval training; distillation quality depends on teacher
• The public cl-nagoya/ruri-v3-30m (same size) scores higher on both JA-JA (0.96 vs 0.22) and EN-JA (0.54 vs 0.48): this is a measured recipe and cost table, not a state-of-the-art model
• Latency is PyTorch on CPU, single measurement; the p50 drop from 4.7 ms (dim 128) to 0.5 ms (dim 64) looks like measurement noise, not a 10× speed-up. scripts/compress.py contains an ONNX export path; no ONNX numbers are reported here

MIT