Repository navigation
LSM vector index graph build stalls on large datasets (13.8M x 768 dims INT8): throughput collapses once vectors exceed the build cache and the heap fills #10543
Description
Reported in discussion #10538 by the discussion author.
13.8M WORK vertices with a 768-dim embeddings property, LSM_VECTOR index (COSINE, INT8), loaded with GraphImporter, -Xmx24g. The import itself finished (vertices 67 min, 116M edges 46 min). The vector graph build that follows then slowed down until it was killed after ~21h:
- 21:26 build starts: ~2,200 vectors/s at the beginning (2.8M after 21 min)
- 4.1M at +43 min, 6.4M at +2h, 8.0M at +3.5h
- from ~8.1M on: ~1 vector every 5 s, heap oscillating between 17 and 24 GB of 24 GB
- log line: Building graph with 13787918 vectors ... (cache enabled: size=2054352 of 13787918)
Root cause
For an INT8/BINARY index whose corpus does not fit the float build cache, every vector the cache did not hold was re-read from the index page, dequantized into a new float[768] and published into the cache, evicting an older entry: constant replacement of long-lived multi-KB objects, which drives G1 into back-to-back full collections. Reproduced at small scale (300K x 768, -Xmx1536m): 156 s, 12,261 GC pauses, 1,253 full GCs.
Fix
PR #10550: the build keeps the quantized rows as the pages store them (4x denser for INT8) and reads them through a shared-value JVector reader with a per-thread scratch vector, so a read allocates nothing. 1M x 768 in a 6 GB heap with default settings: 444 s (8,461 GC pauses, 751 full) -> 196 s (157 pauses, 0 full).
Observed, not fixed here
- NONE (float32) indexes keep the float cache and document reads, and can show the same churn.
- GraphBatch.batchUpdateVertexHeadChunks rewrites whole vertex records (the 32-minute "Batch vertex update" phase in the reporter's log).
- GraphImporter pass 1 is bound by Gson parsing of the 768-number JSONL arrays.
Activity
review(#10543): nested interface after the fields, log a skipped vect…
review(#10543): skip the allocation tests when the JVM has no per-thr…
fix(#10543): INT8/BINARY vector graph build keeps the quantized rows …
lvca commentedon Oct 21, 2026
Fixed in #10550 (milestone 26.11.1).
What was wrong: when an
INT8orBINARYvector index was larger than the build cache, the graph build kept re-reading vectors from the index pages and converting them to new float arrays. At this scale that kept the heap almost full and the build slowed to a crawl.What changed: the build now keeps the compact quantized vectors in memory (4x smaller for
INT8) and converts them only when needed, without allocating. A build that fits the float cache works as before.Measured: 1M x 768, 6 GB heap, default settings: 444 s (751 full GCs) -> 196 s (0 full GCs). With a tight heap the difference is larger.
For your setup: the build log now tells you how many vectors are kept in memory. If not all of them are, raise
arcadedb.vectorIndex.graphBuildCacheMaxHeapPercent(the log suggests a value) so the whole index stays in memory.Not covered here: indexes without quantization (float32) still use the old cache, and the slow parts of the import itself (parsing the JSONL embeddings, updating vertex edge pointers) are unchanged.
❤️ Thanks for helping make ArcadeDB better! ArcadeDB is free and open source, sustained by its community. If it's useful to you or your company, please consider becoming a sponsor to keep development going.
fix(#10543): INT8/BINARY vector graph build keeps the quantized rows …