Repository navigation
Export and import of a database: time, memory and exactness, native and JVM wheel #369
Description
We will export and import databases often now: the tower's nightly dump, the per-host agents' state files, restore drills, and moving datasets between the laptop and the mini. The only recorded problem with EXPORT DATABASE and IMPORT DATABASE is correctness (ArcadeData/arcadedb#8871: infinite FLOAT and DOUBLE values and DECIMAL precision were lost, since closed). Nobody has measured their speed or memory, and nobody has checked that a round trip is exact for the data we actually have.
Questions
- How long and how much memory do
EXPORT DATABASEandIMPORT DATABASEneed per million records, for documents, for vertices with edges, and for a vector property? - What does the file cost on disk, plain and compressed?
- Is the round trip exact: record counts per type, the content of every property, the edges, and the special values (infinite and NaN floats, DECIMAL scale, dates, lists, embedded maps)?
- How do they compare with loading the same data through
insert_manyor Arrow, and withBACKUP DATABASEand a restore? - Native wheel or JVM wheel.
Design
- Data: synthetic and deterministic. 10,000, 100,000, and 1,000,000 vertices, 5 edges per vertex, a property mix that includes the special values above, and a variant with a 768-dimension float32 vector column.
- Wheels:
arcadedb(native) andarcadedb-jvm, both at 26.11.1.post2, JDK 25 for the JVM wheel. - Host: the laptop first, untimed and directional. The mini only if a finding matters.
- Repetitions: 3 per cell. Figures: wall time, peak resident memory, file size. Correctness: counts per type and an order-independent content digest before and after, with a rule that any difference is a failure listed per property and per type, never rounded away.
- Pre-registered rules, a README and a harness follow the layout of the other bindings studies; numbers are checked with
numcheck.pybefore anything is posted.
Acceptance
- The harness and its rules are written and a smoke run at 10,000 vertices passes.
- The laptop run is queued and its results are posted here.
- Each failed round trip becomes an upstream issue with a Java repro on the engine API.
- The docs of the bindings say how long a dump takes per million records and what it needs in memory.
Activity
tae898 commentedon Nov 11, 2026
The harness and its pre-registered rules are written, and the smoke run at 10,000 vertices passed on both wheels (26.11.1.post2, engine build
a3a5d12457): 90 comparisons passed, 0 failed, and every negative control I planted (one changed property value, one deleted record, per shape and wheel) was caught, so a round trip that loses data cannot score as a pass. A probe of 27 single-value cases round-trips exactly on both wheels, including infinite and NaN floats, which agrees with ArcadeData/arcadedb#8871 being closed.BACKUP DATABASEcan be measured too: pointingarcadedb.server.backupDirectoryat the scratch folder is in force on both wheels. One repetition at 10,000 vertices is a smoke, not a result; the queued runs are untimed and directional, 10,000 and 100,000 vertices first, 3 repetitions each, and the 1,000,000 cells after I have read the 100,000 memory.Three things from the smoke that I will check at size before I claim anything:
EXPORT DATABASEtook 8.93 s on the native wheel and 10.25 s on the JVM wheel, against 0.48 s and 1.17 s for the same vertices and edges without the vector, and the plain JSONL is 96 MB against 10 MB. Import goes from 1.26 s to 4.52 s (native) and from 2.92 s to 4.99 s (JVM).SELECTof a record with anARRAY_OF_FLOATSproperty raisedModuleNotFoundError: No module named 'numpy'in an environment with only the wheel, and numpy is not a dependency of the wheel.Next: the 10,000 and 100,000 runs, then a Java repro on the engine API for anything that fails or looks wrong at size.
tae898 commentedon Nov 11, 2026
Verdict: PASS at 10,000 vertices, INVALID at 100,000 vertices under my pre-registered rule R7, so I do not claim a pass at the larger size.
At 10,000 vertices every rule held on both wheels. At 100,000 vertices every comparison of counts, content, schema, load and restore also agreed on both wheels (table below), but R7 plants one change in a copy of the source database and requires the comparison to report exactly one difference. The deleted-record control was caught as exactly one at both sizes. The control that adds 1 to one
p_longwas caught at 10,000 vertices (1 of 10,000 records, in that property and no other), but the harness keeps per-record hashes only up to 50,000 vertices, so at 100,000 vertices all 12 of those controls reportedp_long: ? of 100,000(a digest difference in the right property and in no other, with the number of records unknown). As written, the rule calls that not caught, marks the check as not trusted, and voids every PASS at that size. A rule change after the first run above 10,000 vertices is a new study, so I leave the rule as it is. I have queued a rerun of the controls at 50,000 vertices with the same seed, 1 repetition and all six cells on both wheels (queue lineseix-native-50k-ctlandeix-jvm-50k-ctl, new output directories, nothing of the finished runs touched), and I will post its result here as an addendum. At 50,000 vertices the per-record hashes are kept, so it can close R7 at that size only; it does not change the 100,000-vertex verdict.Method. Laptop, Intel(R) Core(TM) Ultra X9 388H, Linux 7.0.0, Python 3.12.13,
nice -n 19, cores 12 to 15 (the four low-power cores), untimed and directional, in a lane that no other queued job shares. Wheelsarcadedb==26.11.1.post2(native) andarcadedb-jvm==26.11.1.post2(JDK 25.0.5, max heap 2g at 10,000 vertices and 4g at 100,000 vertices), engine 26.11.1 builda3a5d12457, engine defaults. Data is synthetic and deterministic: documents, or vertices with 5 edges per vertex, or the same with a 768-dimension float32 vector, each in amixvariant (infinite and NaN floats, -0.0, extremes, DECIMAL scales 0 to 12, dates, nested lists and maps, a NUL, unicode) and afinvariant (finite substitutes in the same slots). 3 repetitions per cell, every phase in its own fresh process. Time istime.perf_counteraround theEXPORT DATABASE(SQL) orIMPORT DATABASEcall only; peak memory isVmHWMof that process right after the call; a record is a document, or a vertex or an edge. Exactness compares, per type and property, an order-independent digest of every record read back from the source database, from the imported database, and from a restoredBACKUP DATABASEarchive.BACKUP DATABASEis written througharcadedb.server.backupDirectory, which was in force on every backup row. All times are seconds.Pass rule, criterion by criterion (R4, the
k of Nreporting rule, is built into every line; R9 to R13 are information only).BACKUP DATABASEarchive equals the source (counts and content)p_longchanged, caught as exactly one difference in that propertyRound trip exact, per comparison (each comparison is one repetition of one cell; the
finandmixcells are both in these counts):No export or import raised, and every probe case of the 27 per wheel (infinite and NaN floats, -0.0, the extremes of a 64-bit integer, DECIMAL scales, a NUL, unicode, nested list and map, a vector, the date types) round-trips exactly on both wheels, which agrees with ArcadeData/arcadedb#8871 being closed. No round trip failed, so this run files no upstream issue.
Time, median (min to max) of 3 repetitions,
mixcells:Seconds per million records (the median at that size scaled to a million, not measured at a million), and the time of the original load (
insert_manyfor documents,graph_batchfor the graph shapes, engine defaults, so its durability is not matched to import) and ofBACKUP DATABASE, medians in seconds:gen) s nativePeak resident memory, MiB,
VmHWMright after the call, median (min to max); the last two columns are the same counter right after opening the database (the baseline of the interpreter and the native library or the JVM):Disk, megabytes (10^6 bytes): the export file as written (
.jsonl.tgz), the plain JSONL after decompressing it, and theBACKUP DATABASEarchive:The same four tables for the
fincellsgen) s nativeReading it. At 100,000 vertices on the native wheel,
EXPORT DATABASEtook 2.50 s for the documents, 4.56 s for the vertices with edges and 87.25 s with the vector column, andIMPORT DATABASE3.06, 16.79 and 53.03 s. On the JVM wheel export took 2.81, 4.97 and 97.54 s and import 3.73, 13.45 and 33.95 s. In seconds per million records that is 25.0 (documents), 7.6 (vertices with edges) and 145.4 (vector) for export on native, and 30.6, 28.0 and 88.4 for import. The vector column is what costs: export of the vector cell is 19.1 times the export of the graph cell on native and 19.6 times on JVM, with 9.2 times the plain JSONL.Native or JVM wheel: at 10,000 vertices the JVM wheel was slower in every document and graph export and import, by 2.3x to 3.9x. At 100,000 vertices export on JVM took 1.08 to 1.14 times as long as on native, and import reversed for the two shapes with edges: 13.45 s on JVM against 16.79 s on native for vertices with edges, 33.95 against 53.03 s with the vector column, while documents stayed 1.2x slower on JVM. I did not test why; each phase is a fresh process, so the JVM wheel pays its warm-up in every call, which fits the gap at the small size and is a guess for the rest.
The native import of the graph cell grew 16.0x for 10 times the vertices (the JVM wheel 5.2x). I looked at the stderr of every export and import phase at both sizes: engine INFO lines only (plus the JDK's incubator-module notice on the JVM wheel), no engine warning or error, and every phase exited 0. I did not find the reason, so I would not scale the 100,000 figures to a million.
Memory at 100,000 vertices: export of the document cell peaked at 161 MiB (native) and 324 MiB (JVM), against 88 and 151 MiB right after opening; import of the vector cell peaked at 1,668 MiB (native) and 2,414 MiB (JVM,
-Xmx4g). The JVM peak depends on the heap setting, the native wheel has no such setting here.Against the alternatives:
EXPORT DATABASEtook 11.1 to 18.0 times as long asBACKUP DATABASEof the same database (ratio of medians over the three shapes and both wheels), and import took 0.34, 1.98 and 0.77 of the original load time on native for documents, vertices with edges and the vector shape (JVM 0.37, 0.84, 0.94), so on native the import of the vertices with edges costs more than the load that built them. An Arrow load was not measured (it needs pyarrow, outside the wheel and the standard library).Disk: at 100,000 vertices the compressed export is 11.62 MB for documents (native-loaded records; 11.45 MB when the JVM wheel loaded them), 18.48 MB for vertices with edges and 384.06 MB with the vector (plain JSONL 52.06, 105.01 and 961.89 MB on native). The compressed graph and vector files of the two wheels differ by at most 0.3 percent. The documents again export to different text on the two wheels, as in the smoke: plain JSONL 52.06 MB from native-loaded records against 41.16 MB from JVM-loaded records (1.26x), and both round-trip exactly, so I read it as a stored-type difference between the two wheels'
insert_many, which I have not traced.Against the smoke comment above: the 10,000-vertex medians of this run are 0.82 to 1.03 times the smoke's single repetition for export and import on both wheels (the smoke read 8.93 s native and 10.25 s JVM for the vector export, this run reads 8.55 and 10.26 s), and the plain JSONL sizes are byte for byte the same as in the smoke. The smoke ran the two wheels at the same time on the same four cores with the JVM wheel at
-Xmx1g; this run took one wheel at a time at-Xmx2g. I think that explains the differences, which I did not test.Acceptance items. Harness, rules and the smoke: done earlier. Laptop run: done for 10,000 and 100,000 vertices, results above; the 1,000,000 cells have not run. Failed round trips become upstream issues: none failed, so none filed. Docs of the bindings: not written; I would put per-million figures in them only after a 1,000,000 run, given the native import scaling above.
What the numbers do not say. One host (this laptop, on its low-power cores, so absolute times are slow ones), 3 repetitions per cell, 100,000 vertices at most, JDK 25 only, embedded mode only, a warm page cache, a clean close and no crash test, no server-mode export, engine defaults. The wheels' two families are never pooled.
Edited 2026-11-12: the R3 row of the table above is wrong for the JVM wheel (the schema comparison was empty there). The correction is in #369 (comment).
tae898 commentedon Nov 11, 2026
R7 holds at 50,000 vertices (24 of 24 negative controls caught as exactly 1 difference on both wheels, every R1 to R3, R5 and R6 comparison agrees at that size), and the 100,000-vertex verdict stays INVALID because the rule is unchanged.
This is the control-only rerun I queued above. Method: 50,000 vertices, one repetition, the same seed (20261111) as the 100,000-vertex run, all six cells (documents, vertices with 5 edges each, and the same with a 768-dimension float32 vector, each in the
mixand thefinvariant) on both wheels (arcadedb==26.11.1.post2andarcadedb-jvm==26.11.1.post2, engine builda3a5d12457, JDK 25.0.5 with max heap 4g), the same cores 12 to 15 undernice -n 19, and the same harness: the hashes ofrun.py,run_all.shandscore.pyrecorded by these runs equal the pinned ones of the earlier runs. At 50,000 vertices the harness keeps per-record hashes, so every difference is listed per record. I scored it as queued,score.py --reps 1, with no rule and no file edited. Both runs exited 0 in every phase, every row is ok, all six cells are present on each wheel, and the scorer saysPASS (PASS=126, FAIL=0, INVALID=0, INCOMPLETE=0).Rule R7 applied literally: the control that adds 1 to one
p_long(ofp_id3) was caught in 12 of 12 cells (6 per wheel) as exactly 1 difference in that property and in no other, for exampleDoc.p_long: 1 of 50000 e.g. 3; the control that deletes one record was caught in 12 of 12 cells as exactly 1 difference in the count of that type (Doc count: 1 of 50000andLink count: 1 of 250000), with the properties of the deleted record differing by that one record. The other comparisons at this size: import equals source for counts, content digests, schema and indexes in 12 of 12 comparisons (R1 to R3), source equals generator in 12 of 12 (R5), the restoredBACKUP DATABASEarchive equals the source in 12 of 12 (R6), and the probe of single awkward values is exact in 54 of 54 cases (27 per wheel). No round trip failed.What it does not show. It does not change the 100,000-vertex verdict: R7 as pre-registered applies to that run, where the changed-value control could only be seen as a digest difference with the number of records unknown, so that run stays INVALID and I do not rescore it under another rule. What this run adds is the same harness, seed and code path at a size where the controls can be counted per record, and there they are caught exactly. It is one repetition per cell, so it has no spread and I do not use its times; the three-repetition tables above stand. It is the same host and the same two wheels, and 50,000 vertices is still not 1,000,000.
tae898 commentedon Nov 12, 2026
A correction to my R3 row, which I got wrong for the JVM wheel.
The R3 row of the table in my verdict comment says R3 (types, property names and declared types, and index definitions equal after import) held (36 of 36) at 10,000 and at 100,000 vertices. The 36 is 18 comparisons per wheel. On the native wheel those comparisons were real: all 18 of 18 at each size read a non-empty schema. On the JVM wheel they were not: at 10,000 vertices 18 of 18, at 100,000 vertices 18 of 18 (and 6 of 6 in the 50,000-vertex run) compared an empty description with an empty one. My schema reader kept only the nested rows that are Python dicts, and the JVM wheel hands the nested property and index rows of
SELECT FROM schema:typesover as Java objects, so every property and every index was dropped without an error and "equal" meant "both empty". Only the type names were compared there.What this does not touch: R1, R2, R5, R6 and the record controls of R7 (the changed
p_longand the deleted record) compare content digests, and those were not empty on either wheel (12 to 17 digested properties in every digest file at 100,000 vertices, everyp_idcounted). The native schema comparison also had little to find, one index per main type (p_id) and no vector index, so even there the claim was narrow.So the R3 claim holds for the native wheel with that narrow schema and is not established for the JVM wheel. I have queued a corrected check as a separate folder. It reads the schema with three independent readers, refuses to pass unless a reader returns exactly the schema that was declared (so an empty comparison is INVALID, not a pass), uses a richer schema (an
LSM_VECTORindex with cosine and 768 dimensions, a UNIQUE and two NOTUNIQUE indexes, an edge index, DECIMAL and DATETIME properties), and plants one difference in each of two copies of the source (an index dropped, a declared type changed) that has to be caught as exactly one difference on both wheels. It runs at 1,000, 10,000 and 100,000 vertices on both pinned wheels, and I will post the result here, whichever way it comes out. Sorry for the wrong claim.tae898 commentedon Nov 12, 2026
Result of the corrected schema check, as promised in my correction above: PASS at 10,000 and at 100,000 vertices on both wheels (
arcadedb==26.11.1.post2andarcadedb-jvm==26.11.1.post2, engine 26.11.1 builda3a5d12457). The scorer saysPASSwith 120 rules passed, 0 failed, 0 invalid and 0 incomplete.R3 (types, property names with declared types, and index definitions equal after import) compared 24 cell pairs and 24 passed. This time the comparison is not empty: the vector cells hold 2 types, 15 properties and 5 index definitions (an
LSM_VECTORcosine index on 768 dimensions, a UNIQUE and two NOTUNIQUE indexes, an edge index), the graph cells 2 types, 14 properties and 4 index definitions, and both wheels read the same schema. A cell only counts when its source database reads back exactly as declared, so an empty read would have been INVALID, not a pass.Each planted difference (one index dropped, one declared type changed) was caught as exactly one difference, on both wheels. The 100,000-vertex content rules passed as well.
Same limits as before: one repetition, one laptop, 100,000 vertices at most, and the schema was read through independent readers on each wheel. The first comment's timings and memory tables are unchanged.