Visitar URL original
benchmarks: official arcadedb-driver 0.4.0 in the served ArcadeDB arm… · humemai/arcadedb-embedded-python@413d390 · GitHub
Skip to content

Commit 413d390

Browse files
tae898claude
andcommitted
benchmarks: official arcadedb-driver 0.4.0 in the served ArcadeDB arms, in a layer of its own (DECISIONS #258, #259)
OfficialSession (lean_http.py) against arcadedb-driver 0.4.0, read from the wheel (sha256 00fd23e3...): - db handles are cached per (root, auth, timeout, database); srv.db() per call threw away the per-database URL cache that arcadedb-drivers PR #168 (our #108 finding) relies on. - the per-call timeout was accepted and dropped (driver default: no timeout). The driver has no per-call timeout, only ArcadeDBServer(timeout=httpx.Timeout), so each distinct timeout gets a server of its own (the lanes use 30, 60, 300, 600). httpx timeouts surface as lean_http.Timeout, other transport errors as ConnectionError, as for LeanSession. - command with limit goes through the driver (0.3.0 added it; present in 0.4.0). - OFFICIAL_PIN is 0.4.0 and OfficialSession refuses any other installed version, so a stale image cannot run 0.2.0 under a 0.4.0 pin. The image: the served ArcadeDB client cells ran in the shared dbbench:client. Moving arcadedb-driver in PKGS[client] would change that build arg, rebuild the image under a new digest, and split every comparator's measured rows (the stage scripts run build_images.sh on every stage). So PKGS[client] keeps 0.2.0 (commented) and the driver arrives in dbbench:client-arcadedb: Dockerfile.client-arcadedb, FROM dbbench:client plus one pip layer of ARCADEDB_DRIVER_PKGS (driver, httpx, httpcore, h11, anyio, idna, certifi, attrs, typing-extensions, websockets, all ==, PyPI latest of 2026-11-07). runner.py points every served arcadedb arm (client_server, name arcadedb*) at it; e4 and the embedded arms keep dbbench:arcadedb. engine_pin.FLOATING_OK knows the name; make_2611_stages derives the target from BACKENDS. Docs: pin strings 0.2.0 -> 0.4.0 in COMPARATORS.md, CAMPAIGN.md row 82, PROTOCOL.md (labelled: official for /query and /command, lean for bulk loads and streams), the stamp in three tests and the publish fixture rows. COMPARATORS.md: the two stale 'newest stable' sentences (pymongo, SurrealDB SDK) are replaced by a Client holds table (each lagging client, newer version and PyPI date, why held). Tests (stubbed driver and httpx, no network, no docker): handle caching, timeout forwarding, limit path, timeout and transport errors, the pin and the refusal, the registry swap, the build script text, one build of the layer under a fake docker. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
1 parent 48efdd6 commit 413d390

14 files changed

Lines changed: 294 additions & 32 deletions

‎benchmarks/experiments/CAMPAIGN.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -221,7 +221,7 @@ Rules every campaign keeps:
221221
| 77 | ArcadeDB's query heap budget (`arcadedb.queryMaxHeapRAM`, default half the JVM heap) raised for the cells the default refuses: the l1tpc tpch10 olap `top_parts` GROUP BY in both ArcadeDB arms (DECISIONS #175, ArcadeData/arcadedb#9402) | ArcadeDB embedded and served, the documents lane, ONLY in the repair stage | **mechanism written, default OFF (humemai/arcadedb-embedded-python#260); the repair stage is generated after qP09 and run before the tier 1 to tier 2 boundary.** `BENCH_ARCADE_QUERY_MAX_HEAP_RAM_MB` adds the JVM option to the embedded arm and to the served arm's JAVA_OPTS and stamps `arcadedb_query_max_heap_ram_mb` and a source on the row; `override: arcadedb_query_ram`. Which rows are the headline (the default rows with the refusal shown, or the raised rows as an escape hatch like row 21's cap and row 75's body limit) is decided at landing. | not an instrument change: unset, nothing differs |
222222
| 78 | The comparator client paths are recorded per row (issue #261; DECISIONS #173, #174): `neo4j_rust_ext`, `redis_hiredis`, `es_json_serializer`, and `pg_client_path` (`autocommit`, `copy_binary`, `binary_cursor`); `qdrant_transport` (#172) was already stamped | comparators only: no ArcadeDB arm | built (DECISIONS #173, #174) | issue #261 |
223223
| 81 | The sparse lane warms up before its timed pass and the runner holds the CPU idle state for each timed cell (DECISIONS #177). Text on the page too: CPU idle states are not controlled on the host, and single-client times under a millisecond include the cores' wake-up latency (PAGE-SPEC section 0a, `setup.cpu_idle_note`) | the sparse lane (warm-up); every lane's rows (hold) | **built (humemai/arcadedb-embedded-python#278).** `bench_common.SPARSE_WARMUP_DEFAULT` untimed searches per engine, 1,000, one full round as the study behind the warm-up used, drawn from queries DISJOINT from the timed set so a query or result cache cannot flatter an engine: the dataset's extra queries when it holds them, else a fixed-seed sample of documents used as queries; the row stamps `sparse_warmup_queries` as run and `sparse_warmup_source` = `extra_queries` or `document_sample`. The hold is `/dev/cpu_dma_latency` at `BENCH_CPU_DMA_LATENCY_US` for the length of each timed cell: 10 microseconds in the 2026-11 pass, because the tree default `bench_common.CPU_DMA_LATENCY_DEFAULT_US` (0, no sleep state deeper than polling) kept every core in a busy-wait loop and made the host hot and slow (PROTOCOL.md section 7 has the A/B); it needs write access to the device on the host, and a hold that cannot be taken never fails the cell: rows stamp `cpu_dma_latency_us`, `cpu_dma_latency_held`, and `cpu_dma_latency_error` when it failed. `BENCH_SPARSE_WARMUP=N` and `BENCH_CPU_DMA_LATENCY_US=N` set the count and the target (PROTOCOL.md section 7); neither can be turned off. `override: sparse_warmup`, `override: cpu_dma_latency`; `test_idle_and_warmup.py`, `test_instrument_method.py` | part of the instrument |
224-
| 82 | The served ArcadeDB headline is the vendor's own `arcadedb-driver` 0.2.0 (sync, httpx; `lean_http.OfficialSession`, stamp `arcadedb-driver 0.2.0 (sync, httpx)` with `arcadedb_http_client_role` = `headline`), and the lean HTTP client (`lean_http.LeanSession`) is the labelled sensitivity row | every ArcadeDB SERVED arm that talks HTTP through `lean_http.Session()`; no comparator client moves | **built (humemai/arcadedb-embedded-python#282, the instrument's client row).** The sensitivity row is the same cell run again as a twin job (`runner.ARCADEDB_HTTP_ARMS`, run_id ending `_clientlean`, `arcadedb_http_client_role` = `sensitivity`; the runner sets `BENCH_ARCADEDB_HTTP_CLIENT=lean` for it). `lean` is the client the 26.10.1 pass and the published October rows used, so the sensitivity row stays comparable with them; rows measured with the two clients are never mixed in one column. The driver carries `/query` and `/command` calls; the `GET /server` probe, the `/batch` bulk load, transactions, NDJSON streams and bodies with a field the driver has no argument for stay on the lean client. The PostgreSQL wire is the second sensitivity row (row 6): the served SQL cells also run as a `_clientpgwire` twin (`arcadedb_client_wire` = `pgwire`, role `sensitivity`), the read statements only. The `client` image gains `arcadedb-driver==0.2.0` at its next rebuild (`build_images.sh`, COMPARATORS.md pin table). Measured on the official 26.10.1 image (d36b4ca3a), 7 interleaved rounds: the bound Cypher point read costs the client 55.5 us of CPU per call with `lean` against 11.6 for a bare-socket variant that was tried and retired, identical answer digests; the 100-row read is engine-bound and moves less | part of the instrument |
224+
| 82 | The served ArcadeDB headline is the vendor's own `arcadedb-driver` 0.4.0 (sync, httpx; `lean_http.OfficialSession`, stamp `arcadedb-driver 0.4.0 (sync, httpx)` with `arcadedb_http_client_role` = `headline`), and the lean HTTP client (`lean_http.LeanSession`) is the labelled sensitivity row | every ArcadeDB SERVED arm that talks HTTP through `lean_http.Session()`; no comparator client moves | **built (humemai/arcadedb-embedded-python#282, the instrument's client row).** The sensitivity row is the same cell run again as a twin job (`runner.ARCADEDB_HTTP_ARMS`, run_id ending `_clientlean`, `arcadedb_http_client_role` = `sensitivity`; the runner sets `BENCH_ARCADEDB_HTTP_CLIENT=lean` for it). `lean` is the client the 26.10.1 pass and the published October rows used, so the sensitivity row stays comparable with them; rows measured with the two clients are never mixed in one column. The driver carries `/query` and `/command` calls; the `GET /server` probe, the `/batch` bulk load, transactions, NDJSON streams and bodies with a field the driver has no argument for stay on the lean client. The PostgreSQL wire is the second sensitivity row (row 6): the served SQL cells also run as a `_clientpgwire` twin (`arcadedb_client_wire` = `pgwire`, role `sensitivity`), the read statements only. The driver is at 0.4.0 since DECISIONS #259 (it was 0.2.0 when this row was written): it runs in `dbbench:client-arcadedb`, a layer over `dbbench:client` (`Dockerfile.client-arcadedb`, `build_images.sh` target `client-arcadedb`, the driver's environment pinned in `ARCADEDB_DRIVER_PKGS` and in COMPARATORS.md), so `dbbench:client` keeps its digest and its comparators' measured rows stay on one client; `OfficialSession` refuses any other driver version (`lean_http.OFFICIAL_PIN`). The session keeps one database handle per (root, auth, timeout, database) so the driver's URL cache works (arcadedb-drivers PR #168), forwards each call's timeout as an `httpx.Timeout`, and sends `command` with `limit` through the driver. Measured on the official 26.10.1 image (d36b4ca3a), 7 interleaved rounds: the bound Cypher point read costs the client 55.5 us of CPU per call with `lean` against 11.6 for a bare-socket variant that was tried and retired, identical answer digests; the 100-row read is engine-bound and moves less | part of the instrument |
225225
| 84 | The dense lane's id index for the delete (humemai/arcadedb-embedded-python#291, DECISIONS #178 index parity): ArcadeDB, pgvector, Neo4j, Memgraph, FalkorDB, DuckDB, and LanceDB get an index on the vector id before or right after the load, the other engines already delete through their own key | the dense lane (l3d), every arm | **built.** Every dense row stamps `dense_id_index` (the statement, or `key`); `override: dense_id_index`; `test_dense_id_index.py`. The index build is inside `ingest_s`. The /next page says ArcadeDB's delete ran without an id index in rows that lack the stamp (`_dense_id_index_note`, list line `dense_id_index`; `test_next_measurement_wording.py` holds the wording) | part of the instrument |
226226
| 85 | Streamed results for the served ArcadeDB time-series arms, `BENCH_ARCADEDB_TS_NDJSON=1`, default OFF: both served arms ask the query endpoint for NDJSON and read the records line by line, as the PostgreSQL and MongoDB arms read rows as their server sends them | the time-series lane (l4), `arcadedb_ts_doc_server` and `arcadedb_ts_native_server` only | **mechanism written, default OFF; MEASURED 2026-10-09 and NOT USED at 26.11.1:** at ts100 the switch helped the documents arm (0.853 [0.713, 0.928]) and hurt the native arm (1.087 [1.083, 1.088]), identical rows, so it stays off and the page promises nothing; it needs a reader that parses as data arrives and a ts1000 re-measure first. Unset, nothing differs (`test_ts_ndjson.py`); set, rows stamp `arcadedb_ts_result_format` = `ndjson`; `override: ts_ndjson` | not an instrument change: unset, nothing differs |
227227
| 87 | Two failures of the 26.10.1 pass that were not results (humemai/arcadedb-embedded-python#344, the failure triage of 2026-10-20): (1) `chroma_dense` never ran, because the override registry `overrides.py` shadowed chromadb's dependency `overrides` on `/work` (`ImportError: cannot import name 'overrides'`); the registry is now `override_registry.py`. (2) Memgraph's storage access timeout (default 1 s) refused the deep10m vector index build in both dense arms; the dense arms now run `--storage-access-timeout-sec=172800`, longer than any cell cap | Chroma dense (every scale) and Memgraph dense (both precisions); no other arm moves (the rename changes no measurement) | **built on branch issue/344-chroma-memgraph, not on main**: `override_registry.py` and every import, `runner.MEMGRAPH_STORAGE_ACCESS_TIMEOUT_S`, the read-back `memgraph_storage_access_timeout_s`, PROTOCOL.md section 7, COMPARATORS.md. Mini picks it up at the `git pull` of the stage after the merge, which is the user's to time; a repair stage reruns Chroma dense and Memgraph dense deep10m | `.notes/bench/repros/failure-triage-20261020` |

‎benchmarks/experiments/COMPARATORS.md‎

Lines changed: 20 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -72,16 +72,33 @@ Pins follow DECISIONS #87 (the latest self-hosted stable release at campaign tim
7272
| Neo4j driver (Neo4j AND Memgraph arms) | `neo4j==6.4.1` | client package |
7373
| Neo4j driver Rust extension (the vendor's documented accelerator, DECISIONS #173) | `neo4j-rust-ext==6.4.1.0` | client package |
7474
| pgvector Python adapter (binary COPY, DECISIONS #174) | `pgvector==0.5.1` | client package |
75-
| ArcadeDB official Python driver (sync, httpx; the served ArcadeDB arms' headline client, CAMPAIGN section 7 row 72) | `arcadedb-driver==0.2.0` | client package |
75+
| ArcadeDB official Python driver (sync, httpx; the served ArcadeDB arms' headline client, CAMPAIGN section 7 rows 72 and 82) | `arcadedb-driver==0.4.0` | client package, in the `dbbench:client-arcadedb` layer (below) |
76+
77+
**The served ArcadeDB client image (DECISIONS #258, #259).** The served ArcadeDB arms run in `dbbench:client-arcadedb`: `dbbench:client` plus one pip layer (`Dockerfile.client-arcadedb`, `build_images.sh` target `client-arcadedb`, list `ARCADEDB_DRIVER_PKGS`), so `dbbench:client` stays byte-identical and every comparator row measured on it stays on one client. `arcadedb-driver==0.2.0` remains in `dbbench:client` (the `[client]` line of `build_images.sh`, unused by any cell now); `lean_http.OfficialSession` refuses to start on any driver version but `lean_http.OFFICIAL_PIN` (0.4.0). The driver's environment is pinned at the latest PyPI release of 2026-11-07 (the driver itself asks only `attrs>=23.2`, `httpx>=0.27`, `typing-extensions>=4.10`, `websockets>=14`): `arcadedb-driver==0.4.0` (2026-11-05), `httpx==0.28.1`, `httpcore==1.0.9`, `h11==0.16.0`, `anyio==4.16.0`, `idna==3.20`, `certifi==2026.10.23`, `attrs==26.1.0`, `typing-extensions==4.16.0`, `websockets==17.4`. This layer moves httpx, anyio, certifi and the other listed libraries only inside the served ArcadeDB client image, where no comparator client runs. Scope of the driver in a served cell: `/query` and `/command` (with `limit`); bulk `/batch` loads, NDJSON streams, transactions, probes and raw text bodies stay on the lean HTTP client (DECISIONS #259 (4)).
7678

7779
**Held below the newest, and why.**
7880

7981
- **MongoDB 8.3.x is skipped, and 9.0.4 is the pin** (the one engine where #103d overrode #87; hold lifted 2026-10-23). 8.3.13, the newest 8.3 patch (Docker Hub 2026-10-17), still refuses to start on the bench kernel with the SERVER-121912 guard ("Linux kernel versions 6.19 and newer has a known incompatibility"), both as plain `mongo` and inside the Search image, with mongot 1.70.4 and with 1.70.5. 9.0.4 (Docker Hub 2026-10-17, amd64 manifest `sha256:b609cdfba3a8…`; 9.0.0 shipped 2026-09-28 and the release notes call 9.0 the current stable release) starts on the same kernel. Why it starts was not investigated: SERVER-125742 exempts kernels from upstream 7.0.14 but reads `uname -r`, which Ubuntu keeps at 7.0.0 (open as SERVER-136848, reported against 8.3.11 and 9.0.2); the bench kernel 7.0.0-38-generic is upstream 7.0.14, so it carries the userspace fix the guard exists for, and the old argument that an older 8.3.x lacking the warning would run a configuration its vendor calls unsafe no longer holds. Why not stay on 8.2.12, which also boots: MongoDB's compatibility table for MongoDB Search lists mongot 1.70.1 and later against MongoDB Server 8.3 only and says 8.2 "is no longer a supported source deployment for MongoDB Search", so 8.2.12 + mongot 1.70.4 was a pairing the vendor no longer supports. 9.0 is not in that table either: 9.0.4 + mongot 1.70.5 is exercised, not documented. Laptop test 2026-10-23, kernel 7.0.0-38-generic, containers on cores 12-15 with `-m 3g`/`-m 4g`, `pymongo==4.18.1`, through the repo's own `l1_tpc.MongoTPC` on 30,000 seeded synthetic lineitem rows and `mongo_common` on 3,000 seeded 32-d vectors: 8.2.12 boots, 8.3.13 does not (also with mongot 1.70.4 and 1.70.5), 9.0.4 boots; on 9.0.4 against 8.2.12 the five analytics pipelines, 200 committed new-order transactions, 50 payments, the abort (nothing changed), the write-conflict error (code 112), `with_transaction` and 100 single-record CRUD operations gave identical digests; `$vectorSearch` through mongot 1.70.5 returned the same top-10 lists as 8.2.12 + 1.70.4 on 99.3% of 300 queries (mean set overlap 0.9993, recall 1.000 against exact search) for `quantization: none`, and 0.960 mean set overlap for `scalar` (recall 0.878 against 0.873, a quantized index is not deterministic across builds); both indexes read their quantization back, and a `$vectorSearch` inside a multi-document transaction is still refused. Not tested: performance (every MongoDB row has to be re-measured on 9.0.4), a full TPC-H scale, restart/recovery, the time-series and graph lanes. mini runs kernel 7.0.0-31-generic, the same 7.0 line; a one-minute `docker run --rm mongo@sha256:b609cdfb… mongod --version`-style start check there is a sanity step, not a requirement.
8082
- **DuckDB and DuckPGQ 1.5.6**, not 2.0.0 (checked 2026-10-23, laptop, duckdb 2.0.0 from PyPI): the `duckpgq` community build (`29ca668a22`) and `vss` both exist for v2.0.0 and both LOAD, VSS HNSW works, but DuckPGQ registers no syntax: `CREATE PROPERTY GRAPH` and `GRAPH_TABLE ... MATCH` answer `Parser Error: syntax error`, with or without `SET enable_legacy_parser` or `allow_parser_override_extension`. So the graph arm cannot be measured on 2.0.0 (DECISIONS #103d: the newest version at which every arm can be measured). Lift the hold when a DuckPGQ build for 2.x parses; the recipe is in `build_images.sh`. 1.5.5 had no DuckPGQ build (HTTP 404) and 1.5.6 does (`61f5d082`).
81-
- **SurrealDB server v3.2.4 and SDK 2.0.0**: the SDK is the newest stable. The server is held one minor behind the newest stable (`v3.3.4`, `latest` on 2026-10-23) on purpose: on 3.3.4 the l2 graph OLTP delete left one person and its edge behind with no error in 2 of 3 runs (0 of 2 on 3.2.4), so the bump waits for a repeat study (humemai/arcadedb-embedded-python#483). Lift the hold when 3.3.x or later deletes cleanly.
82-
- **Chroma 1.5.9, sqlite-vec 0.1.9, Memgraph 3.14.0, pymongo 4.18.1, python-arango 8.3.6, falkordb 1.7.1, redis 8.1.0**: the newest stable releases (sqlite-vec's `0.1.10a4` is a pre-release). **pymilvus 3.0.2** (with the `bulk_writer` extra, which brings `minio`, `azure-storage-blob` and `ml-dtypes`) is what Milvus's compatibility table names for a 3.0.3 server.
83+
- **SurrealDB server v3.2.4 and SDK 2.0.0**: the SDK is pinned at the October re-pin; a newer release exists (3.0.0, 2026-11-04); not changed because its rows are already measured ("Client holds" below). The server is held one minor behind the newest stable (`v3.3.4`, `latest` on 2026-10-23) on purpose: on 3.3.4 the l2 graph OLTP delete left one person and its edge behind with no error in 2 of 3 runs (0 of 2 on 3.2.4), so the bump waits for a repeat study (humemai/arcadedb-embedded-python#483). Lift the hold when 3.3.x or later deletes cleanly.
84+
- **Chroma 1.5.9, sqlite-vec 0.1.9, Memgraph 3.14.0, redis 8.1.0**: the newest stable releases (sqlite-vec's `0.1.10a4` is a pre-release). **pymilvus 3.0.2** (with the `bulk_writer` extra, which brings `minio`, `azure-storage-blob` and `ml-dtypes`) is what Milvus's compatibility table names for a 3.0.3 server.
8385
- **SQLite 3.46.1** is not a pin: it is the stdlib `sqlite3` of the `python:3.12-slim` base, read back at connect. **ArcadeDB is not a comparator**: it is pinned to our own wheel and the matching `arcadedb-c25` image (its row above).
8486

87+
**Client holds (checked on PyPI 2026-11-07; DECISIONS #259 (3)).** Each comparator client library is pinned at the October re-pin (2026-09-19, individual bumps to 2026-10-25) and not changed afterwards because its rows are already measured: a newer client mid-campaign would split a comparator's rows across two client versions. Each package below has a newer release today. The hold is a decision, not a claim that the pin is the newest.
88+
89+
| Client | Pinned | Newer release (PyPI date) | Why held |
90+
|---|---|---|---|
91+
| psycopg (binary) | 3.3.6 | 3.3.8 (2026-11-04) | rows already measured |
92+
| pymongo | 4.18.1 | 4.19.0 (2026-10-14) | rows already measured |
93+
| python-arango | 8.3.6 | 8.3.7 (2026-10-27) | rows already measured |
94+
| falkordb | 1.7.1 | 1.8.1 (2026-10-25) | rows already measured |
95+
| lancedb | 0.39.0 | 0.41.0 (2026-10-29) | rows already measured |
96+
| surrealdb (Python SDK) | 2.0.0 | 3.0.0 (2026-11-04) | rows already measured; a major version |
97+
| ladybug | 0.22.0 | 0.22.3 (2026-11-06) | tied to the baked `vector` extension sha256 (`LADYBUG_EXTENSIONS`); moving it means re-reading that sha and re-measuring |
98+
| duckdb | 1.5.6 | 2.0.0 (2026-10-21) | documented hold: DuckPGQ registers no syntax on 2.0.0 (above) |
99+
| azure-storage-blob | 12.31.0 | 12.31.1 (2026-10-29) | rows already measured (a pymilvus `bulk_writer` dependency) |
100+
| numpy, pandas, pyarrow (harness libraries) | 2.5.3, 3.0.6, 25.0.1 | 2.5.4 (2026-10-10), 3.1.1 (2026-11-04), 26.0.1 (2026-11-03) | rows already measured; "move them deliberately" (`build_images.sh`) |
101+
85102
**Why some moves were smoked.** The PostgreSQL set moves as one major (#103d); AGE gated it, being the only member packaged per major (`postgresql-18-age`). PostgreSQL 18 turns data checksums on by default, a write-path cost the page keeps because it is PostgreSQL's own default. Neo4j is pinned at .1 because 2026.08.0 could fail queries through its resource monitoring, and a new index provider (`vector-2026.08`) is auto-selected for new indexes, so the dense operating point is a measurement and not an inheritance (the Neo4j row says how).
86103

87104
**AGE 1.8.0 is not a labelling change.** Holding a 5,000-edge batch fixed and varying the vertex count, AGE 1.7.0 takes 1.56 s at 160,000 vertices and 11.14 s at 500,000 (a jump of the shape of a planner flip, so the ratio is not a constant and must not be quoted as one), where 1.8.0 takes 0.18 s at both: 62x at 500,000. October's pg_age build numbers differ from September's for that upstream reason and not for an instrument change; a pg_age cell that ran the stale image was censored at the 2 h cap (BUGS F90, F76).

0 commit comments

Comments
 (0)