Visitar URL original
bench: MongoDB 9.0.4 + mongot 1.70.5 pin (8.3.13 still cannot boot; 8… · humemai/arcadedb-embedded-python@0da87f7 · GitHub
Skip to content

Commit 0da87f7

Browse files
tae898claude
andauthored
bench: MongoDB 9.0.4 + mongot 1.70.5 pin (8.3.13 still cannot boot; 8.2 unsupported for Search) (#484)
* bench: MongoDB pin 8.2.12 -> 9.0.4, mongot 1.70.4 -> 1.70.5 (hold on 8.3 stays: 8.3.13 cannot boot) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * bench: MongoDB pin follow-ups: mark 8.2.12 values as history, journalCommitInterval re-read on 9.0.4; rewrite stage_drafts_2611.sha256 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
1 parent 5ec2a0b commit 0da87f7

13 files changed

Lines changed: 101 additions & 88 deletions

‎benchmarks/experiments/COMPARATORS.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -22,8 +22,8 @@ Which arms are on the October page, with their rows and columns, is PAGE-SPEC.md
2222
| QuestDB | `questdb/questdb@sha256:931af415…` | 10.0.1 | time series | served | InfluxDB line protocol over TCP on 9009. QuestDB 10 adds QWP, a binary columnar protocol over WebSocket on the HTTP port, beside the line protocol; `line.tcp.enabled` still defaults to true on 0.0.0.0:9009 in 10.x and only the UDP receiver is deprecated, so this lane keeps the path it has always measured. Moving to QWP would be a new measurement, not a version bump. |
2323
| SurrealDB (embedded) | `surrealdb==2.0.0` (Python SDK, in-process) | core 2.3.10, SDK 2.0.0 | cross-model, documents (TPC), graph, dense, time series (plain table), lifecycle (2026-09-16, DECISIONS #95b: its own SurrealKV store on disk, every situation in SurrealQL, the HNSW index for the vector situation; sparse vectors and the analytical view declared with the parse error captured verbatim, and no JVM start on its rows) | embedded, SurrealKV on disk | SDK `insert()` batches, bulk relation insert for edges. Time series (2026-09-15): no time-series type, so a SCHEMALESS table with a datetime `ts` and `DEFINE INDEX p_host_ts ON p FIELDS host, ts` defined before the load, SQLite's footing; buckets by `time::floor(ts, 1m)` / `time::floor(ts, 1h)`; the grouped-ordered-limited query is a subquery because core 2.3.10 sorts by the group key ascending after `GROUP BY` (BUGS F31). |
2424
| SurrealDB (served) | `surrealdb/surrealdb@sha256:6a500236…` (`v3.2.4`) | 3.2.4, RocksDB storage | documents (TPC), graph, dense, cross-model, time series (plain table) | served | SDK `insert()` and bulk relation insert over ws. Time series (2026-09-15): the same table, index and SurrealQL as the embedded arm, over ws through the reconnecting client (DECISIONS #91). Dense and cross-model: the HNSW index is defined before the load, which the 3.2.4 server fills in the background, so the build waits inside its index timer until a KNN probe's latency stops falling (`surreal_common.await_hnsw_settled`, recorded as `settle_s`; BUGS F134, DECISIONS #117). |
25-
| MongoDB | `mongo@sha256:41afd6e1…` (`mongo:8.2.12`) | 8.2.12 | documents (TPC), time series (native time-series collection), graph | served | insert_many batches; single-node replica set, because TPC-C new-order needs a multi-document transaction. Client `pymongo==4.18.1`. Held below 8.3.11, which cannot boot on the bench kernel (SERVER-121912); 8.2.12 is the latest measurable MongoDB (the held-below list). The graph arm runs the SAME digest. Every hop and every analytics query is a chained `$lookup`, never `$graphLookup` (PROTOCOL.md section 7). The loader refuses a self-loop, on which the undirected reads depend. The write looks up its anchor person first and writes nothing when it is absent, as the Cypher's MATCH does (until 2026-10-02 it inserted unconditionally, which only a capped slice could show). Every graph question, LSQB's nine included, is expressed as an aggregation pipeline and agrees on the slice (DECISIONS #104b). |
26-
| MongoDB + MongoDB Search | built image `dbbench:mongo-search` (Dockerfile.mongosearch) FROM `mongo@sha256:41afd6e1…` + `mongodb/mongodb-community-search@sha256:24c0cd30…`; both digests pinned in the Dockerfile | mongod 8.2.12 + mongot-community 1.70.4 | dense vectors, cross-model | served, one container holding both processes | `$vectorSearch` over a `vectorSearch` index at the lane's matched point (PROTOCOL.md section 7). The vector field's `quantization` is `none` on `mongodb_dense` and `scalar` on `mongodb_dense_int8` (DECISIONS #135, from the 26.10.1 measurement), read back from mongot's index definition onto the row (`mongot_vector_quantization`). mongod runs WITH access control, unlike the document arm above (PROTOCOL.md section 7). Not `mongodb/mongodb-atlas-local`: MongoDB documents that image as "For evaluation only" and "not suitable for production use", and its vector-capable variant is a `:preview` tag. |
25+
| MongoDB | `mongo@sha256:b609cdfb…` (`mongo:9.0.4`) | 9.0.4 | documents (TPC), time series (native time-series collection), graph | served | insert_many batches; single-node replica set, because TPC-C new-order needs a multi-document transaction. Client `pymongo==4.18.1`. 9.0.4 from 2026-10-23; it was held at 8.2.12 while 8.3.x could not boot on the bench kernel (SERVER-121912), and 8.3.13 still cannot (the held-below list). Rows measured before the bump read 8.2.12 and are re-measured, because one engine wears one version per page. The graph arm runs the SAME digest. Every hop and every analytics query is a chained `$lookup`, never `$graphLookup` (PROTOCOL.md section 7). The loader refuses a self-loop, on which the undirected reads depend. The write looks up its anchor person first and writes nothing when it is absent, as the Cypher's MATCH does (until 2026-10-02 it inserted unconditionally, which only a capped slice could show). Every graph question, LSQB's nine included, is expressed as an aggregation pipeline and agrees on the slice (DECISIONS #104b). |
26+
| MongoDB + MongoDB Search | built image `dbbench:mongo-search` (Dockerfile.mongosearch) FROM `mongo@sha256:b609cdfb…` + `mongodb/mongodb-community-search@sha256:6232a8ac…`; both digests pinned in the Dockerfile | mongod 9.0.4 + mongot-community 1.70.5 | dense vectors, cross-model | served, one container holding both processes | `$vectorSearch` over a `vectorSearch` index at the lane's matched point (PROTOCOL.md section 7). The vector field's `quantization` is `none` on `mongodb_dense` and `scalar` on `mongodb_dense_int8` (DECISIONS #135, from the 26.10.1 measurement), read back from mongot's index definition onto the row (`mongot_vector_quantization`). mongod runs WITH access control, unlike the document arm above (PROTOCOL.md section 7). Not `mongodb/mongodb-atlas-local`: MongoDB documents that image as "For evaluation only" and "not suitable for production use", and its vector-capable variant is a `:preview` tag. |
2727
| pgvector | `pgvector/pgvector@sha256:1d50c689…` (`0.8.6-pg18`) | PostgreSQL 18 + pgvector 0.8.6 | dense (`vector`, HNSW m=16, ef_construction=100, ef_search=100), sparse (`sparsevec`, HNSW, inner product) | served | COPY, then HNSW build. Memory and pools fitted to the cell (PROTOCOL.md section 7, FAIRNESS F6). |
2828
| TimescaleDB | `timescale/timescaledb@sha256:f7036933…` (`2.30.1-pg18`) | 2.30.1 on PostgreSQL 18 | time series (hypertable, `time_bucket`) | served | COPY into a hypertable, index on (host, ts). One of TSBS's home engines. Pinned at 2.30.1 and never 2.30.0: 2.30.0 introduced the `DeferredChunkAppend` planner node and 2.30.1 fixes that node producing DUPLICATE ROWS, which is an answer bug on this lane's own query shapes. Plain PostgreSQL runs beside it on the same image family (`postgres_ts`: COPY into a plain table, the same (host, ts DESC) index, `date_trunc` buckets), so the table shows what the extension buys. |
2929
| Neo4j | `neo4j@sha256:e702d6b5…` (`2026.08.1-community`) | 2026.08.1 | graph, dense (its vector index, HNSW), cross-model | served | UNWIND batches over bolt. Dense queries use the Cypher 25 `SEARCH` clause; it has no candidate-count option and `LIMIT k` alone gave recall 0.50 at micro scale, so the adapter asks for `ef_search` (100) candidates and keeps the best k, the operating point every other engine runs at. Its vector index's quantization is set, never left to the default: 2026.08.1 builds BINARY (with a 3x search expansion) when the definition names none, which ours did through October (BUGS F164, DECISIONS #135); from the 26.10.1 measurement the fp32 arms set `NONE`, a `neo4j_dense_int8` arm sets `SCALAR`, and both read the applied type back onto the row (`neo4j_vector_quantization`). |
@@ -51,7 +51,7 @@ Pins follow DECISIONS #87 (the latest self-hosted stable release at campaign tim
5151
| pgvector | 0.8.6-pg18 | `sha256:1d50c689b0a6…` |
5252
| PostgreSQL + pgvector + AGE | PG18 + AGE 1.8.0 | built image, base as pgvector |
5353
| TimescaleDB | 2.30.1-pg18 | `sha256:f7036933154c…` |
54-
| MongoDB Search (mongot) | 1.70.4 | `sha256:24c0cd30baf0…` |
54+
| MongoDB Search (mongot) | 1.70.5 | `sha256:6232a8acec5e…` |
5555
| Neo4j | 2026.08.1-community | `sha256:e702d6b535d9…` |
5656
| Qdrant | v1.19.1 | `sha256:0699e7733a6f…` |
5757
| Milvus | v3.0.3 | `sha256:812cb91ad04e…` (etcd v3.5.25 `sha256:dc2bdc588d2a…`, MinIO RELEASE.2024-12-18T13-15-44Z `sha256:34c8e2f52a59…`) |
@@ -69,7 +69,7 @@ Pins follow DECISIONS #87 (the latest self-hosted stable release at campaign tim
6969

7070
**Held below the newest, and why.**
7171

72-
- **MongoDB 8.2.12**, not 8.3.11 (the one engine where #103d overrode #87): 8.3.11 refuses to start on the bench kernel (SERVER-121912), and an older 8.3.x that merely lacks the warning would run a configuration its vendor calls unsafe. Every MongoDB arm, the mongod half of the Search image included, holds at 8.2.12.
72+
- **MongoDB 8.3.x is skipped, and 9.0.4 is the pin** (the one engine where #103d overrode #87; hold lifted 2026-10-23). 8.3.13, the newest 8.3 patch (Docker Hub 2026-10-17), still refuses to start on the bench kernel with the SERVER-121912 guard ("Linux kernel versions 6.19 and newer has a known incompatibility"), both as plain `mongo` and inside the Search image, with mongot 1.70.4 and with 1.70.5. 9.0.4 (Docker Hub 2026-10-17, amd64 manifest `sha256:b609cdfba3a8…`; 9.0.0 shipped 2026-09-28 and the release notes call 9.0 the current stable release) starts on the same kernel. Why it starts was not investigated: SERVER-125742 exempts kernels from upstream 7.0.14 but reads `uname -r`, which Ubuntu keeps at 7.0.0 (open as SERVER-136848, reported against 8.3.11 and 9.0.2); the bench kernel 7.0.0-38-generic is upstream 7.0.14, so it carries the userspace fix the guard exists for, and the old argument that an older 8.3.x lacking the warning would run a configuration its vendor calls unsafe no longer holds. Why not stay on 8.2.12, which also boots: MongoDB's compatibility table for MongoDB Search lists mongot 1.70.1 and later against MongoDB Server 8.3 only and says 8.2 "is no longer a supported source deployment for MongoDB Search", so 8.2.12 + mongot 1.70.4 was a pairing the vendor no longer supports. 9.0 is not in that table either: 9.0.4 + mongot 1.70.5 is exercised, not documented. Laptop test 2026-10-23, kernel 7.0.0-38-generic, containers on cores 12-15 with `-m 3g`/`-m 4g`, `pymongo==4.18.1`, through the repo's own `l1_tpc.MongoTPC` on 30,000 seeded synthetic lineitem rows and `mongo_common` on 3,000 seeded 32-d vectors: 8.2.12 boots, 8.3.13 does not (also with mongot 1.70.4 and 1.70.5), 9.0.4 boots; on 9.0.4 against 8.2.12 the five analytics pipelines, 200 committed new-order transactions, 50 payments, the abort (nothing changed), the write-conflict error (code 112), `with_transaction` and 100 single-record CRUD operations gave identical digests; `$vectorSearch` through mongot 1.70.5 returned the same top-10 lists as 8.2.12 + 1.70.4 on 99.3% of 300 queries (mean set overlap 0.9993, recall 1.000 against exact search) for `quantization: none`, and 0.960 mean set overlap for `scalar` (recall 0.878 against 0.873, a quantized index is not deterministic across builds); both indexes read their quantization back, and a `$vectorSearch` inside a multi-document transaction is still refused. Not tested: performance (every MongoDB row has to be re-measured on 9.0.4), a full TPC-H scale, restart/recovery, the time-series and graph lanes. mini runs kernel 7.0.0-31-generic, the same 7.0 line; a one-minute `docker run --rm mongo@sha256:b609cdfb… mongod --version`-style start check there is a sanity step, not a requirement.
7373
- **DuckDB and DuckPGQ 1.5.4**, not 1.5.5: the community-extensions registry has no DuckPGQ build for 1.5.5 or 1.6.0 (DECISIONS #103d, #103e).
7474
- **SurrealDB server v3.2.4 and SDK 2.0.0**: the only higher numbers (`v3.3.0-beta.*`, `3.0.0b8`) are pre-releases.
7575
- **Chroma 1.5.9, sqlite-vec 0.1.9, Memgraph 3.13.1, pymongo 4.18.1, python-arango 8.3.5, falkordb 1.7.1, redis 8.1.0**: the newest stable releases (sqlite-vec's `0.1.10a4` is a pre-release). **pymilvus 3.0.2** (with the `bulk_writer` extra, which brings `minio`, `azure-storage-blob` and `ml-dtypes`) is what Milvus's compatibility table names for a 3.0.3 server.

‎benchmarks/experiments/Dockerfile.mongosearch‎

Lines changed: 21 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,23 +1,25 @@
1-
# MongoDB Community 8.2.12 + MongoDB Search (mongot) Community 1.70.4 in one
1+
# MongoDB Community 9.0.4 + MongoDB Search (mongot) Community 1.70.5 in one
22
# container: the self-hosted, generally available way to run $vectorSearch.
33
#
4-
# mongod DID NOT MOVE IN THE OCTOBER RE-PIN, and this is the one engine where
5-
# DECISIONS #103d overrode #87. 8.3.11 is the newer release and it cannot be
6-
# measured on the bench kernel: it refuses to boot with "Linux kernel versions
7-
# 6.19 and newer has a known incompatibility with this version of MongoDB"
8-
# (SERVER-121912), the same guard that moved this project off 8.0. Reproduced
9-
# on the laptop at kernel 7.0.0-29-generic on 2026-09-19; 8.2.12 boots on the
10-
# same kernel. The reasoning is in runner.py beside the "mongodb" backend.
11-
# It is still the SAME digest every other MongoDB arm runs, so the vector rows
12-
# and the document rows come from one engine build.
4+
# THE OCTOBER HOLD AT mongod 8.2.12 / mongot 1.70.4 WAS LIFTED ON 2026-10-23.
5+
# 8.3.13 still refuses to boot on the bench kernel (SERVER-121912, the
6+
# kernel-6.19 guard; SERVER-136848 is the open ticket for Ubuntu kernels whose
7+
# `uname -r` stays 7.0.0 although they are upstream 7.0.14), with mongot 1.70.4
8+
# or 1.70.5 alike. 9.0.4 boots and answered the repo's TPC adapter checks like
9+
# 8.2.12 and a $vectorSearch run through mongot 1.70.5 with recall 1.000 (fp32)
10+
# and 0.878 (scalar) against exact search. MongoDB's own compatibility table for
11+
# Search lists mongot 1.70.1+ against MongoDB Server 8.3 only and calls 8.2
12+
# "no longer a supported source deployment", so the old pairing was outside
13+
# vendor support; 9.0 is also not in the table, which is the one gap left.
14+
# The reasoning and the check table are in COMPARATORS.md, "Held below the
15+
# newest, and why". It is still the SAME mongod digest every other MongoDB arm
16+
# runs, so the vector rows and the document rows come from one engine build.
1317
#
14-
# mongot's VERSION did not move -- 1.70.4 is still the newest MongoDB Search
15-
# Community release -- but its DIGEST did, because the 1.70.4 tag was rebuilt
16-
# on 2026-09-16. The old pin was also the wrong KIND of digest: 63ebc805 is
17-
# the multi-arch INDEX digest, not the amd64 manifest digest this file and
18-
# COMPARATORS.md say every pin is. Both forms resolve, so nothing was ever
19-
# wrong on the wire; the pin is normalised here to the amd64 manifest digest
20-
# so the convention the docs state is the convention the file keeps.
18+
# mongot 1.70.5 (2026-09-29) is the newest Search Community release: security
19+
# dependency upgrades only, and its default image moved to a distroless UBI9
20+
# base; /mongot-community is laid out as before, so the COPY below is unchanged.
21+
# Both digests are the amd64 MANIFEST digest (not the multi-arch index digest),
22+
# the convention COMPARATORS.md states for every pin.
2123
#
2224
# WHY THIS AND NOT mongodb/mongodb-atlas-local. atlas-local is the same two
2325
# processes in one image and it would have been less work, but MongoDB's own
@@ -39,8 +41,8 @@
3941
# document rows come from one engine build rather than two.
4042
#
4143
# Built as dbbench:mongo-search by build_images.sh.
42-
FROM mongodb/mongodb-community-search@sha256:24c0cd30baf06c93607f3dbd4816587036996de347f5f11ef47cf998cf4e898c AS mongot
43-
FROM mongo@sha256:41afd6e1183f57e4e4d03ab733070671fca8553da2b36f15d6e3fc9760494d17
44+
FROM mongodb/mongodb-community-search@sha256:6232a8acec5ea4cce2cf52fa7c17b57da5758eac11538ffc35dab43abde8d366 AS mongot
45+
FROM mongo@sha256:b609cdfba3a865331378cc4158bc52522a3bf176d59a7ff2c43fab52f5d9d5cb
4446
# mongot bundles its own jlink'd JDK under bin/jdk, so the whole directory is
4547
# self-contained and nothing has to be installed into the mongod image.
4648
COPY --from=mongot /mongot-community /mongot-community

‎benchmarks/experiments/PROTOCOL.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -238,7 +238,7 @@ An override nobody can find is a defect, whichever way it moves the number. Ever
238238
| pgvector (dense) and PG+AGE (cross-model) | the dense and product loads use binary COPY with pgvector-python's types (`register_vector`, `cp.set_types([...])`, numpy rows), in place of text COPY with each float formatted in Python inside the ingest timer. The sparse pgvector load keeps text COPY (2026-10, DECISIONS #174) | client path (the vendor's documented path), not an override | The pgvector README's bulk-loading path; the text path was client-bound inside the ingest timer. Only the ingest column moves, and it shrinks ArcadeDB's ingest lead over pgvector. `pgvector==0.5.0` is pinned in the `client` image. | adapter comments in `l3d_dense.PgVector` and `e2_hybrid.PgAgeE2`, this table; CAMPAIGN.md section 7 row 76 |
239239
| QuestDB (L4) | the six read queries run on a binary-format cursor (`binary=True`) (2026-10, DECISIONS #174) | client path (the vendor's documented path), not an override | The QuestDB documentation recommends `binary=True`; answers are identical in both modes. Ingest stays on the raw-TCP line protocol. | adapter comment in `l4_tsbs.QuestTS._q`, this table |
240240
| MongoDB (documents, time series) | timed writes at write concern `w=1, j=false` (2026-10) | operating-point (durability class) | A replica set's default is `w:majority` with journaling, which waits for the disk | row field `durability`, FAIRNESS F10 |
241-
| MongoDB Search (dense, cross-model) | the server is `dbbench:mongo-search`, mongod 8.2.12 plus mongot-community 1.70.4 in ONE container, and mongod runs WITH access control while the document arm runs without it (2026-10) | escape hatch (forced) | mongot authenticates to its sync source with SCRAM through the `searchCoordinator` role, so it cannot run against an open mongod. One container, so the whole engine lands in the one cgroup the cell caps and samples. | row field `mongo_access_control`, COMPARATORS.md |
241+
| MongoDB Search (dense, cross-model) | the server is `dbbench:mongo-search`, mongod 9.0.4 plus mongot-community 1.70.5 in ONE container, and mongod runs WITH access control while the document arm runs without it (2026-10) | escape hatch (forced) | mongot authenticates to its sync source with SCRAM through the `searchCoordinator` role, so it cannot run against an open mongod. One container, so the whole engine lands in the one cgroup the cell caps and samples. | row field `mongo_access_control`, COMPARATORS.md |
242242
| MongoDB Search (dense) | `hnswOptions.maxEdges`=16, `numEdgeCandidates`=100, stage `numCandidates`=100, passed explicitly although they are also mongot's defaults | operating-point | The lane's matched point, set rather than inherited, so a future default change cannot move this arm without anyone noticing; the created index is read back onto the row | row fields `degree_param`, `degree_family`, `mongot_hnsw_options` |
243243
| MongoDB Search (dense) | `index_s` carries a fixed 12-30 s floor that is mongot noticing the index, not building it (measured: a one-document index becomes queryable in 12.51/29.63/30.12/30.10 s, and 1,000 and 20,000 documents take the same 28-30 s) | **disclosure, not an override** | At micro and tiny the floor IS the number, so this arm's small-tier build timers are not comparable with another engine's. Not subtracted: an adapter that removed a constant from its own engine's timer would be doing the comparator's arithmetic for it | adapter docstring, this table |
244244
| MongoDB (graph) | multi-hop reads use chained `$lookup` (two indexed lookups per hop, on `src` and on `dst`) rather than `$graphLookup`, although $graphLookup was 30.01 ms against 38.27 ms per three-hop operation on the directed question at micro | operating-point, runs AGAINST the comparator | DECISIONS #131 item 2 prefers `$graphLookup` where it expresses the question. It walks one direction, and every graph question is asked undirected over one stored edge per friendship (CAMPAIGN section 7 row 56), so it would need a symmetric second copy of the edges. No timed query uses it. | adapter docstring, COMPARATOR-DIALECTS.md |

0 commit comments

Comments
 (0)