Visitar URL original
Benchmarks: Redis 8.10.2 documents (JSON + FT.AGGREGATE) and time series (TS.*) lanes by tae898 · Pull Request #488 · humemai/arcadedb-embedded-python · GitHub
Skip to content

Benchmarks: Redis 8.10.2 documents (JSON + FT.AGGREGATE) and time series (TS.*) lanes - #488

Merged
tae898 merged 3 commits into
mainfrom
bench/redis-docs-ts
Oct 23, 2026
Merged

Benchmarks: Redis 8.10.2 documents (JSON + FT.AGGREGATE) and time series (TS.*) lanes#488
tae898 merged 3 commits into
mainfrom
bench/redis-docs-ts

Conversation

@tae898

@tae898 tae898 commented Oct 23, 2026 •
edited
Loading

Copy link
Copy Markdown

Follow-up to #481: Redis 8.10.2 joins the documents table (RedisJSON + FT.AGGREGATE) and the time-series table (TS.*), on the same pinned image (redis@sha256:13ad9044b4b2fd2abe4c5aad8c384b0504a71006fa33f023fc68998453a2eeca), client redis==8.1.0 with protocol=2, memory and thread fit as the dense arm. Draft until the smoke table below is filled from the laptop run (see "Smoke").

What it does to existing rows

Nothing. Two new arms, redis_tpc (runner.LANES["l1tpc"], l1_tpc.BACKENDS) and redis_ts (LANES["l4"], l4_tsbs.BACKENDS), both under the existing redis engine of make_2611_stages.ENGINES. stage_drafts_2611.sha256 is rewritten (only the l1tpc and l4 stages move). Both arms are in NOT_YET_MEASURED until their first full measurement, like CrateDB.

Lanes and queries

Lane In / out Why
documents analytics (l1tpc olap) in, all five queries Q1 (all ten TPC-H columns), Q6, top parts, ship mode, revenue by month: one FT.AGGREGATE each (APPLY, GROUPBY with REDUCE SUM / AVG / COUNT, SORTBY with the lane's total order). None dropped
documents transactions (l1tpc oltp) in, with a disclosed gap new-order and payment as the vendor's optimistic transaction (WATCH, read, MULTI, JSON.SET / JSON.NUMINCRBY, EXEC): atomic and isolated, durable per class, no rollback (a failing command inside EXEC does not undo the others). The lane's transactions always commit (no abort path), so rollback is never exercised; the row says so in redis_txn_model. This lane is not official TPC-C (its docstring says "TPC-C-inspired"). A Lua script has the same property, so nothing better exists; dropping the OLTP table for Redis is one ARM_RUNS entry if you read the gap as disqualifying
the four single-record operations in single commands, atomic by themselves
time series (l4) in, all six queries q_last TS.GET; q_range TS.RANGE ... AGGREGATION max; q_global TS.MRANGE ... GROUPBY field REDUCE avg (average of per-host hourly averages; equal to the global average because every host has the same sample count per hour, and the digest checks it); q_groupby TS.MRANGE ... AGGREGATION avg; q_high TS.MRANGE ... FILTER_BY_VALUE with the next double above 90 as the (inclusive) lower bound; q_orderlimit TS.MREVRANGE ... COUNT 5 ... GROUPBY field REDUCE max
dense already in (#481)
graph, cross-model, sparse, lifecycle, restart out as in #481

Settings: ours, the existing engines', the vendor's

Setting Our value Existing engine's equivalent Vendor doc
Data model, documents one JSON document per line item / part / order / payment / row (li:<n>, part:<k>, ord:<k>, pay:<seq>, crud:<k>); the key is the id index MongoDB collections with unique id indexes https://redis.io/docs/latest/develop/data-types/json/
Load batch, documents 50,000 documents per pipeline, no MULTI MongoDB insert_many 50,000 https://redis.io/docs/latest/develop/use/pipelining/
Index, documents one FT.CREATE ... ON JSON after the load inside the index timer: l_shipdate_n (YYYYMMDD) NUMERIC, aggregated fields SORTABLE MongoDB / PostgreSQL index l_shipdate after the load https://redis.io/docs/latest/commands/ft.create/ and https://redis.io/docs/latest/develop/ai/search-and-query/advanced-concepts/aggregations/
Query timeout TIMEOUT 0 on every FT.AGGREGATE, server --search-on-timeout fail (default policy returns a partial answer after 500 ms) none needed (no engine returns partial answers on timeout) https://redis.io/docs/latest/develop/ai/search-and-query/administration/configuration/#search-on-timeout
Transactions WATCH / MULTI / EXEC, no rollback MongoDB multi-document transaction, PostgreSQL BEGIN/COMMIT https://redis.io/docs/latest/develop/using-commands/transactions/
Series layout, time series one series per (host, field), LABELS host field, vendor defaults for encoding, chunk size, retention, duplicate policy MongoDB time-series collection (metaField host) https://redis.io/docs/latest/develop/data-types/timeseries/
Ingest batch, time series TS.MADD, 50,000 points per round trip (one command per field) MongoDB insert_many 50,000 https://redis.io/docs/latest/commands/ts.madd/
maxmemory / policy 90% of the cap, noeviction Memgraph --memory-limit https://raw.githubusercontent.com/redis/redis/8.10.2/redis.conf
io-threads cpuset - 1 FalkorDB THREAD_COUNT, Memgraph Bolt workers (F6) redis.conf, THREADED I/O
search-workers (documents only) cpuset - 1 PostgreSQL parallel workers = cpuset - 1 https://redis.io/docs/latest/develop/ai/search-and-query/administration/configuration/#search-workers
Durability relaxed = image default (appendonly no, save 3600 1 300 100 60 10000); strict = --appendonly yes --appendfsync always; both read back with CONFIG GET onto the row FalkorDB, same switches https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/
Caps, cpuset, reps, warm-up, iterations, per-phase memory probes the lane's own (runner cap rule, 0-11, OLTP_WARMUP, OLAP_ITER, QITER) every arm FAIRNESS F1 to F4

Each override is on the row (redis_search_on_timeout, redis_txn_model, redis_index_docs, redis_io_threads, redis_maxmemory, ...) and in PROTOCOL.md section 7 and COMPARATORS.md.

SF10

Redis holds the data in memory under noeviction. The SF0.1 smoke (600,572 line items) put the server at 681 MiB, 1.19 KB a row (JSON document plus the SORTABLE index). Extrapolated: SF1 about 7 GB (fits the 14.4 GB maxmemory of the 16g cap), SF10 about 71 GB (against 28.8 GB at 32g). So runner.ARM_RUNS["redis_tpc"] stages the arm at tpch1 only, both workloads, both durability classes (CrateDB's entry is the model); the tpch10 cells are not queued. The extrapolation is linear, from one reading.

Smoke

Laptop smoke, one rep per cell, generated from results/runs_smoke_redis2.jsonl by smoke_table.py (each Redis row against the other engines' rows of the same lane, workload, TPC-H SF and point count; the durability column is what the adapter read back with CONFIG GET):

lane, workload, scale Redis durability compared with answer digests all equal durability read back
l1tpc olap, tpch1 SF0.01 relaxed duckdb 5 yes appendonly=no, RDB save '3600 1 300 100 60 10000' (image default): nothing is synced at commit
l1tpc oltp, tpch1 SF0.01 relaxed mongodb, sqlite 8 yes appendonly=no, RDB save '3600 1 300 100 60 10000' (image default): nothing is synced at commit
l1tpc oltp, tpch1 SF0.01 strict mongodb, sqlite 8 yes appendonly=yes, appendfsync=always: the AOF is fdatasynced at every write
l1tpc olap, tpch1 SF0.1 relaxed duckdb 5 yes appendonly=no, RDB save '3600 1 300 100 60 10000' (image default): nothing is synced at commit
l4 ingest, ts100 500000 points strict duckdb, mongodb, timescaledb 6 yes appendonly=yes, appendfsync=always: the AOF is fdatasynced at every write
l4 ingest, ts100 500000 points relaxed duckdb, mongodb, timescaledb 6 yes appendonly=no, RDB save '3600 1 300 100 60 10000' (image default): nothing is synced at commit

The first time-series smoke failed on TS.MRANGE argument order (GROUPBY should always come after filter); fixed in this PR, and the six time-series digests above are from the rerun. Bytes per row from the SF0.1 run (about 1.19 KB) put tpch10 at about 71 GB against a 32 GB cap, so redis_tpc runs tpch1 only (ARM_RUNS, as for CrateDB).

Tests

uv run --no-project --with pytest --with numpy --with pandas --with pyarrow --with requests python -m pytest -q -p no:cacheprovider . in benchmarks/experiments: 1042 passed, 24 skipped (new: test_redis_docs_ts.py, offline fake of the commands the adapters send).

🤖 Generated with Claude Code

tae898 and others added 3 commits October 23, 2026 20:36
@tae898 @claude
Benchmarks: Redis 8.10.2 documents (JSON + FT.AGGREGATE) and time ser…
b663078 
…ies (TS.*) lanes

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@tae898 @claude
Benchmarks: Redis documents and time series: stage drafts, not-yet-me…
8826bc1 
…asured roster, readback alias

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@tae898 @claude
Benchmarks: Redis TS.MRANGE argument order (FILTER before GROUPBY), d…
59de324 
…ocuments at tpch1 only (ARM_RUNS, memory measured)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@tae898
tae898 marked this pull request as ready for review October 23, 2026 14:48
@tae898
tae898 merged commit 7a8cc25 into main Oct 23, 2026
4 checks passed
@tae898
tae898 deleted the bench/redis-docs-ts branch October 23, 2026 14:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@tae898