Visitar URL original
Rename BENCH_RESTARTABLE -> BENCH_DURABLE; reload non-durable between… · ClickHouse/ClickBench@b282aa4 · GitHub
Skip to content

Commit b282aa4

Browse files
alexey-milovidovclaude
alexey-milovidov
and
claude
committed
Rename BENCH_RESTARTABLE -> BENCH_DURABLE; reload non-durable between cold tries
Semantic shift: the old flag conflated "does this system have a daemon to restart" with "does the data survive a restart". The new BENCH_DURABLE asks the second question directly, and the driver acts on it: BENCH_DURABLE=yes (default) — data is on disk (daemons like clickhouse, or CLI tools like duckdb operating on a .db file). Cold cycle is the existing stop -> wait_stopped -> drop_caches -> start -> check. Each ./query then runs 3 times: first cold, two warm. BENCH_DURABLE=no — data lives in process memory (in-process servers: pandas, polars, duckdb-dataframe, duckdb-memory, chdb-dataframe, daft-*, sirius). The restart wipes it, so after the start+check we re-run ./load and roll its wall-clock into the first ("cold") try. The recorded cold timing is now load+query, which is the honest cost of "fresh-start engine + first query" instead of "warm query against a freshly-loaded RAM dataset". Migrations: * All 108 benchmark.sh shims renamed BENCH_RESTARTABLE -> BENCH_DURABLE. * 9 in-process server shims set DURABLE=no (chdb-dataframe, daft-parquet, daft-parquet-partitioned, duckdb-dataframe, duckdb-memory, pandas, polars, polars-dataframe, sirius). * Previously RESTARTABLE=no CLI systems (duckdb, sqlite, datafusion, hyper, octosql, opteryx, etc.) become DURABLE=yes — they're stateless per-process so the cold cycle is essentially drop_caches + a no-op stop/start. * 6 in-process load scripts (chdb-dataframe, duckdb-dataframe, duckdb-memory, pandas, polars-dataframe, sirius) used to rm hits.parquet right after the first load — drop that, since bench_run_query now needs the source file to reload after each restart. Moved the cleanup into bench_main (DURABLE=no branch only). * Back-compat: BENCH_RESTARTABLE is still read as an alias if BENCH_DURABLE isn't set, so stale env/scripts keep working for one cycle. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent c7b6c4e commit b282aa4

116 files changed

Lines changed: 174 additions & 139 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎arc/benchmark.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
#!/bin/bash
22
# Thin shim — actual flow is in lib/benchmark-common.sh.
33
export BENCH_DOWNLOAD_SCRIPT="download-hits-parquet-single"
4-
export BENCH_RESTARTABLE=yes
4+
export BENCH_DURABLE=yes
55
exec ../lib/benchmark-common.sh

‎byconity/benchmark.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
#!/bin/bash
22
# Thin shim — actual flow is in lib/benchmark-common.sh.
33
export BENCH_DOWNLOAD_SCRIPT="download-hits-tsv"
4-
export BENCH_RESTARTABLE=yes
4+
export BENCH_DURABLE=yes
55
# byconity boots a chain of containers (fdb -> tso -> server -> workers
66
# / daemon-manager). Each later step waits up to 600s for its
77
# dependency, so the worst-case cold start is several minutes; the

‎byconity/hdfs/create_users.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
#!/bin/bash
22
set -e -x
33

4-
# Idempotent (-p), so re-running across BENCH_RESTARTABLE iterations
4+
# Idempotent (-p), so re-running across BENCH_DURABLE restart iterations
55
# doesn't fail on "File exists" once the dirs are set up.
66
docker exec hdfs-namenode hdfs dfs -mkdir -p /user/clickhouse
77
docker exec hdfs-namenode hdfs dfs -chown clickhouse /user/clickhouse

‎cedardb-parquet/benchmark.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
#!/bin/bash
22
# Thin shim — actual flow is in lib/benchmark-common.sh.
33
export BENCH_DOWNLOAD_SCRIPT="download-hits-parquet-single"
4-
export BENCH_RESTARTABLE=yes
4+
export BENCH_DURABLE=yes
55
exec ../lib/benchmark-common.sh

‎cedardb/benchmark.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
#!/bin/bash
22
# Thin shim — actual flow is in lib/benchmark-common.sh.
33
export BENCH_DOWNLOAD_SCRIPT="download-hits-tsv"
4-
export BENCH_RESTARTABLE=yes
4+
export BENCH_DURABLE=yes
55
exec ../lib/benchmark-common.sh

‎chdb-dataframe/benchmark.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
#!/bin/bash
22
# Thin shim — actual flow is in lib/benchmark-common.sh.
33
export BENCH_DOWNLOAD_SCRIPT="download-hits-parquet-single"
4-
export BENCH_RESTARTABLE=no
4+
export BENCH_DURABLE=no
55
exec ../lib/benchmark-common.sh

‎chdb-dataframe/load‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,5 +22,4 @@ elapsed=$(printf '%s' "$body" | python3 -c \
2222

2323
echo "Load (server-reported): ${elapsed}s"
2424

25-
rm -f hits.parquet
2625
sync

‎chdb-parquet-partitioned/benchmark.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
#!/bin/bash
22
# Thin shim — actual flow is in lib/benchmark-common.sh.
33
export BENCH_DOWNLOAD_SCRIPT="download-hits-parquet-partitioned"
4-
export BENCH_RESTARTABLE=no
4+
export BENCH_DURABLE=yes
55
exec ../lib/benchmark-common.sh

‎chdb/benchmark.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
#!/bin/bash
22
# Thin shim — actual flow is in lib/benchmark-common.sh.
33
export BENCH_DOWNLOAD_SCRIPT="download-hits-csv"
4-
export BENCH_RESTARTABLE=no
4+
export BENCH_DURABLE=yes
55
exec ../lib/benchmark-common.sh

‎chyt/benchmark.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,5 +4,5 @@
44
export YT_USE_HOSTS=0
55
export CHYT_ALIAS="${CHYT_ALIAS:-*ch_public}"
66
export BENCH_DOWNLOAD_SCRIPT=""
7-
export BENCH_RESTARTABLE=no
7+
export BENCH_DURABLE=yes
88
exec ../lib/benchmark-common.sh

0 commit comments

Comments
 (0)