pinot: work around STRING MIN/MAX limitation in queries.sql (apache/pinot#19603) - #2125

…d stop/start race

pinot/ had three independent bugs, found while running the real ~74GB
ClickBench hits.tsv dataset on a c6a.4xlarge (32 GiB RAM) EC2 instance:

1. Cold-cycle data loss (root cause of 0/43 query failures)
   Pinot's benchmark.sh has effectively run with BENCH_DURABLE=yes since
   the BENCH_RESTARTABLE -> BENCH_DURABLE rename in commit b282aa4 --
   the wrong setting for a system whose loaded state lives only in
   process memory. Commit b422b2d ("Remove unnecessary
   BENCH_DURABLE=yes") later deleted the explicit `export
   BENCH_DURABLE=yes` line, but that's a red herring: BENCH_DURABLE
   already defaults to `yes` in lib/benchmark-common.sh, so deleting the
   redundant explicit line changed nothing -- Pinot's effective value was
   `yes` before that commit and stayed `yes` after it.

   Pinot's own `-dataDir` persistence flag does not work as documented
   either (verified locally: throws IllegalStateException on the second
   QuickStart start, Pinot 1.5.1). Since a full CSV reload takes ~78
   minutes and can't run before every one of 43 queries, `load` now sets
   BENCH_DURABLE=no and re-pushes the existing on-disk segment .tar.gz
   files via Pinot's official Tar Push API
   (docs.pinot.apache.org/.../segment-upload) instead of re-parsing
   hits.tsv from scratch on every cold cycle.

2. Silent partial segment generation under memory pressure
   pinot-admin.sh defaults to `-Xms4G` with no `-Xmx`, so it falls back to
   the JVM's default heap ceiling (~25% of physical RAM). On the real
   74GB/100-split hits.tsv this OOMed mid-job around segment 60/100
   (OutOfMemoryError in SegmentDictionaryCreator) -- but
   LaunchDataIngestionJob still exited 0 and bench_load's >5GB data-size
   guard didn't catch it either, since 60 segments already exceeded 5GB.
   `load` now sets `JAVA_OPTS="-Xms4G -Xmx20G"` and explicitly compares
   the number of generated segment tars against the number of input
   splits, failing loudly on any mismatch instead of silently proceeding
   to the query phase with incomplete data.

3. stop/start race corrupting the next cold cycle
   `stop` only did `pkill` + a fixed `sleep 2`, but a broker can keep
   answering for 7-8s after SIGTERM while the JVM shutdown hook is still
   running, and the embedded ZooKeeper's own port teardown lags even
   further behind that. The next `start` then either treats the still-alive
   old broker as "already up" (skipping the real restart) or fails to bind
   ZooKeeper's port. `stop` now polls until the pinot-admin JVM process
   itself is fully gone (matched via `pgrep -f
   'org.apache.pinot.tools.admin.PinotAdministrator'`) instead of guessing
   from a single component's HTTP port. `check` similarly no longer treats
   broker-responsive as "fully started": QuickStart only registers its
   clean-shutdown hook, and finishes bootstrapping, after printing
   "Quick start setup complete" to pinot.log; check now greps for that
   line (scoped to the current invocation via a recorded log offset) in
   addition to the broker health check, so `stop` is never sent while
   Pinot is still mid-startup.

Verified end-to-end on EC2 (c6a.4xlarge, real 74GB hits.tsv, 100
segments): 37/43 queries now pass across all 3 cold-cycle trials (up from
0/43). The remaining 6 failures are a separate, pre-existing Apache Pinot
limitation (standard MIN()/MAX() rejects STRING columns; reported
upstream at apache/pinot#19603) unrelated to this benchmark-driver fix.