Skip to content

Tag the remaining systems that do not restart before a cold run - #1648

Open
toschmidt wants to merge 1 commit into
ClickHouse:mainfrom
toschmidt:schmidt/missing-cold
Open

Tag the remaining systems that do not restart before a cold run#1648
toschmidt wants to merge 1 commit into
ClickHouse:mainfrom
toschmidt:schmidt/missing-cold

Conversation

@toschmidt

Copy link
Copy Markdown
Contributor

df35291 brought the classification up to date but only added the tag to the five managed services it looked at (ClickHouse Cloud, AlloyDB, Hologres, Databricks, MotherDuck). The same argument applies to every other entry whose first try of each query hits a live engine, and those were left untagged. README section 2.b makes the tag mandatory for submissions that do not restart the database, so add it to them.

Remote services driven by a hand-written run.sh that neither restarts anything server-side nor clears any cache. Tag every result file:

  • athena (1), athena-partitioned (1): start-query-execution only.
  • aurora-mysql (1), aurora-postgresql (1): plain psql/mysql loops.
  • bigquery (2): --use_cache=false disables the result cache, which section 2.b does not accept as a substitute for a restart.
  • bytehouse (8): bytehouse-cli against a remote warehouse.
  • chyt (4, one was already tagged): BENCH_RESTARTABLE=no, and ./stop is exit 0 because the YT cluster is remote.
  • crunchy-bridge-for-analytics (1): turns the cache manager off for the first pass, but the server keeps running - the same situation as Hologres' freecache call, which df35291 tagged.
  • hydra (2, the serverless run was already tagged): benchmark.sh loads and then calls run.sh, with no restart and no drop_caches anywhere in between.
  • redshift (6), redshift-serverless (2): only enable_result_cache_for_session = off.
  • singlestore (4): template.json already carried the tag as the one system df35291 kept it on, but its result files never did.
  • snowflake (27), timescale-cloud (27), tinybird (1).

Local daemons that the shared driver deliberately does not restart. lib/benchmark-common.sh only runs the ./stop -> drop_caches -> ./start -> ./check cycle when BENCH_RESTARTABLE=yes; these four set it to no and fall through to bench_flush_caches alone, so the page cache is dropped while the engine keeps its internal caches. Each has a reason recorded in its own benchmark.sh: restarting catalogd would empty Impala's in-memory catalog, and firebolt-core keeps the loaded database in its bind-mounted volume. Tag the result files and template.json so future runs stay flagged:

  • impala (5 + template.json)
  • firebolt (50 + template.json; the 13 managed-cloud runs from 2025-06-23 onwards were already tagged, the two historical 2025-06-07 entries were not)
  • firebolt-parquet (33 + template.json)
  • firebolt-parquet-partitioned (33 + template.json)

The other systems that set BENCH_RESTARTABLE=no are genuine embedded CLIs whose ./start is exit 0, so they satisfy section 2.a implicitly and are left alone. That includes zighouse, which does have a server but whose ./query invokes the binary directly instead.

df35291 brought the classification up to date but only added the tag
to the five managed services it looked at (ClickHouse Cloud, AlloyDB,
Hologres, Databricks, MotherDuck). The same argument applies to every
other entry whose first try of each query hits a live engine, and those
were left untagged. README section 2.b makes the tag mandatory for
submissions that do not restart the database, so add it to them.

Remote services driven by a hand-written run.sh that neither restarts
anything server-side nor clears any cache. Tag every result file:

  - athena (1), athena-partitioned (1): start-query-execution only.
  - aurora-mysql (1), aurora-postgresql (1): plain psql/mysql loops.
  - bigquery (2): --use_cache=false disables the result cache, which
    section 2.b does not accept as a substitute for a restart.
  - bytehouse (8): bytehouse-cli against a remote warehouse.
  - chyt (4, one was already tagged): BENCH_RESTARTABLE=no, and ./stop
    is `exit 0` because the YT cluster is remote.
  - crunchy-bridge-for-analytics (1): turns the cache manager off for
    the first pass, but the server keeps running - the same situation
    as Hologres' freecache call, which df35291 tagged.
  - hydra (2, the serverless run was already tagged): benchmark.sh
    loads and then calls run.sh, with no restart and no drop_caches
    anywhere in between.
  - redshift (6), redshift-serverless (2): only
    enable_result_cache_for_session = off.
  - singlestore (4): template.json already carried the tag as the one
    system df35291 kept it on, but its result files never did.
  - snowflake (27), timescale-cloud (27), tinybird (1).

Local daemons that the shared driver deliberately does not restart.
lib/benchmark-common.sh only runs the ./stop -> drop_caches -> ./start
-> ./check cycle when BENCH_RESTARTABLE=yes; these four set it to no
and fall through to bench_flush_caches alone, so the page cache is
dropped while the engine keeps its internal caches. Each has a reason
recorded in its own benchmark.sh: restarting catalogd would empty
Impala's in-memory catalog, and firebolt-core keeps the loaded database
in its bind-mounted volume. Tag the result files and template.json so
future runs stay flagged:

  - impala (5 + template.json)
  - firebolt (50 + template.json; the 13 managed-cloud runs from
    2025-06-23 onwards were already tagged, the two historical
    2025-06-07 entries were not)
  - firebolt-parquet (33 + template.json)
  - firebolt-parquet-partitioned (33 + template.json)

The other systems that set BENCH_RESTARTABLE=no are genuine embedded
CLIs whose ./start is `exit 0`, so they satisfy section 2.a implicitly
and are left alone. That includes zighouse, which does have a server
but whose ./query invokes the binary directly instead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@toschmidt
toschmidt requested a deployment to benchmark-approval August 28, 2026 16:35 — with GitHub Actions Waiting
@toschmidt

Copy link
Copy Markdown
Contributor Author

@alexey-milovidov #1646 missed some no-cold systems like redshift, snowflake. Also firebolt seems to do a lukewarm could run, as they start a docker container and mark their run with restartable=no

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant