Tag the remaining systems that do not restart before a cold run - #1648
Open
toschmidt wants to merge 1 commit into
Open
Tag the remaining systems that do not restart before a cold run#1648toschmidt wants to merge 1 commit into
toschmidt wants to merge 1 commit into
Conversation
df35291 brought the classification up to date but only added the tag to the five managed services it looked at (ClickHouse Cloud, AlloyDB, Hologres, Databricks, MotherDuck). The same argument applies to every other entry whose first try of each query hits a live engine, and those were left untagged. README section 2.b makes the tag mandatory for submissions that do not restart the database, so add it to them. Remote services driven by a hand-written run.sh that neither restarts anything server-side nor clears any cache. Tag every result file: - athena (1), athena-partitioned (1): start-query-execution only. - aurora-mysql (1), aurora-postgresql (1): plain psql/mysql loops. - bigquery (2): --use_cache=false disables the result cache, which section 2.b does not accept as a substitute for a restart. - bytehouse (8): bytehouse-cli against a remote warehouse. - chyt (4, one was already tagged): BENCH_RESTARTABLE=no, and ./stop is `exit 0` because the YT cluster is remote. - crunchy-bridge-for-analytics (1): turns the cache manager off for the first pass, but the server keeps running - the same situation as Hologres' freecache call, which df35291 tagged. - hydra (2, the serverless run was already tagged): benchmark.sh loads and then calls run.sh, with no restart and no drop_caches anywhere in between. - redshift (6), redshift-serverless (2): only enable_result_cache_for_session = off. - singlestore (4): template.json already carried the tag as the one system df35291 kept it on, but its result files never did. - snowflake (27), timescale-cloud (27), tinybird (1). Local daemons that the shared driver deliberately does not restart. lib/benchmark-common.sh only runs the ./stop -> drop_caches -> ./start -> ./check cycle when BENCH_RESTARTABLE=yes; these four set it to no and fall through to bench_flush_caches alone, so the page cache is dropped while the engine keeps its internal caches. Each has a reason recorded in its own benchmark.sh: restarting catalogd would empty Impala's in-memory catalog, and firebolt-core keeps the loaded database in its bind-mounted volume. Tag the result files and template.json so future runs stay flagged: - impala (5 + template.json) - firebolt (50 + template.json; the 13 managed-cloud runs from 2025-06-23 onwards were already tagged, the two historical 2025-06-07 entries were not) - firebolt-parquet (33 + template.json) - firebolt-parquet-partitioned (33 + template.json) The other systems that set BENCH_RESTARTABLE=no are genuine embedded CLIs whose ./start is `exit 0`, so they satisfy section 2.a implicitly and are left alone. That includes zighouse, which does have a server but whose ./query invokes the binary directly instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
toschmidt
requested a deployment
to
benchmark-approval
August 28, 2026 16:35 — with
GitHub Actions
Waiting
Contributor
Author
|
@alexey-milovidov #1646 missed some no-cold systems like redshift, snowflake. Also firebolt seems to do a lukewarm could run, as they start a docker container and mark their run with restartable=no |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
df35291 brought the classification up to date but only added the tag to the five managed services it looked at (ClickHouse Cloud, AlloyDB, Hologres, Databricks, MotherDuck). The same argument applies to every other entry whose first try of each query hits a live engine, and those were left untagged. README section 2.b makes the tag mandatory for submissions that do not restart the database, so add it to them.
Remote services driven by a hand-written run.sh that neither restarts anything server-side nor clears any cache. Tag every result file:
exit 0because the YT cluster is remote.Local daemons that the shared driver deliberately does not restart. lib/benchmark-common.sh only runs the ./stop -> drop_caches -> ./start -> ./check cycle when BENCH_RESTARTABLE=yes; these four set it to no and fall through to bench_flush_caches alone, so the page cache is dropped while the engine keeps its internal caches. Each has a reason recorded in its own benchmark.sh: restarting catalogd would empty Impala's in-memory catalog, and firebolt-core keeps the loaded database in its bind-mounted volume. Tag the result files and template.json so future runs stay flagged:
The other systems that set BENCH_RESTARTABLE=no are genuine embedded CLIs whose ./start is
exit 0, so they satisfy section 2.a implicitly and are left alone. That includes zighouse, which does have a server but whose ./query invokes the binary directly instead.