Make combined metric fairer and increase transparency - #1439
Conversation
GitHub Actions
Error
|
@alexey-milovidov what's your onpinion on this? |
@toschmidt, this seems like a nice idea. What do you propose to do with cases where a system reports "unreasonable" times? E.g., load-time 2 secs. Would you ignore these as well? |
|
@hall-alex for systems with unreasonable load time or data size these results are excluded and are not factored in the combined metric. The metricExcludes function in index.html defines when a result is included. This function is also used to exclude systems from the load/size/cold views. |
@toschmidt, ah, nice -- makes sense to me 👍 |
4781505 to
6f7a74e
Compare
|
|
GitHub Actions
Error
ClickBench's "cold run" is only truly cold when the data must be read back from the storage device: the engine is restarted (or never persists between queries) AND the OS page cache is dropped before the first try. A system that instead runs an unseen query against a live engine with warm internal caches does a "lukewarm" run and is supposed to carry the "lukewarm-cold-run" tag. Several systems that do lukewarm runs were never tagged; this adds the tag to them. Newly flagged: - Persistent local daemons that are never restarted between queries, so only the page cache is dropped while engine-internal caches survive: impala, firebolt. - Managed/remote services that cannot be restarted or have their caches flushed at all: clickhouse-cloud, motherduck, databricks, snowflake, redshift(+serverless), bigquery, athena(+partitioned), aurora-mysql, aurora-postgresql, alloydb, hologres, supabase, tablespace, tembo-olap, chyt, bytehouse, crunchy-bridge-for-analytics, timescale-cloud, tinybird, singlestore, hydra. The tag is added to the displayed result files and, so future runs stay flagged, to each system's metadata source: template.json (impala, firebolt) and the inline JSON generators (clickhouse-cloud/collect-results.sh, motherduck/benchmark.py, databricks/benchmark.py). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Combined score is a weighted geomean of load (10%), data size (10%), cold (20%) and hot (60%) ratios. That unfairly penalizes systems for metrics that don't apply to them, and lets lukewarm "cold" numbers (really warm queries) distort the cold component. Unify the per-metric exclusion rules in a single metricExcludes() helper (stateless from load, in-memory from cold/combined/load, lukewarm from cold, missing data size from size) and reuse it everywhere: - Cold Run metric: lukewarm systems are excluded from the ranking by default. - Combined per-query baseline: the cold-run minimum excludes lukewarm / in-memory systems, so their warm "cold" numbers can't depress the baseline and inflate every true-cold system's cold ratio. min load time / min data size likewise exclude systems that don't qualify. - Combined score: a metric that doesn't apply to a system is dropped and the remaining weights are renormalized, instead of feeding a bogus ratio. Lukewarm systems keep a cold component of 0 with its weight folded into hot (load 10% / size 10% / hot 80%); a stateless engine that still reports a load time (e.g. Polars (Parquet)) drops the load component; etc. The cold term is guarded so an all-lukewarm selection (empty cold baseline) can't poison the score with NaN. The Combined view still shows only the single overall score; the per-component breakdown is added in a follow-up commit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Combined metric collapses load, data size, cold and hot into a single weighted-geomean "×N" score, which hides how a system earned it. In the Combined view, expand the score cell to also list the four component ratios that feed it: hot, cold, load and storage, each as the relative "×N" ratio (a component that doesn't apply to the system, e.g. cold for a lukewarm engine or load for a stateless one, shows "n/a"). Each ratio is padded to a fixed width with non-breaking spaces so the columns line up in the monospace cell (6 fits "×12.34" for hot/cold/load, 5 fits "×1.23" for storage), and the overall score is shown in bold. Other metric views are unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
6f7a74e to
999c158
Compare
GitHub Actions
Waiting
|
@alexey-milovidov, @rschu1ze can you take a look at this? |
|
@toschmidt, it might be easier to get the changes reviewed if you split this into 3 (?) separate PRs. To my mind the "reweight the combined metric" seems like an easy and clear candidate to discuss. Why should systems that report almost 0 secs for one of the metrics get an "unfair" advantage in the combined metric? I like the simplicity and the idea of "not judging": one simply excludes the "unrealistic metric" from the combined score -- no need to explain / fix stuff. (Side note: personally, I have found UX decisions like showing 4 metrics instead of 1 always hard to agree on.) |
|
@hall-alex yeah you are right. The UX change is debatable, I find my self switching between the different metrics quite often to see where systems perform well. However, 5 numbers instead of 1 might be confusing if you are not familiar with the benchmark. I will split the PR into 3 requests (first one is #1634). I saw you updated the Firebolt submission, is the new version |
|
We do not restart the system on each query, so we should be in the lukewarm bucket afaict ... I think that is the definition and I thought we were in that bucket. |
3 participants
The combined metric is difficult to interpret, so this PR displays the four individual values alongside it to make system strengths and weaknesses easier to identify.
Lukewarm-cold flagging: Several systems (especially cloud services) cannot be properly restarted, reporting lukewarm-cold times (new query, cached data) as true cold times (data loaded from storage). The PR marks several systems as
lukewarm-cold-runs: MotherDuck, Bigquery, ClickHouse Cloud, Redshift, Snowflake, ...Note that existing ClickHouse Cloud results remain unchanged; the template has been updated so the next daily run will report results with the new flag.
Reweight the combined metric: if a system does not report a result for one of the four categories (hot, cold, load, storage), that category is ignored and the remaining score is rescaled.
If a system does not report storage/load the metric is rescaled:
exp( (0.6*log(hot) + 0.2*log(cold) + 0.1*log(storage)) / 0.9)(if load is missing)If a system does not report a cold time, we keep the 80%/20% split between query and load/storage and compute the combined metric as
exp(0.8*log(hot) + 0.1*log(load) + 0.1*log(storage))(we could also remove this special case and just rescale like before)