diff --git a/docs/README.md b/docs/README.md index 22eaecc..aa1af75 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,7 +1,7 @@ # xvec benchmark website -An English, static Astro + TypeScript site for the HNSW, Flat, and DiskANN measurements in -`benchmark-hnsw.csv`, `benchmark-flat.csv`, and `benchmark-diskann.csv`. ECharts is bundled locally; no backend or external chart +An English, static Astro + TypeScript site for the HNSW, Flat, DiskANN, and Vamana measurements in +`benchmark-hnsw.csv`, `benchmark-flat.csv`, `benchmark-diskann.csv`, and `benchmark-vamana.csv`. ECharts is bundled locally; no backend or external chart service is needed. The CSV files and `logo.png` stay in this directory and are imported through Astro/Vite to generate the homepage at build time. @@ -15,7 +15,7 @@ native `zvec_index_params_set_quantizer_enable_rotate` setter when selecting INT4/INT8, matching xvec. The native library is unchanged, and rotation is verified through its parameter getter. The Index selector between Dataset and Test configuration switches all charts and the SVG export between HNSW -(the default), Flat, and DiskANN. Configuration labels show the selected index's parameters. +(the default), Flat, DiskANN, and Vamana. Configuration labels show the selected index's parameters. `benchmark-diskann.csv` contains four runs: xvec and zvec with FP16 scalar quantization or unquantized FP32. It uses the same `Performance768D100K` dataset @@ -26,6 +26,43 @@ disabled. FP32 means no scalar quantization; DiskANN still uses the configured product quantization for graph traversal. Both collections use `enable_mmap=true`; the benchmark does not flush filesystem caches between queries. +`benchmark-vamana.csv` contains eight runs: xvec and zvec with INT4, INT8, +FP16, and unquantized FP32 on the same Cohere `Performance768D100K` dataset. +Both backends use maximum degree 64, construction list size 100, alpha 1.2, +maximum occlusion size 750, and search list size 200. Graph saturation, +two-pass construction, contiguous-memory mode, ID maps, and refinement are +disabled. INT4/INT8 enable rotation, verified through the native parameter getter. +The Vamana-specific CSV columns record these settings and participate in grouping. + +These runs use xvec commit `6a8b120d4284bf16853b2b4ad18465c599e72d82` (merged +PR #91), zvec-go `v0.7.0+rotate`, and the unchanged native zvec library built from +`8321c1314a559fd5f909e92498f43e5194bf9b99`. They use Go 1.27.1 with +`CGO_ENABLED=0`, `GOMAXPROCS=8`, `GOMEMLIMIT=24GiB`, and CPU affinity 0–7 +on e2-standard-8. Each run uses a fresh collection, all 100,000 vectors, +1,000 serial queries, K=100, batch size 100, optimize concurrency 8, +30 seconds of 8-worker concurrent queries, and a 3-second serial cooldown. +A representative run (repeat with a fresh path for each backend and precision): + +```sh +CGO_ENABLED=0 go build -o /tmp/vector-db-bench ./cmd/vector-db-bench +env GOMAXPROCS=8 GOMEMLIMIT=24GiB taskset -c 0-7 /tmp/vector-db-bench xvec \ + --path /tmp/xvec-vamana-fp32 --case-type Performance768D100K \ + --dataset-dir /path/to/dataset --skip-download --index-type vamana \ + --ef-search 200 --k 100 --batch-size 100 --max-docs-per-segment 10000000 \ + --optimize-concurrency 8 --num-concurrency 8 --concurrency-duration 30s \ + --serial-cooldown 3s --payload-profile ids_only --enable-mmap=true \ + --is-using-refiner=false --output /tmp/xvec-vamana-fp32.json +``` + +Run from the repository root. For FP16/INT8/INT4, add `--quantize-type fp16`, +`int8`, or `int4`. For zvec, set `ZVEC_LIBRARY_PATH` to the native library and +build with a temporary modfile replacing zvec-go with the local rotation-enabled +binding described above. An unmodified binding does not reproduce the rotated runs. + +Peak RSS is the entire benchmark process's high-water mark from `wait4`, including +loading, optimization, reopening, and querying. Measurements are single runs; +QPS should be interpreted together with recall, not as equal-recall comparisons. + ## Development Use Node.js 22.12+ and pnpm 10.33.0 (the version pinned in `package.json`). @@ -78,9 +115,9 @@ setting cannot silently disappear from comparison grouping. `src/lib/benchmark.ts` defines the typed schema, shared metric labels and units, and grouping rules. All fields in `suiteFields` must match: machine, dataset, document count, HNSW and runtime parameters, payload, concurrency duration, -cooldown, and serial query count. HNSW and DiskANN parameters are required only -for their respective indexes; Flat omits both sets. DiskANN's four parameters -all participate in grouping. Machine, dataset, index, and test configuration +cooldown, and serial query count. HNSW, DiskANN, and Vamana parameters are required only +for their respective indexes; Flat omits all three sets. All applicable index +parameters participate in grouping. Machine, dataset, index, and test configuration selectors expose separate groups as data is added. Single-choice selectors are disabled. Quantization and rotation define individual chart categories. An xvec and zvec bar are paired only when both category settings match. Incomplete @@ -89,9 +126,9 @@ pairs remain visible, with missing measurements represented as gaps, not zero. There must be at most one record per backend, suite, quantization, and rotation. A different backend version does not permit a duplicate: choose the intended run explicitly rather than silently combining repeated measurements. Backend -versions are preserved in the source CSV. In the current data, xvec INT4/INT8, -FP16, and FP32 use different commits, so these are not controlled quantization-only -comparisons. +versions are preserved in the source CSV. The index CSVs were measured at different revisions and times, so comparisons +across index types are not controlled index-only experiments. All four Vamana +precisions use the same xvec revision. `quantize_type=fp32` denotes unquantized FP32 vectors (`--quantize-type` omitted in the benchmark runner, whose JSON reports this as `none`). Rotation and @@ -149,7 +186,7 @@ keep generated files out of Git. Deployment is separate from this build. ## Verification `pnpm test` checks all 104 HNSW chart measurements, Flat QPS and recall, -DiskANN parameters and its FP16/FP32-only categories, index separation and configuration labels, every metric +DiskANN parameters and its FP16/FP32-only categories, all Vamana settings and four precisions, index separation and configuration labels, every metric series, CSV quoting/BOM/CRLF, malformed data, duplicate records, and separation of incompatible configurations. SVG tests cover deterministic output, safe text, self-contained chart references, and configuration-specific asset paths. `pnpm check` checks Astro and TypeScript; diff --git a/docs/benchmark-vamana.csv b/docs/benchmark-vamana.csv new file mode 100644 index 0000000..45d6fff --- /dev/null +++ b/docs/benchmark-vamana.csv @@ -0,0 +1,9 @@ +machine,backend,backend_version,case,index_type,quantize_type,rotate,use_refiner,enable_mmap,vamana_max_degree,vamana_build_list,vamana_query_list,vamana_alpha,vamana_max_occlusion_size,vamana_saturate_graph,vamana_two_pass_build,vamana_use_contiguous_memory,vamana_use_id_map,k,batch_size,max_docs_per_segment,optimize_concurrency,query_concurrency,concurrency_duration_sec,serial_cooldown_sec,payload_profile,gomaxprocs,gomemlimit,cpu_affinity,go_version,inserted_count,insert_duration_sec,optimize_duration_sec,load_duration_sec,insert_rows_per_sec,serial_queries,serial_qps,recall_at_k_pct,serial_latency_avg_ms,serial_latency_p95_ms,serial_latency_p99_ms,concurrent_queries,concurrent_qps,concurrent_latency_avg_ms,concurrent_latency_p95_ms,concurrent_latency_p99_ms,peak_rss_kib,peak_rss_mib,wall_seconds,user_cpu_seconds,system_cpu_seconds +e2-standard-8,xvec,6a8b120d4284bf16853b2b4ad18465c599e72d82,Performance768D100K,vamana,int4,true,false,true,64,100,200,1.2,750,false,false,false,false,100,100,10000000,8,8,30,3,ids_only,8,24GiB,0-7,go1.27.1,100000,6.450255,49.679974,56.135672,15503.263734,1000,330.151683,87.012,2.796118,3.577116,3.940061,60109,2003.053308,3.992393,5.781849,7.197208,3166476,3092.261719,100.682135,552.081874,9.451192 +e2-standard-8,zvec,v0.7.0+rotate,Performance768D100K,vamana,int4,true,false,true,64,100,200,1.2,750,false,false,false,false,100,100,10000000,8,8,30,3,ids_only,8,24GiB,0-7,go1.27.1,100000,5.726085,31.247228,37.006987,17463.938926,1000,388.831267,81.039,2.355339,3.555305,3.973072,79847,2661.173599,3.004251,4.884632,6.706344,580228,566.628906,73.133591,448.838842,10.488299 +e2-standard-8,xvec,6a8b120d4284bf16853b2b4ad18465c599e72d82,Performance768D100K,vamana,int8,true,false,true,64,100,200,1.2,750,false,false,false,false,100,100,10000000,8,8,30,3,ids_only,8,24GiB,0-7,go1.27.1,100000,6.464561,55.287267,61.7595,15468.955289,1000,276.881767,98.73,3.33458,4.513977,5.536994,57449,1913.679863,4.178597,6.025895,7.338456,3221456,3145.953125,106.379817,577.352607,14.930293 +e2-standard-8,zvec,v0.7.0+rotate,Performance768D100K,vamana,int8,true,false,true,64,100,200,1.2,750,false,false,false,false,100,100,10000000,8,8,30,3,ids_only,8,24GiB,0-7,go1.27.1,100000,6.096447,28.252815,34.383313,16402.995921,1000,551.288631,98.355,1.622462,2.329426,2.686843,94285,3142.44868,2.5443,3.962252,5.503418,659776,644.3125,69.623703,420.824085,12.125019 +e2-standard-8,xvec,6a8b120d4284bf16853b2b4ad18465c599e72d82,Performance768D100K,vamana,fp16,false,false,true,64,100,200,1.2,750,false,false,false,false,100,100,10000000,8,8,30,3,ids_only,8,24GiB,0-7,go1.27.1,100000,8.872388,94.384487,103.266854,11270.922316,1000,152.714245,99.363,6.258708,10.97716,15.707088,38405,1279.504062,6.250287,9.105754,11.102209,3828872,3739.132812,154.680377,810.329211,25.327972 +e2-standard-8,zvec,v0.7.0+rotate,Performance768D100K,vamana,fp16,false,false,true,64,100,200,1.2,750,false,false,false,false,100,100,10000000,8,8,30,3,ids_only,8,24GiB,0-7,go1.27.1,100000,5.294007,77.133349,82.463938,18889.284977,1000,382.938654,99.198,2.399853,3.193495,3.536057,68113,2270.108375,3.522075,5.363077,7.251954,802112,783.3125,118.577794,784.075142,17.826853 +e2-standard-8,xvec,6a8b120d4284bf16853b2b4ad18465c599e72d82,Performance768D100K,vamana,fp32,false,false,true,64,100,200,1.2,750,false,false,false,false,100,100,10000000,8,8,30,3,ids_only,8,24GiB,0-7,go1.27.1,100000,4.953501,50.42262,55.383697,20187.742056,1000,317.498132,99.375,2.912781,3.781935,4.197044,48226,1607.41641,4.974847,6.636764,7.634434,2864312,2797.179688,95.84813,545.816312,12.665932 +e2-standard-8,zvec,v0.7.0+rotate,Performance768D100K,vamana,fp32,false,false,true,64,100,200,1.2,750,false,false,false,false,100,100,10000000,8,8,30,3,ids_only,8,24GiB,0-7,go1.27.1,100000,4.376264,57.116215,61.51392,22850.542618,1000,357.309009,99.392,2.593317,3.435763,3.753755,55841,1860.602216,4.297414,6.194429,8.125008,788484,770.003906,97.844471,638.635022,14.392509 diff --git a/docs/src/lib/benchmark-data.ts b/docs/src/lib/benchmark-data.ts index d465a77..c631969 100644 --- a/docs/src/lib/benchmark-data.ts +++ b/docs/src/lib/benchmark-data.ts @@ -1,3 +1,4 @@ +import vamanaCsv from '../../benchmark-vamana.csv?raw'; import hnswCsv from '../../benchmark-hnsw.csv?raw'; import flatCsv from '../../benchmark-flat.csv?raw'; import diskannCsv from '../../benchmark-diskann.csv?raw'; @@ -9,4 +10,5 @@ export const groups = groupBenchmarks([ ...parseBenchmarks(hnswCsv, 'benchmark-hnsw.csv'), ...parseBenchmarks(flatCsv, 'benchmark-flat.csv'), ...parseBenchmarks(diskannCsv, 'benchmark-diskann.csv'), + ...parseBenchmarks(vamanaCsv, 'benchmark-vamana.csv'), ]); diff --git a/docs/src/lib/benchmark.ts b/docs/src/lib/benchmark.ts index cf11b8b..dc72619 100644 --- a/docs/src/lib/benchmark.ts +++ b/docs/src/lib/benchmark.ts @@ -1,11 +1,14 @@ // Shared by build-time parsing, CSV metadata, and browser charts. export const backends = ['xvec', 'zvec'] as const; -export const indexTypes = ['hnsw', 'flat', 'diskann'] as const; +export const indexTypes = ['hnsw', 'flat', 'diskann', 'vamana'] as const; export type IndexType = typeof indexTypes[number]; -export const indexLabels: Record = { hnsw: 'HNSW', flat: 'Flat', diskann: 'DiskANN' }; +export const indexLabels: Record = { hnsw: 'HNSW', flat: 'Flat', diskann: 'DiskANN', vamana: 'Vamana' }; export const hnswFields = ['m', 'ef_construction', 'ef_search'] as const; export const diskannFields = ['diskann_max_degree', 'diskann_build_list', 'diskann_pq_chunks', 'diskann_query_list'] as const; -export const indexFields = [...hnswFields, ...diskannFields]; +export const vamanaIntegerFields = ['vamana_max_degree', 'vamana_build_list', 'vamana_query_list', 'vamana_max_occlusion_size'] as const; +export const vamanaBooleanFields = ['vamana_saturate_graph', 'vamana_two_pass_build', 'vamana_use_contiguous_memory', 'vamana_use_id_map'] as const; +export const vamanaFields = [...vamanaIntegerFields, 'vamana_alpha', ...vamanaBooleanFields] as const; +export const indexFields = [...hnswFields, ...diskannFields, ...vamanaFields]; export const quantizations = ['int4', 'int8', 'fp16', 'fp32'] as const; export const colors = { xvec: '#b47c00', zvec: '#4977cd' }; @@ -13,13 +16,14 @@ export const textFields = [ 'machine', 'backend', 'backend_version', 'case', 'index_type', 'quantize_type', 'payload_profile', 'gomemlimit', 'cpu_affinity', 'go_version', ] as const; -export const booleanFields = ['rotate', 'use_refiner', 'enable_mmap'] as const; +export const booleanFields = ['rotate', 'use_refiner', 'enable_mmap', ...vamanaBooleanFields] as const; export const integerFields = [ - ...indexFields, 'k', 'batch_size', 'max_docs_per_segment', + ...hnswFields, ...diskannFields, ...vamanaIntegerFields, 'k', 'batch_size', 'max_docs_per_segment', 'optimize_concurrency', 'query_concurrency', 'gomaxprocs', 'inserted_count', 'serial_queries', 'concurrent_queries', 'peak_rss_kib', ] as const; export const decimalFields = [ + 'vamana_alpha', 'concurrency_duration_sec', 'serial_cooldown_sec', 'insert_duration_sec', 'optimize_duration_sec', 'load_duration_sec', 'insert_rows_per_sec', 'serial_qps', 'recall_at_k_pct', 'serial_latency_avg_ms', 'serial_latency_p95_ms', @@ -30,9 +34,10 @@ export const decimalFields = [ export const requiredFields = [...textFields, ...booleanFields, ...integerFields, ...decimalFields]; export type NumericField = typeof integerFields[number] | typeof decimalFields[number]; export type Benchmark = Record - & Record + & Record, boolean> + & Partial> & Record, number> - & Partial> + & Partial, number>> & { backend: typeof backends[number]; quantize_type: typeof quantizations[number]; index_type: IndexType }; // Quantization and rotation define the individual x-axis categories. Every @@ -43,7 +48,7 @@ export const suiteFields = [ 'optimize_concurrency', 'query_concurrency', 'concurrency_duration_sec', 'serial_cooldown_sec', 'payload_profile', 'gomaxprocs', 'gomemlimit', 'cpu_affinity', 'go_version', 'inserted_count', 'serial_queries', - ...diskannFields, + ...diskannFields, ...vamanaFields, ] as const satisfies readonly (keyof Benchmark)[]; export interface ComparisonGroup { @@ -55,8 +60,9 @@ export interface ComparisonGroup { } export function configurationKey(record: Benchmark): string { - // Preserve existing HNSW/Flat asset keys when adding DiskANN-only settings. + // Preserve existing asset keys when adding settings for a new index. return JSON.stringify(suiteFields + .filter((field) => record.index_type === 'vamana' || !(vamanaFields as readonly string[]).includes(field)) .filter((field) => record.index_type === 'diskann' || !(diskannFields as readonly string[]).includes(field)) .map((field) => record.index_type !== 'hnsw' && (hnswFields as readonly string[]).includes(field) ? null : record[field])); } diff --git a/docs/src/lib/chart-options.ts b/docs/src/lib/chart-options.ts index 6cda176..cfc6985 100644 --- a/docs/src/lib/chart-options.ts +++ b/docs/src/lib/chart-options.ts @@ -12,6 +12,9 @@ export const chartCards = [ export function configurationLabel(group: ComparisonGroup): string { const record = group.records[0]; + if (record.index_type === 'vamana') { + return `Degree ${record.vamana_max_degree} · Search ${record.vamana_query_list} · Concurrency ${record.query_concurrency}`; + } if (record.index_type === 'diskann') { return `Degree ${record.diskann_max_degree} · Search ${record.diskann_query_list} · Concurrency ${record.query_concurrency}`; } diff --git a/docs/src/lib/parse-benchmark.ts b/docs/src/lib/parse-benchmark.ts index 5136f05..1244ca0 100644 --- a/docs/src/lib/parse-benchmark.ts +++ b/docs/src/lib/parse-benchmark.ts @@ -1,6 +1,6 @@ import { parse } from 'csv-parse/sync'; import { - backends, quantizations, indexTypes, hnswFields, diskannFields, indexFields, textFields, booleanFields, integerFields, decimalFields, + backends, quantizations, indexTypes, hnswFields, diskannFields, vamanaFields, vamanaBooleanFields, indexFields, textFields, booleanFields, integerFields, decimalFields, requiredFields, configurationKey, type Benchmark, } from './benchmark'; @@ -33,11 +33,13 @@ export function parseBenchmarks(csv: string, source = 'benchmark-hnsw.csv'): Ben value[field] = raw[field].trim(); } for (const field of booleanFields) { + if (raw.index_type !== 'vamana' && (vamanaBooleanFields as readonly string[]).includes(field) && raw[field] === undefined) continue; if (!['true', 'false'].includes(raw[field])) invalid(field, 'expected true or false'); value[field] = raw[field] === 'true'; } for (const field of [...integerFields, ...decimalFields]) { const irrelevant = (raw.index_type !== 'hnsw' && (hnswFields as readonly string[]).includes(field)) + || (raw.index_type !== 'vamana' && (vamanaFields as readonly string[]).includes(field)) || (raw.index_type !== 'diskann' && (diskannFields as readonly string[]).includes(field)); if (irrelevant && raw[field] === undefined) continue; const input = raw[field]; @@ -48,13 +50,13 @@ export function parseBenchmarks(csv: string, source = 'benchmark-hnsw.csv'): Ben if ((integerFields as readonly string[]).includes(field) && !Number.isSafeInteger(number)) invalid(field, 'expected a safe integer'); value[field] = number; } - for (const field of ['m', 'ef_construction', 'ef_search', 'diskann_max_degree', 'diskann_build_list', 'diskann_query_list', 'k', 'batch_size', 'max_docs_per_segment', + for (const field of ['vamana_max_degree', 'vamana_build_list', 'vamana_query_list', 'vamana_alpha', 'm', 'ef_construction', 'ef_search', 'diskann_max_degree', 'diskann_build_list', 'diskann_query_list', 'k', 'batch_size', 'max_docs_per_segment', 'optimize_concurrency', 'query_concurrency', 'gomaxprocs', 'inserted_count', 'serial_queries', 'concurrent_queries', 'concurrency_duration_sec']) { if (value[field] === 0) invalid(field, 'must be greater than zero'); } if (!(backends as readonly string[]).includes(String(value.backend))) invalid('backend', 'expected xvec or zvec'); if (!(quantizations as readonly string[]).includes(String(value.quantize_type))) invalid('quantize_type', 'expected int4, int8, fp16, or fp32 (unquantized)'); - if (!(indexTypes as readonly string[]).includes(String(value.index_type))) invalid('index_type', 'expected hnsw, flat, or diskann'); + if (!(indexTypes as readonly string[]).includes(String(value.index_type))) invalid('index_type', 'expected hnsw, flat, diskann, or vamana'); if (Number(value.recall_at_k_pct) > 100) invalid('recall_at_k_pct', 'must be between 0 and 100'); const benchmark = value as Benchmark; const key = JSON.stringify([configurationKey(benchmark), benchmark.quantize_type, benchmark.rotate, benchmark.backend]); diff --git a/docs/src/pages/index.astro b/docs/src/pages/index.astro index afc66ab..ab88114 100644 --- a/docs/src/pages/index.astro +++ b/docs/src/pages/index.astro @@ -16,7 +16,7 @@ const repository = 'https://github.com/gorse-io/xvec'; - + xvec — HNSW benchmarks diff --git a/docs/tests/benchmark.test.ts b/docs/tests/benchmark.test.ts index ee06d9b..dee024a 100644 --- a/docs/tests/benchmark.test.ts +++ b/docs/tests/benchmark.test.ts @@ -2,7 +2,7 @@ import assert from 'node:assert/strict'; import { readFileSync } from 'node:fs'; import test from 'node:test'; import { parseBenchmarks } from '../src/lib/parse-benchmark'; -import { categories, diskannFields, formatBytes, groupBenchmarks, metricKeys, seriesFor, suiteFields, type Benchmark } from '../src/lib/benchmark'; +import { categories, diskannFields, vamanaFields, formatBytes, groupBenchmarks, metricKeys, seriesFor, suiteFields, type Benchmark } from '../src/lib/benchmark'; import { chartDefinition, configurationLabel } from '../src/lib/chart-options'; const csv = readFileSync(new URL('../benchmark-hnsw.csv', import.meta.url), 'utf8'); @@ -89,7 +89,7 @@ test('duplicate configurations fail even when versions differ', () => { test('every suite setting splits incompatible runs, including machine and dataset', () => { const record = parseBenchmarks(fixture())[0]; for (const field of suiteFields) { - if ((diskannFields as readonly string[]).includes(field)) continue; + if (([...diskannFields, ...vamanaFields] as readonly string[]).includes(field)) continue; const current = record[field]; const different = typeof current === 'number' ? current + 1 : typeof current === 'boolean' ? !current : `${current}-other`; const changed = { ...record, [field]: different } as Benchmark; @@ -169,3 +169,37 @@ test('DiskANN requires and groups by each of its four index parameters', () => { assert.throws(() => parseBenchmarks(rows.map((row) => row.join(',')).join('\n')), new RegExp(`field "${field}"`)); } }); + +test('Vamana includes all four precisions and stays separate from other indexes', () => { + const records = parseBenchmarks(readFileSync(new URL('../benchmark-vamana.csv', import.meta.url), 'utf8'), 'benchmark-vamana.csv'); + assert.equal(records.length, 8); + assert.ok(records.every((record) => record.index_type === 'vamana' && record.m === undefined && !record.use_refiner)); + assert.deepEqual(categories(records).map((category) => [category.quantization, category.rotate]), [['int4', true], ['int8', true], ['fp16', false], ['fp32', false]]); + const groups = groupBenchmarks([...parseBenchmarks(csv), ...parseBenchmarks(flatCsv), ...parseBenchmarks(diskannCsv), ...records]); + assert.deepEqual(groups.map((group) => group.indexType), ['hnsw', 'flat', 'diskann', 'vamana']); + assert.equal(configurationLabel(groups[3]), 'Degree 64 · Search 200 · Concurrency 8'); + assert.deepEqual(records.map((record) => vamanaFields.map((field) => record[field])), Array.from({ length: 8 }, () => [64, 100, 200, 750, 1.2, false, false, false, false])); + for (const record of records) { + assert.equal(record.inserted_count, 100000); + assert.equal(record.serial_queries, 1000); + assert.equal(record.backend_version, record.backend === 'xvec' ? '6a8b120d4284bf16853b2b4ad18465c599e72d82' : 'v0.7.0+rotate'); + } +}); + +test('Vamana validates and groups each build/search option without splitting other indexes', () => { + const input = readFileSync(new URL('../benchmark-vamana.csv', import.meta.url), 'utf8'); + const record = parseBenchmarks(input)[0]; + const hnsw = parseBenchmarks(csv)[0]; + for (const field of vamanaFields) { + const current = record[field]; + const changed = typeof current === 'boolean' ? !current : current! + 1; + assert.equal(groupBenchmarks([record, { ...record, [field]: changed }]).length, 2, field); + assert.equal(groupBenchmarks([hnsw, { ...hnsw, [field]: changed }]).length, 1, field); + const rows = input.trim().split('\n').map((line) => line.split(',')); + const column = rows[0].indexOf(field); + const missing = rows.map((row) => row.filter((_, index) => index !== column).join(',')).join('\n'); + assert.throws(() => parseBenchmarks(missing), new RegExp(`field "${field}"`)); + rows[1][column] = typeof current === 'boolean' ? 'yes' : '-1'; + assert.throws(() => parseBenchmarks(rows.map((row) => row.join(',')).join('\n')), new RegExp(`field "${field}"`)); + } +}); diff --git a/docs/tests/svg.test.ts b/docs/tests/svg.test.ts index 2e77746..69de963 100644 --- a/docs/tests/svg.test.ts +++ b/docs/tests/svg.test.ts @@ -57,3 +57,14 @@ test('DiskANN SVG identifies its index and only includes FP16 and FP32', () => { assert.doesNotMatch(svg, /INT4|INT8|HNSW|undefined/); assert.notEqual(svgAsset(diskann), svgAsset(group)); }); + +test('Vamana SVG identifies its index, configuration, and all four precisions', () => { + const vamana = groupBenchmarks(parseBenchmarks(readFileSync(new URL('../benchmark-vamana.csv', import.meta.url), 'utf8')))[0]; + const svg = renderBenchmarkSvg(vamana); + assert.match(svg, /Vamana benchmark results/); + assert.match(svg, />Vamana<\/text>/); + assert.match(svg, /Degree 64 · Search 200 · Concurrency 8/); + for (const precision of ['INT4', 'INT8', 'FP16', 'FP32']) assert.ok(svg.includes(`>${precision}`)); + assert.doesNotMatch(svg, /HNSW|undefined/); + assert.notEqual(svgAsset(vamana), svgAsset(group)); +});