docs: add Vamana benchmark comparison

docs/README.md

@@ -1,7 +1,7 @@
 # xvec benchmark website
 
-An English, static Astro + TypeScript site for the HNSW, Flat, and DiskANN measurements in
-`benchmark-hnsw.csv`, `benchmark-flat.csv`, and `benchmark-diskann.csv`. ECharts is bundled locally; no backend or external chart
+An English, static Astro + TypeScript site for the HNSW, Flat, DiskANN, and Vamana measurements in
+`benchmark-hnsw.csv`, `benchmark-flat.csv`, `benchmark-diskann.csv`, and `benchmark-vamana.csv`. ECharts is bundled locally; no backend or external chart
 service is needed. The CSV files and `logo.png` stay in this directory and are imported
 through Astro/Vite to generate the homepage at build time.
 
@@ -15,7 +15,7 @@ native `zvec_index_params_set_quantizer_enable_rotate` setter when selecting
 INT4/INT8, matching xvec. The native library is unchanged, and rotation is
 verified through its parameter getter. The Index selector between Dataset and
 Test configuration switches all charts and the SVG export between HNSW
-(the default), Flat, and DiskANN. Configuration labels show the selected index's parameters.
+(the default), Flat, DiskANN, and Vamana. Configuration labels show the selected index's parameters.
 
 `benchmark-diskann.csv` contains four runs: xvec and zvec with FP16 scalar
 quantization or unquantized FP32. It uses the same `Performance768D100K` dataset
@@ -26,6 +26,43 @@ disabled. FP32 means no scalar quantization; DiskANN still uses the configured
 product quantization for graph traversal. Both collections use `enable_mmap=true`;
 the benchmark does not flush filesystem caches between queries.
 
+`benchmark-vamana.csv` contains eight runs: xvec and zvec with INT4, INT8,
+FP16, and unquantized FP32 on the same Cohere `Performance768D100K` dataset.
+Both backends use maximum degree 64, construction list size 100, alpha 1.2,
+maximum occlusion size 750, and search list size 200. Graph saturation,
+two-pass construction, contiguous-memory mode, ID maps, and refinement are
+disabled. INT4/INT8 enable rotation, verified through the native parameter getter.
+The Vamana-specific CSV columns record these settings and participate in grouping.
+
+These runs use xvec commit `6a8b120d4284bf16853b2b4ad18465c599e72d82` (merged
+PR #91), zvec-go `v0.7.0+rotate`, and the unchanged native zvec library built from
+`8321c1314a559fd5f909e92498f43e5194bf9b99`. They use Go 1.27.1 with
+`CGO_ENABLED=0`, `GOMAXPROCS=8`, `GOMEMLIMIT=24GiB`, and CPU affinity 0–7
+on e2-standard-8. Each run uses a fresh collection, all 100,000 vectors,
+1,000 serial queries, K=100, batch size 100, optimize concurrency 8,
+30 seconds of 8-worker concurrent queries, and a 3-second serial cooldown.
+A representative run (repeat with a fresh path for each backend and precision):
+
+```sh
+CGO_ENABLED=0 go build -o /tmp/vector-db-bench ./cmd/vector-db-bench
+env GOMAXPROCS=8 GOMEMLIMIT=24GiB taskset -c 0-7 /tmp/vector-db-bench xvec \
+  --path /tmp/xvec-vamana-fp32 --case-type Performance768D100K \
+  --dataset-dir /path/to/dataset --skip-download --index-type vamana \
+  --ef-search 200 --k 100 --batch-size 100 --max-docs-per-segment 10000000 \
+  --optimize-concurrency 8 --num-concurrency 8 --concurrency-duration 30s \
+  --serial-cooldown 3s --payload-profile ids_only --enable-mmap=true \
+  --is-using-refiner=false --output /tmp/xvec-vamana-fp32.json
+```
+
+Run from the repository root. For FP16/INT8/INT4, add `--quantize-type fp16`,
+`int8`, or `int4`. For zvec, set `ZVEC_LIBRARY_PATH` to the native library and
+build with a temporary modfile replacing zvec-go with the local rotation-enabled
+binding described above. An unmodified binding does not reproduce the rotated runs.
+
+Peak RSS is the entire benchmark process's high-water mark from `wait4`, including
+loading, optimization, reopening, and querying. Measurements are single runs;
+QPS should be interpreted together with recall, not as equal-recall comparisons.
+
 ## Development
 
 Use Node.js 22.12+ and pnpm 10.33.0 (the version pinned in `package.json`).
@@ -78,9 +115,9 @@ setting cannot silently disappear from comparison grouping.
 `src/lib/benchmark.ts` defines the typed schema, shared metric labels and units,
 and grouping rules. All fields in `suiteFields` must match: machine, dataset,
 document count, HNSW and runtime parameters, payload, concurrency duration,
-cooldown, and serial query count. HNSW and DiskANN parameters are required only
-for their respective indexes; Flat omits both sets. DiskANN's four parameters
-all participate in grouping. Machine, dataset, index, and test configuration
+cooldown, and serial query count. HNSW, DiskANN, and Vamana parameters are required only
+for their respective indexes; Flat omits all three sets. All applicable index
+parameters participate in grouping. Machine, dataset, index, and test configuration
 selectors expose separate groups as data is added. Single-choice selectors are
 disabled. Quantization and rotation define individual chart categories. An xvec
 and zvec bar are paired only when both category settings match. Incomplete
@@ -89,9 +126,9 @@ pairs remain visible, with missing measurements represented as gaps, not zero.
 There must be at most one record per backend, suite, quantization, and rotation.
 A different backend version does not permit a duplicate: choose the intended
 run explicitly rather than silently combining repeated measurements. Backend
-versions are preserved in the source CSV. In the current data, xvec INT4/INT8,
-FP16, and FP32 use different commits, so these are not controlled quantization-only
-comparisons.
+versions are preserved in the source CSV. The index CSVs were measured at different revisions and times, so comparisons
+across index types are not controlled index-only experiments. All four Vamana
+precisions use the same xvec revision.
 
 `quantize_type=fp32` denotes unquantized FP32 vectors (`--quantize-type` omitted
 in the benchmark runner, whose JSON reports this as `none`). Rotation and
@@ -149,7 +186,7 @@ keep generated files out of Git. Deployment is separate from this build.
 ## Verification
 
 `pnpm test` checks all 104 HNSW chart measurements, Flat QPS and recall,
-DiskANN parameters and its FP16/FP32-only categories, index separation and configuration labels, every metric
+DiskANN parameters and its FP16/FP32-only categories, all Vamana settings and four precisions, index separation and configuration labels, every metric
 series, CSV quoting/BOM/CRLF, malformed data, duplicate records, and separation
 of incompatible configurations. SVG tests cover deterministic output, safe text,
 self-contained chart references, and configuration-specific asset paths. `pnpm check` checks Astro and TypeScript;

docs/benchmark-vamana.csv

@@ -0,0 +1,9 @@
+machine,backend,backend_version,case,index_type,quantize_type,rotate,use_refiner,enable_mmap,vamana_max_degree,vamana_build_list,vamana_query_list,vamana_alpha,vamana_max_occlusion_size,vamana_saturate_graph,vamana_two_pass_build,vamana_use_contiguous_memory,vamana_use_id_map,k,batch_size,max_docs_per_segment,optimize_concurrency,query_concurrency,concurrency_duration_sec,serial_cooldown_sec,payload_profile,gomaxprocs,gomemlimit,cpu_affinity,go_version,inserted_count,insert_duration_sec,optimize_duration_sec,load_duration_sec,insert_rows_per_sec,serial_queries,serial_qps,recall_at_k_pct,serial_latency_avg_ms,serial_latency_p95_ms,serial_latency_p99_ms,concurrent_queries,concurrent_qps,concurrent_latency_avg_ms,concurrent_latency_p95_ms,concurrent_latency_p99_ms,peak_rss_kib,peak_rss_mib,wall_seconds,user_cpu_seconds,system_cpu_seconds
+e2-standard-8,xvec,6a8b120d4284bf16853b2b4ad18465c599e72d82,Performance768D100K,vamana,int4,true,false,true,64,100,200,1.2,750,false,false,false,false,100,100,10000000,8,8,30,3,ids_only,8,24GiB,0-7,go1.27.1,100000,6.450255,49.679974,56.135672,15503.263734,1000,330.151683,87.012,2.796118,3.577116,3.940061,60109,2003.053308,3.992393,5.781849,7.197208,3166476,3092.261719,100.682135,552.081874,9.451192
+e2-standard-8,zvec,v0.7.0+rotate,Performance768D100K,vamana,int4,true,false,true,64,100,200,1.2,750,false,false,false,false,100,100,10000000,8,8,30,3,ids_only,8,24GiB,0-7,go1.27.1,100000,5.726085,31.247228,37.006987,17463.938926,1000,388.831267,81.039,2.355339,3.555305,3.973072,79847,2661.173599,3.004251,4.884632,6.706344,580228,566.628906,73.133591,448.838842,10.488299
+e2-standard-8,xvec,6a8b120d4284bf16853b2b4ad18465c599e72d82,Performance768D100K,vamana,int8,true,false,true,64,100,200,1.2,750,false,false,false,false,100,100,10000000,8,8,30,3,ids_only,8,24GiB,0-7,go1.27.1,100000,6.464561,55.287267,61.7595,15468.955289,1000,276.881767,98.73,3.33458,4.513977,5.536994,57449,1913.679863,4.178597,6.025895,7.338456,3221456,3145.953125,106.379817,577.352607,14.930293
+e2-standard-8,zvec,v0.7.0+rotate,Performance768D100K,vamana,int8,true,false,true,64,100,200,1.2,750,false,false,false,false,100,100,10000000,8,8,30,3,ids_only,8,24GiB,0-7,go1.27.1,100000,6.096447,28.252815,34.383313,16402.995921,1000,551.288631,98.355,1.622462,2.329426,2.686843,94285,3142.44868,2.5443,3.962252,5.503418,659776,644.3125,69.623703,420.824085,12.125019
+e2-standard-8,xvec,6a8b120d4284bf16853b2b4ad18465c599e72d82,Performance768D100K,vamana,fp16,false,false,true,64,100,200,1.2,750,false,false,false,false,100,100,10000000,8,8,30,3,ids_only,8,24GiB,0-7,go1.27.1,100000,8.872388,94.384487,103.266854,11270.922316,1000,152.714245,99.363,6.258708,10.97716,15.707088,38405,1279.504062,6.250287,9.105754,11.102209,3828872,3739.132812,154.680377,810.329211,25.327972
+e2-standard-8,zvec,v0.7.0+rotate,Performance768D100K,vamana,fp16,false,false,true,64,100,200,1.2,750,false,false,false,false,100,100,10000000,8,8,30,3,ids_only,8,24GiB,0-7,go1.27.1,100000,5.294007,77.133349,82.463938,18889.284977,1000,382.938654,99.198,2.399853,3.193495,3.536057,68113,2270.108375,3.522075,5.363077,7.251954,802112,783.3125,118.577794,784.075142,17.826853
+e2-standard-8,xvec,6a8b120d4284bf16853b2b4ad18465c599e72d82,Performance768D100K,vamana,fp32,false,false,true,64,100,200,1.2,750,false,false,false,false,100,100,10000000,8,8,30,3,ids_only,8,24GiB,0-7,go1.27.1,100000,4.953501,50.42262,55.383697,20187.742056,1000,317.498132,99.375,2.912781,3.781935,4.197044,48226,1607.41641,4.974847,6.636764,7.634434,2864312,2797.179688,95.84813,545.816312,12.665932
+e2-standard-8,zvec,v0.7.0+rotate,Performance768D100K,vamana,fp32,false,false,true,64,100,200,1.2,750,false,false,false,false,100,100,10000000,8,8,30,3,ids_only,8,24GiB,0-7,go1.27.1,100000,4.376264,57.116215,61.51392,22850.542618,1000,357.309009,99.392,2.593317,3.435763,3.753755,55841,1860.602216,4.297414,6.194429,8.125008,788484,770.003906,97.844471,638.635022,14.392509

docs/src/lib/benchmark-data.ts

@@ -1,3 +1,4 @@
+import vamanaCsv from '../../benchmark-vamana.csv?raw';
 import hnswCsv from '../../benchmark-hnsw.csv?raw';
 import flatCsv from '../../benchmark-flat.csv?raw';
 import diskannCsv from '../../benchmark-diskann.csv?raw';
@@ -9,4 +10,5 @@ export const groups = groupBenchmarks([
   ...parseBenchmarks(hnswCsv, 'benchmark-hnsw.csv'),
   ...parseBenchmarks(flatCsv, 'benchmark-flat.csv'),
   ...parseBenchmarks(diskannCsv, 'benchmark-diskann.csv'),
+  ...parseBenchmarks(vamanaCsv, 'benchmark-vamana.csv'),
 ]);

docs/src/lib/benchmark.ts

@@ -1,25 +1,29 @@
 // Shared by build-time parsing, CSV metadata, and browser charts.
 export const backends = ['xvec', 'zvec'] as const;
-export const indexTypes = ['hnsw', 'flat', 'diskann'] as const;
+export const indexTypes = ['hnsw', 'flat', 'diskann', 'vamana'] as const;
 export type IndexType = typeof indexTypes[number];
-export const indexLabels: Record<IndexType, string> = { hnsw: 'HNSW', flat: 'Flat', diskann: 'DiskANN' };
+export const indexLabels: Record<IndexType, string> = { hnsw: 'HNSW', flat: 'Flat', diskann: 'DiskANN', vamana: 'Vamana' };
 export const hnswFields = ['m', 'ef_construction', 'ef_search'] as const;
 export const diskannFields = ['diskann_max_degree', 'diskann_build_list', 'diskann_pq_chunks', 'diskann_query_list'] as const;
-export const indexFields = [...hnswFields, ...diskannFields];
+export const vamanaIntegerFields = ['vamana_max_degree', 'vamana_build_list', 'vamana_query_list', 'vamana_max_occlusion_size'] as const;
+export const vamanaBooleanFields = ['vamana_saturate_graph', 'vamana_two_pass_build', 'vamana_use_contiguous_memory', 'vamana_use_id_map'] as const;
+export const vamanaFields = [...vamanaIntegerFields, 'vamana_alpha', ...vamanaBooleanFields] as const;
+export const indexFields = [...hnswFields, ...diskannFields, ...vamanaFields];
 export const quantizations = ['int4', 'int8', 'fp16', 'fp32'] as const;
 export const colors = { xvec: '#b47c00', zvec: '#4977cd' };
 
 export const textFields = [
   'machine', 'backend', 'backend_version', 'case', 'index_type', 'quantize_type',
   'payload_profile', 'gomemlimit', 'cpu_affinity', 'go_version',
 ] as const;
-export const booleanFields = ['rotate', 'use_refiner', 'enable_mmap'] as const;
+export const booleanFields = ['rotate', 'use_refiner', 'enable_mmap', ...vamanaBooleanFields] as const;
 export const integerFields = [
-  ...indexFields, 'k', 'batch_size', 'max_docs_per_segment',
+  ...hnswFields, ...diskannFields, ...vamanaIntegerFields, 'k', 'batch_size', 'max_docs_per_segment',
   'optimize_concurrency', 'query_concurrency', 'gomaxprocs', 'inserted_count',
   'serial_queries', 'concurrent_queries', 'peak_rss_kib',
 ] as const;
 export const decimalFields = [
+  'vamana_alpha',
   'concurrency_duration_sec', 'serial_cooldown_sec', 'insert_duration_sec',
   'optimize_duration_sec', 'load_duration_sec', 'insert_rows_per_sec', 'serial_qps',
   'recall_at_k_pct', 'serial_latency_avg_ms', 'serial_latency_p95_ms',
@@ -30,9 +34,10 @@ export const decimalFields = [
 export const requiredFields = [...textFields, ...booleanFields, ...integerFields, ...decimalFields];
 export type NumericField = typeof integerFields[number] | typeof decimalFields[number];
 export type Benchmark = Record<typeof textFields[number], string>
-  & Record<typeof booleanFields[number], boolean>
+  & Record<Exclude<typeof booleanFields[number], typeof vamanaBooleanFields[number]>, boolean>
+  & Partial<Record<typeof vamanaBooleanFields[number], boolean>>
   & Record<Exclude<NumericField, typeof indexFields[number]>, number>
-  & Partial<Record<typeof indexFields[number], number>>
+  & Partial<Record<Extract<NumericField, typeof indexFields[number]>, number>>
   & { backend: typeof backends[number]; quantize_type: typeof quantizations[number]; index_type: IndexType };
 
 // Quantization and rotation define the individual x-axis categories. Every
@@ -43,7 +48,7 @@ export const suiteFields = [
   'optimize_concurrency', 'query_concurrency', 'concurrency_duration_sec',
   'serial_cooldown_sec', 'payload_profile', 'gomaxprocs', 'gomemlimit',
   'cpu_affinity', 'go_version', 'inserted_count', 'serial_queries',
-  ...diskannFields,
+  ...diskannFields, ...vamanaFields,
 ] as const satisfies readonly (keyof Benchmark)[];
 
 export interface ComparisonGroup {
@@ -55,8 +60,9 @@ export interface ComparisonGroup {
 }
 
 export function configurationKey(record: Benchmark): string {
-  // Preserve existing HNSW/Flat asset keys when adding DiskANN-only settings.
+  // Preserve existing asset keys when adding settings for a new index.
   return JSON.stringify(suiteFields
+    .filter((field) => record.index_type === 'vamana' || !(vamanaFields as readonly string[]).includes(field))
     .filter((field) => record.index_type === 'diskann' || !(diskannFields as readonly string[]).includes(field))
     .map((field) => record.index_type !== 'hnsw' && (hnswFields as readonly string[]).includes(field) ? null : record[field]));
 }

docs/src/lib/chart-options.ts

@@ -12,6 +12,9 @@ export const chartCards = [
 
 export function configurationLabel(group: ComparisonGroup): string {
   const record = group.records[0];
+  if (record.index_type === 'vamana') {
+    return `Degree ${record.vamana_max_degree} · Search ${record.vamana_query_list} · Concurrency ${record.query_concurrency}`;
+  }
   if (record.index_type === 'diskann') {
     return `Degree ${record.diskann_max_degree} · Search ${record.diskann_query_list} · Concurrency ${record.query_concurrency}`;
   }

docs/src/lib/parse-benchmark.ts

@@ -1,6 +1,6 @@
 import { parse } from 'csv-parse/sync';
 import {
-  backends, quantizations, indexTypes, hnswFields, diskannFields, indexFields, textFields, booleanFields, integerFields, decimalFields,
+  backends, quantizations, indexTypes, hnswFields, diskannFields, vamanaFields, vamanaBooleanFields, indexFields, textFields, booleanFields, integerFields, decimalFields,
   requiredFields, configurationKey, type Benchmark,
 } from './benchmark';
 
@@ -33,11 +33,13 @@ export function parseBenchmarks(csv: string, source = 'benchmark-hnsw.csv'): Ben
       value[field] = raw[field].trim();
     }
     for (const field of booleanFields) {
+      if (raw.index_type !== 'vamana' && (vamanaBooleanFields as readonly string[]).includes(field) && raw[field] === undefined) continue;
       if (!['true', 'false'].includes(raw[field])) invalid(field, 'expected true or false');
       value[field] = raw[field] === 'true';
     }
     for (const field of [...integerFields, ...decimalFields]) {
       const irrelevant = (raw.index_type !== 'hnsw' && (hnswFields as readonly string[]).includes(field))
+        || (raw.index_type !== 'vamana' && (vamanaFields as readonly string[]).includes(field))
         || (raw.index_type !== 'diskann' && (diskannFields as readonly string[]).includes(field));
       if (irrelevant && raw[field] === undefined) continue;
       const input = raw[field];
@@ -48,13 +50,13 @@ export function parseBenchmarks(csv: string, source = 'benchmark-hnsw.csv'): Ben
       if ((integerFields as readonly string[]).includes(field) && !Number.isSafeInteger(number)) invalid(field, 'expected a safe integer');
       value[field] = number;
     }
-    for (const field of ['m', 'ef_construction', 'ef_search', 'diskann_max_degree', 'diskann_build_list', 'diskann_query_list', 'k', 'batch_size', 'max_docs_per_segment',
+    for (const field of ['vamana_max_degree', 'vamana_build_list', 'vamana_query_list', 'vamana_alpha', 'm', 'ef_construction', 'ef_search', 'diskann_max_degree', 'diskann_build_list', 'diskann_query_list', 'k', 'batch_size', 'max_docs_per_segment',
       'optimize_concurrency', 'query_concurrency', 'gomaxprocs', 'inserted_count', 'serial_queries', 'concurrent_queries', 'concurrency_duration_sec']) {
       if (value[field] === 0) invalid(field, 'must be greater than zero');
     }
     if (!(backends as readonly string[]).includes(String(value.backend))) invalid('backend', 'expected xvec or zvec');
     if (!(quantizations as readonly string[]).includes(String(value.quantize_type))) invalid('quantize_type', 'expected int4, int8, fp16, or fp32 (unquantized)');
-    if (!(indexTypes as readonly string[]).includes(String(value.index_type))) invalid('index_type', 'expected hnsw, flat, or diskann');
+    if (!(indexTypes as readonly string[]).includes(String(value.index_type))) invalid('index_type', 'expected hnsw, flat, diskann, or vamana');
     if (Number(value.recall_at_k_pct) > 100) invalid('recall_at_k_pct', 'must be between 0 and 100');
     const benchmark = value as Benchmark;
     const key = JSON.stringify([configurationKey(benchmark), benchmark.quantize_type, benchmark.rotate, benchmark.backend]);

docs/src/pages/index.astro

@@ -16,7 +16,7 @@ const repository = 'https://github.com/gorse-io/xvec';
   <head>
     <meta charset="UTF-8" />
     <meta name="viewport" content="width=device-width, initial-scale=1" />
-    <meta name="description" content="Explore xvec and zvec HNSW, Flat, and DiskANN benchmarks: throughput, recall, latency, loading, and memory." />
+    <meta name="description" content="Explore xvec and zvec HNSW, Flat, DiskANN, and Vamana benchmarks: throughput, recall, latency, loading, and memory." />
     <meta name="theme-color" content="#f7f8fa" />
     <link rel="icon" type="image/png" href={logo.src} />
     <title>xvec — HNSW benchmarks</title>