Skip to content

docs: add Vamana benchmark comparison - #92

Merged
zhenghaoz merged 1 commit into
mainfrom
benchmark/vamana
Sep 26, 2026
Merged

zhenghaoz merged 1 commit into
mainfrom
benchmark/vamana

Conversation

@zhenghaoz

Copy link
Copy Markdown
Contributor

Add the missing Vamana comparison after #91: eight full Cohere 100K × 768D runs covering xvec and zvec with FP32, FP16, INT8, and INT4. Publish the measurements in docs/benchmark-vamana.csv and expose Vamana in the Index selector, charts, and SVG export. HNSW remains the default, and existing benchmark CSVs and export URLs are preserved.

Both backends use degree 64, construction list 100, alpha 1.2, max occlusion 750, search list 200, and no saturation, two-pass build, contiguous-memory mode, ID map, or refinement. INT4/INT8 enable rotation, confirmed through zvec's native getter. Runs use fresh collections on e2-standard-8, K=100, 1,000 serial queries, 30 seconds at concurrency 8, and a 3-second cooldown. All Vamana settings are recorded and validated as comparison-group keys.

Values below are xvec / zvec. These are single runs at matching parameters, not matched recall. Peak RSS covers the entire process, including loading, optimization, reopening, and queries.

Precision Concurrent QPS Recall@100 (%) Peak RSS (MiB)
FP32 1607.4 / 1860.6 99.375 / 99.392 2797.2 / 770.0
FP16 1279.5 / 2270.1 99.363 / 99.198 3739.1 / 783.3
INT8 1913.7 / 3142.4 98.730 / 98.355 3146.0 / 644.3
INT4 2003.1 / 2661.2 87.012 / 81.039 3092.3 / 566.6

Measured xvec revision: 6a8b120d4284bf16853b2b4ad18465c599e72d82 (merged #91). zvec uses the existing local v0.7.0+rotate binding and native library from 8321c1314a559fd5f909e92498f43e5194bf9b99. Go 1.27.1, CGO disabled, GOMAXPROCS=8, GOMEMLIMIT=24GiB, CPU affinity 0–7. The README includes configuration, provenance, reproduction instructions, and measurement limits.

Validation: all eight small smoke runs and eight full benchmark runs passed; native Vamana rotation getter verified; pnpm --dir docs test (21 tests), check (zero diagnostics), and build passed. Playwright verified the default HNSW view, all four index selections, six charts per view, Vamana data, distinct working SVG exports, and no browser errors. Build retains the existing large-bundle warning.

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying xvec with  Cloudflare Pages  Cloudflare Pages

Latest commit: 57a54b9
Status: ✅  Deploy successful!
Preview URL: https://d0deacfc.xvec.pages.dev
Branch Preview URL: https://benchmark-vamana.xvec.pages.dev

View logs

@codecov

codecov Bot commented Sep 26, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.35%. Comparing base (6a8b120) to head (57a54b9).

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #92      +/-   ##
==========================================
- Coverage   81.36%   81.35%   -0.01%     
==========================================
  Files         150      150              
  Lines       30534    30534              
==========================================
- Hits        24843    24842       -1     
- Misses       5689     5690       +1     
  Partials        2        2              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@zhenghaoz
zhenghaoz merged commit 1a610c3 into main Sep 26, 2026
9 checks passed
@zhenghaoz
zhenghaoz deleted the benchmark/vamana branch September 26, 2026 06:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant