Embedding Geometry

Code for the blog post How to win a beer with high-dimensional statistics.

The bet: the months of the year famously trace out a circle in word-embedding space, with a nicely circulant Gram matrix. Is that a real semantic fact, or does high-dimensional geometry hand you circles for free? The claim to be settled over a beer was that you can find ten arbitrary, semantically unrelated words that look just as circular.

You can. Here they are:

increased, has, ones, simply, i, title, john, september, united, conducted

Start here

notebooks/circulant_optimal_greedy.ipynb — the main notebook, and the search that actually won the bet.

It targets a circulant Gram matrix G[i,j] = amplitude · cos(2π·d(i,j)/n) (where d is circular distance) and runs a greedy search: walk through each of the n slots, evaluate every word in the 25k vocabulary as a replacement, take the best one, repeat until no position can be improved. Precomputing the full vocabulary Gram matrix up front makes each candidate evaluation a submatrix lookup instead of a dot product, which is what makes exhaustive replacement affordable.

The notebook also plots the months for comparison, and has a cell at the bottom where you can drop in your own ten words and see how circular they look.

The other notebooks

These are the earlier, less successful attempts, kept because the post walks through them:

• visualize_embeddings.ipynb — initial PCA exploration of semantic word sets (months, weekdays, etc.)
• visualize_embeddings_optimized.ipynb — random search against a "looks circular" objective (radial + angular penalties in the PCA plane). Works, but poorly.
• circulant_challenge.ipynb — first pass at optimizing the Gram matrix directly, with random replacement
• helix_experiment.ipynb — side quest on the Gram structure of points on a 3D helix

Repository layout

notebooks/     Jupyter notebooks (start with circulant_optimal_greedy.ipynb)
scripts/       Data prep, benchmarks, and standalone versions of the search
analysis/      Embedding statistics: SVD, isotropy, inner-product histograms
gpu_optimize/  CUDA port of the search for running many populations in parallel
results/       Output figures
data/          Embeddings and precomputed Gram matrices (gitignored, see below)

Data

The data files are gitignored — they total ~12 GB, mostly the precomputed Gram matrices. The embeddings themselves (dhruva_embeddings.pkl, 25k vocab × 12k dims, courtesy of Dhruva Karkada) are not redistributed here.

Given data/dhruva_embeddings.pkl, regenerate the rest with:

python3 scripts/convert_to_c_contiguous.py     # -> dhruva_embeddings_c.pkl (row-major, much faster lookups)
python3 scripts/precompute_gram_matrices.py    # -> gram_matrices.npz (~4.6 GB)

The notebooks also reference Google News word2vec vectors for comparison; fetch those with scripts/download_word2vec.py.

Setup

pip install numpy matplotlib scikit-learn gensim jupyter

For gpu_optimize/, see the requirements and README in that directory. Note that gpu_optimize/deploy_and_run.sh has hardcoded ngrok addresses for the GPU boxes used at the time — long dead, so edit them before using it.

A note on authorship

All of this — the exploration, the optimization code, the notebooks, the figures, and this README — was written by Claude (Anthropic's Claude Code), working from Jamie's direction. The beer was won by a human.