HNSW Index Size Reference — NodeMind Benchmark
===============================================

This file explains the 48× compression figure.

## What is HNSW?
HNSW (Hierarchical Navigable Small World) is the index structure used by FAISS,
Qdrant, Pinecone, Weaviate, and most production vector databases.

## Size formula
HNSW stores the float32 embeddings PLUS a graph structure (neighbour lists).
The graph overhead is approximately 1.3–1.6× the raw embedding size.

Conservative estimate used in this benchmark:
  HNSW size ≈ float32 embeddings × 1.5

For BGE-M3, 500K chunks:
  float32 embeddings = 500,000 × 1024 dims × 4 bytes = 2,048,000,000 bytes (≈ 2.0 GB)
  HNSW estimate      = 2,048,000,000 × 1.5           = 3,072,000,000 bytes (≈ 3.0 GB)
  NodeMind index     =    64,000,000 bytes                              (≈  64 MB)

  Compression: 3,072 / 64 = 48×

## Notes
- 1.5× is conservative. Real FAISS HNSW with M=16, efConstruction=200 can be 1.3–2.0×
  depending on settings and dimension.
- HNSW offers excellent recall@10 (0.95–0.99) with graph traversal.
- NodeMind achieves recall@10 = 1.000 on this self-retrieval benchmark.
- The comparison is index SIZE only — not a head-to-head on a neutral held-out dataset.

## Verification
You can verify the NodeMind index size yourself:
  import pickle
  with open("nm_bgem3_index.pkl","rb") as f: nm = pickle.load(f)
  print(nm["fps"].nbytes)   # → 64,000,000 bytes = 64 MB

Download the float32 RAG baseline (rag_bgem3_index.pkl, 2 GB) to verify that too.

Source: Malkov & Yashunin 2018 — "Efficient and Robust Approximate Nearest Neighbor Search
Using Hierarchical Navigable Small World Graphs" — https://arxiv.org/abs/1603.09320