Spaces:
Running
Running
| title: Web Graph kNN Spot-check | |
| emoji: 🔎 | |
| colorFrom: indigo | |
| colorTo: gray | |
| sdk: gradio | |
| sdk_version: 6.26.0 | |
| python_version: 3.12.12 | |
| app_file: app.py | |
| pinned: false | |
| short_description: Nearest neighbours of a host in the CC host-graph embedding | |
| # Web graph kNN spot-check | |
| Type a hostname, get its nearest neighbours in the learned 128-d Common Crawl host-graph | |
| embedding. Companion to the 2D viewer: that one shows *where* hosts land after a UMAP | |
| projection, this one answers *who is actually nearest* in the real space. | |
| ## Running locally | |
| ```bash | |
| uv sync --extra faiss --extra viewer | |
| wgl knn build --hf-repo commoncrawl/web-graph-embeddings --shards 8 --row-groups 1 \ | |
| --out data/knn/cc_deg8_poc | |
| WGL_KNN_INDEX=data/knn/cc_deg8_poc uv run python knn-viewer/app.py | |
| ``` | |
| ## Running as a Space | |
| The app reads its index from `$WGL_KNN_INDEX`, defaulting to `/data` — the mount path for a | |
| HuggingFace volume. Point a **private** bucket at it: | |
| ```bash | |
| hf buckets create commoncrawl/web-graph-knn-index --private | |
| hf sync data/knn/cc_deg8_full hf://buckets/commoncrawl/web-graph-knn-index/cc_deg8 --delete | |
| hf spaces volumes set commoncrawl/web-graph-knn \ | |
| -v hf://buckets/commoncrawl/web-graph-knn-index:/data | |
| ``` | |
| `app.py` needs `wgl.knn.index`, which pulls only numpy, faiss and tldextract, so `wgl` is | |
| **vendored** next to `app.py` on the Space rather than installed. Only the *import closure* of | |
| `app.py` ships — 13 modules of 100, so the Space holds the code it runs and not the training | |
| tree. `KnnIndex` stays a `wgl` module rather than being copied into the app on purpose: it reads | |
| the artifact format `wgl.knn.build` writes, `wgl knn query` reads the same one, and | |
| `tests/test_knn_index.py` round-trips build against load to keep them honest. Deploying does the | |
| vendoring for you: | |
| ```bash | |
| make knn-diff # which files the Space is missing or has stale (changes nothing) | |
| make knn-deploy # upload the app + vendored src/wgl, skipping what the Space already has | |
| make knn-serve # run it locally against $KNN_INDEX first | |
| ``` | |