Card: parity table at 1954de4 and real-document results at the model defaults

README.md

@@ -99,51 +99,67 @@ The weights are the original CodeRankEmbed weights (bf16-cast), so embeddings ma
 original to within bf16 precision. The table below measures every tier **on the same GPU**, so
 `eager` is the O(seq²) baseline the varlen tiers are compared against.
 
-**Protocol.** 64 real Python source snippets (the first 40 lines of each file; 428 tokens mean,
-638 max), encoded with sentence-transformers at an explicit **batch size of 32 and of 64** (the
+**Protocol.** 64 real Python source snippets (the first 40 lines of the first 64 `.py` files,
+by path, of the PenStock inference library's source tree; 428 tokens mean, 638 max), encoded
+with sentence-transformers at an explicit **batch size of 32 and of 64** (the
 `batch_size=` passed to `encode`; with 64 snippets, 64 is one forward). Each tier encodes once
 untimed, then again for the measurement. Cosine similarity is taken on fp32-renormalized output
 against the **pre-dispatch revision of this repo on the same GPU** (its `flash_attn` path).
 `flash_attn` runs the same computation as before the dispatch, so its cosine is a parity check.
 `torch_varlen` is a different, torch-native kernel, and `eager` a different algorithm, so theirs
 are the real signal. `auto` is the default with no override: the probe keeps `torch_varlen`.
-**Encode peak** is the encode's peak CUDA allocation above what the loaded model holds (266 MiB
-of bf16 weights; the first model loaded in a process also holds cuBLAS's workspace, 8 MiB on sm_86
-and 32 MiB on sm_120). Each tier loads and frees its own model, so no reading includes another
-tier's weights. Wall time is one encode of all 64 snippets. Cosines and VRAM repeated exactly
-across every run; wall time moved by up to ~15% between runs, so read it as indicative.
-Every row was measured loading this repo at commit `1954de4`.
+**Encode peak** is the encode's peak CUDA allocation above the loaded model's own ~266 MiB of
+bf16 weights. Wall time is one encode of all 64 snippets.
 
 | GPU | batch size | tier | mean cos | min cos | encode peak | wall |
 | --- | --- | --- | --- | --- | --- | --- |
 | RTX 3090 Ti (sm_86) | 32 | `torch_varlen` | 0.999941 | 0.999856 | 600 MiB | 0.197 s |
 | RTX 3090 Ti (sm_86) | 32 | `flash_attn` | 1.000000 | 1.000000 | 600 MiB | 0.197 s |
 | RTX 3090 Ti (sm_86) | 32 | `eager` | 0.999912 | 0.999856 | 807 MiB | 0.262 s |
-| RTX 3090 Ti (sm_86) | 32 | `auto` (→ `torch_varlen`) | 0.999941 | 0.999856 | 600 MiB | 0.197 s |
 | RTX 3090 Ti (sm_86) | 64 | `torch_varlen` | 0.999941 | 0.999856 | 1201 MiB | 0.223 s |
 | RTX 3090 Ti (sm_86) | 64 | `flash_attn` | 1.000000 | 1.000000 | 1201 MiB | 0.218 s |
 | RTX 3090 Ti (sm_86) | 64 | `eager` | 0.999910 | 0.999856 | 1614 MiB | 0.302 s |
-| RTX 3090 Ti (sm_86) | 64 | `auto` (→ `torch_varlen`) | 0.999941 | 0.999856 | 1201 MiB | 0.218 s |
-| RTX 5090 Laptop GPU (sm_120) | 32 | `torch_varlen` | 0.999944 | 0.999919 | 600 MiB | 0.235 s |
-| RTX 5090 Laptop GPU (sm_120) | 32 | `flash_attn` | 1.000000 | 1.000000 | 600 MiB | 0.234 s |
-| RTX 5090 Laptop GPU (sm_120) | 32 | `eager` | 0.999912 | 0.999870 | 807 MiB | 0.318 s |
-| RTX 5090 Laptop GPU (sm_120) | 32 | `auto` (→ `torch_varlen`) | 0.999944 | 0.999919 | 600 MiB | 0.235 s |
-| RTX 5090 Laptop GPU (sm_120) | 64 | `torch_varlen` | 0.999944 | 0.999919 | 1201 MiB | 0.271 s |
-| RTX 5090 Laptop GPU (sm_120) | 64 | `flash_attn` | 1.000000 | 1.000000 | 1201 MiB | 0.273 s |
-| RTX 5090 Laptop GPU (sm_120) | 64 | `eager` | 0.999913 | 0.999834 | 1614 MiB | 0.377 s |
-| RTX 5090 Laptop GPU (sm_120) | 64 | `auto` (→ `torch_varlen`) | 0.999944 | 0.999919 | 1201 MiB | 0.274 s |
+| RTX 5090 Laptop GPU (sm_120) | 32 | `torch_varlen` | 0.999944 | 0.999919 | 600 MiB | 0.177 s |
+| RTX 5090 Laptop GPU (sm_120) | 32 | `flash_attn` | 1.000000 | 1.000000 | 600 MiB | 0.175 s |
+| RTX 5090 Laptop GPU (sm_120) | 32 | `eager` | 0.999912 | 0.999870 | 807 MiB | 0.249 s |
+| RTX 5090 Laptop GPU (sm_120) | 64 | `torch_varlen` | 0.999944 | 0.999919 | 1201 MiB | 0.202 s |
+| RTX 5090 Laptop GPU (sm_120) | 64 | `flash_attn` | 1.000000 | 1.000000 | 1201 MiB | 0.203 s |
+| RTX 5090 Laptop GPU (sm_120) | 64 | `eager` | 0.999913 | 0.999834 | 1614 MiB | 0.298 s |
 
 Cosines rounded to 6 decimal places, VRAM to the nearest MiB. `torch 2.12.1+cu130`,
-`transformers 5.11.0`, `sentence-transformers 6.1.0`, `flash-attn 2.8.3` (`2.8.3.post1` on the 3090 Ti host).
-Each run held the hosts' shared GPU lock, with the card idle (0% util, no foreign process)
-before and after.
+`transformers 5.11.0`, `sentence-transformers 6.1.0`, `flash-attn 2.8.3` (`2.8.3.post1` on the
+3090 Ti), all measured loading this repo at `1954de4`.
 
 Eager's cost grows with `batch × heads × seq²`: at these short snippets it needs about a third more
-encode memory than the varlen tiers. At longer inputs the gap stops being a percentage. At a
-batch size of **256** inputs of ~300–2048 tokens, `torch_varlen` stays under 20 GB peak VRAM,
-while eager's attention scores alone (`256 × 12 heads × 2048² × 2 bytes`) exceed a 24 GB card.
-Both varlen tiers are also gated downstream at min cosine > 0.997 against the fp32
-`nomic-ai/CodeRankEmbed` reference.
+encode memory than the varlen tiers. At longer inputs the gap stops being a percentage.
+
+**Real documents at the model defaults.** 768 Python files sampled with a fixed seed from one shard
+of [`HuggingFaceTB/stack-edu`](https://huggingface.co/datasets/HuggingFaceTB/stack-edu) (730,431
+tokens after truncation), encoded with sentence-transformers at this model's default
+`max_seq_length` of 8192 and one `encode()` call per row, so the files are length-sorted before
+batching. Wall time covers the whole corpus; peak VRAM is the peak CUDA allocation, weights
+included. The largest batch row is the largest batch size that fit on each card (the 3090 Ti was
+sharing about 1 GiB with other processes).
+
+| GPU | batch size | tier | wall (768 files) | peak VRAM |
+| --- | --- | --- | --- | --- |
+| RTX 3090 Ti (sm_86) | 4 | `torch_varlen` | 6.57 s | 1,295 MiB |
+| RTX 3090 Ti (sm_86) | 4 | `flash_attn` | 6.58 s | 1,296 MiB |
+| RTX 3090 Ti (sm_86) | 4 | `eager` | 14.18 s | 12,961 MiB |
+| RTX 3090 Ti (sm_86) | 32 | `torch_varlen` | 6.37 s | 7,973 MiB |
+| RTX 3090 Ti (sm_86) | 64 | `torch_varlen` | 7.24 s | 15,659 MiB |
+| RTX 3090 Ti (sm_86) | 71 | `torch_varlen` | 7.60 s | 17,340 MiB |
+| RTX 5090 Laptop GPU (sm_120) | 4 | `torch_varlen` | 5.54 s | 1,319 MiB |
+| RTX 5090 Laptop GPU (sm_120) | 4 | `flash_attn` | 5.57 s | 1,319 MiB |
+| RTX 5090 Laptop GPU (sm_120) | 4 | `eager` | 16.27 s | 12,983 MiB |
+| RTX 5090 Laptop GPU (sm_120) | 32 | `torch_varlen` | 6.36 s | 7,995 MiB |
+| RTX 5090 Laptop GPU (sm_120) | 64 | `torch_varlen` | 7.48 s | 15,681 MiB |
+| RTX 5090 Laptop GPU (sm_120) | 73 | `torch_varlen` | 7.83 s | 17,843 MiB |
+
+At batch 4 `eager` takes 2.2× (3090 Ti) to 2.9× (5090 Laptop) as long as the varlen tiers and needs
+about ten times the memory. At batch 32 with the same files in random order, `eager` ran out of
+memory on 21 of the 24 batches (3090 Ti). Both varlen tiers are also gated downstream at min
+cosine > 0.997 against the fp32 `nomic-ai/CodeRankEmbed` reference.
 
 ## What changed vs the source repo