Spaces:
Running on CPU Upgrade
Running on CPU Upgrade
| title: Quantization picker | |
| emoji: 🧮 | |
| colorFrom: gray | |
| colorTo: gray | |
| sdk: docker | |
| app_port: 7860 | |
| pinned: false | |
| license: apache-2.0 | |
| short_description: Work out what fits on your card, and which method to use | |
| # Quantization picker | |
| An interactive version of the quantization method table from the Transformers | |
| docs. Instead of a 22×13 grid of status emoji, it answers the two questions | |
| people actually arrive with: **will this fit on my card**, and **which method | |
| should I use**. | |
| ## Screens | |
| | Route | What it does | | |
| | --- | --- | | |
| | `/` | Pick a model, a card, and a goal. Shows total memory footprint against your VRAM, with every precision marked on one bar. | | |
| | `/compare` | The methods that match, plotted by when they quantize and whether the result can be saved, plus a capability table. | | |
| ## The memory model | |
| Figures are computed in `lib/data.ts`, not hardcoded: | |
| ``` | |
| weights = params × bits / 8 # decimal GB | |
| kv_cache = 2 × layers × kv_heads × head_dim × 2 × context_tokens | |
| total = weights + kv_cache + activations | |
| ``` | |
| Two things this makes visible that a bits-only table cannot: | |
| - **KV cache is a fixed reservation.** It does not shrink when you quantize | |
| weights, so it is drawn as its own block against the capacity line. | |
| - **Fitting is not the same as being a good idea.** 8-bit often clears the line | |
| with only a few GB spare, which will not survive a longer context. | |
| `ACTIVATIONS_GB` is a flat 1.5 GB allowance and context is fixed at 8k. Both are | |
| estimates — see Caveats. | |
| ## Design | |
| Vercel's system (`#fafafa` canvas, hairline rings, 6px radius, Geist Sans with | |
| Geist Mono reserved for labels and metadata) with four accent colours borrowed | |
| from Cursor used only to identify precisions: | |
| | | | | |
| | --- | --- | | |
| | 2-bit | Crimson `#cf2d56` | | |
| | 3-bit | Ember `#f54e00` | | |
| | 4-bit | Forest `#34785c` | | |
| | 8-bit | Amber `#c08532` | | |
| Fit and overflow are encoded as a **tonal value shift** — solid obsidian for what | |
| fits, graphite for the KV reservation, hairline grey for what spills — rather | |
| than red/green, so the chart reads without relying on colour. | |
| ## Caveats | |
| - Capability flags mirror the current docs table, including its gaps. `?` means | |
| genuinely unverified upstream, not "not yet filled in". | |
| - Config snippets in the comparison table need checking against the installed | |
| Transformers version before anyone copies them. | |
| - Setup minutes are order-of-magnitude estimates for a 70B calibration pass. | |
| - Quality at a given precision is deliberately **not** plotted — there is no | |
| trustworthy per-method number to plot. | |
| ## Running it | |
| ```bash | |
| npm install | |
| npm run dev | |
| ``` | |
| ## Hardware data | |
| `lib/hardware.ts` is scraped from the [Hugging Face hardware | |
| table](https://huggingface.co/hardware?view=table) — the Hardware and Memory | |
| columns only. 316 entries, one per memory configuration rather than per device, | |
| because an RTX 3060 is sold as both 8GB and 12GB and only one of those runs a | |
| 7B at 8-bit. Order is the table's own, most owners first. | |
| Two caveats: | |
| - **CPUs are omitted.** All 59 CPU rows list `—` for memory: they draw on | |
| system RAM, so there is no fixed pool to size a bar against. | |
| - **`backend` is derived, not scraped.** The table has no backend column, so | |
| vendor maps to `cuda` / `rocm` / `metal` / `intel`. That is coarse — a GTX | |
| 1060 and a B300 are both `cuda` — so the picker will offer a method on | |
| hardware too old to run it. Method filtering by compute capability, Metal | |
| kernel availability, and ROCm version needs a source the table does not have. | |