Blame for index.html

<!doctype html>
<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1">
<title>PolyStrata</title>
<style>
body{margin:0;background:#0f1116;color:#e6e6e6;font:16px/1.55 system-ui,-apple-system,Segoe UI,Roboto,sans-serif}
main{max-width:960px;margin:0 auto;padding:32px 20px 60px}
h1{font-size:2rem;margin:0 0 6px} h2{font-size:1.3rem;margin:36px 0 10px;border-bottom:1px solid #2a2e3a;padding-bottom:6px}
h3{font-size:1.05rem;margin:22px 0 8px;color:#c9d1e0}
p{margin:10px 0} a{color:#8ab4ff} img{max-width:100%;border-radius:8px;display:block}
table{border-collapse:collapse;width:100%;font-size:15px} th,td{text-align:left;padding:6px 8px;border-bottom:1px solid #2a2e3a} th{color:#aab}
.sub{color:#9aa3b2} code{background:#1b1f2a;padding:1px 5px;border-radius:4px}
pre{background:#1b1f2a;padding:10px 12px;border-radius:8px;overflow-x:auto}
.btn{display:inline-block;background:#2f6fed;color:#fff;padding:10px 16px;border-radius:8px;text-decoration:none;margin:8px 8px 8px 0}
</style></head><body><main>
<h1>PolyStrata</h1>
<p class="sub">Large mixture-of-experts models on the graphics card you already have. Open source, MIT, built on <a href="https://github.com/Niko1221/Strata">Strata</a>.</p>
<p><a class="btn" href="PolyStrata-b0e3b57a.zip">Download the source (6.6 MB, commit b0e3b57a)</a></p>
<p>Or clone the full repository with its history and tags from this Space (install <a href="https://git-lfs.com/">git-lfs</a> first, the binaries are stored with it):</p>
<pre><code>git clone -b source https://huggingface.co/spaces/VecSzn/PolyStrata</code></pre>
<p class="sub"><b>12 October:</b> the GitHub pages of this project (github.com/VecSzn/PolyStrata, its releases and issues) are not visible at the moment: GitHub has put the account under review, and an appeal is in. The code is unchanged and is here meanwhile. The zip above is the source at the latest release, 0.1.43.1-poly.2, without the benchmark records and the media files. Unpack it and run <code>START-HERE.bat</code> on Windows or <code>./setup.sh</code> on Linux. The ready-made Windows engine (<code>strata-poly-windows-x64-bb7d247465918148.zip</code>) will be put next to it; until then setup compiles the engine on your PC, which needs Visual Studio's build tools and the CUDA toolkit (setup offers to install them). Once the engine zip is here, set <code>STRATA_POLY_PREBUILT_URL=https://huggingface.co/spaces/VecSzn/PolyStrata/resolve/main/</code> before running setup and it downloads it from here instead of GitHub.</p>
<p>A mixture-of-experts model only uses a few of its experts for each token. PolyStrata keeps the experts a model uses most on the GPU and computes the rest on the CPU from RAM, so a 35B model runs next to an 8 GB card and 32 GB of RAM. It reads the same GGUF files as llama.cpp, and its output is checked against llama.cpp token by token.</p>
<h2>Same laptop, same model file, same request</h2>
<p>PolyStrata on the left, llama.cpp b11540 on the right, on a laptop with an RTX 4070 (8 GB) and 32 GB of RAM, Qwen3.6-35B-A3B UD-Q3_K_M. On this laptop PolyStrata reads prompts 2.7 to 3.5 times as fast as llama.cpp and writes 1.5 to 1.8 times as fast; the comparison script is in the repository.</p>
<img src="llamacpp-comparison.gif" alt="PolyStrata and llama.cpp answering the same request on the same laptop">
<h2>Tokens per second</h2>
<img src="speed.png" alt="Tokens per second while writing an answer, laptop and workstation">
<h3>Laptop: RTX 4070 (8 GB), Core i7-14700HX, 32 GB RAM</h3>
<table><tr><th>Model</th><th>Size</th><th>Writes, tokens/s</th><th>Reads a 10K prompt, tokens/s</th></tr>
<tr><td>Qwen3.6-35B-A3B</td><td>UD-Q3_K_M</td><td>94-100</td><td>1,690</td></tr>
<tr><td>Qwen3.6-35B-A3B</td><td>UD-Q2_K_XL</td><td>123-124</td><td>1,590</td></tr>
<tr><td>Qwen3.6-35B-A3B</td><td>UD-Q4_K_M</td><td>82-86</td><td>1,560-1,570</td></tr>
<tr><td>Ornith 1.5</td><td>APEX-I-COMPACT</td><td>76-82</td><td>1,620</td></tr>
<tr><td>Ornith 1.5</td><td>APEX-I-MINI</td><td>80-86</td><td>1,550-1,560</td></tr>
<tr><td>Ornith 1.5</td><td>APEX-I-QUALITY</td><td>59-63</td><td>1,570-1,580</td></tr>
<tr><td>Gemma 4 26B-A4B</td><td>UD-Q3_K_M</td><td>92-93</td><td>2,240</td></tr>
<tr><td>Gemma 4 26B-A4B</td><td>UD-Q2_K_XL</td><td>101-104</td><td>2,150-2,160</td></tr>
<tr><td>Gemma 4 26B-A4B</td><td>UD-Q4_K_M</td><td>85</td><td>2,150</td></tr>
<tr><td>Tiel Coder 35B-A3B</td><td>UD-Q3_K_XL</td><td>83</td><td>1,700</td></tr>
<tr><td>Tiel Coder 35B-A3B</td><td>UD-IQ3_XXS</td><td>96-97</td><td>1,610</td></tr>
<tr><td>Tiel Coder 35B-A3B</td><td>UD-Q4_K_XL</td><td>74-79</td><td>1,660-1,670</td></tr>
</table>
<h3>Workstation: RTX 5090 (32 GB), Xeon Platinum 8470Q, 120 GB RAM</h3>
<table><tr><th>Model</th><th>Size</th><th>Writes, tokens/s</th><th>Reads a 10K prompt, tokens/s</th></tr>
<tr><td>GLM 5.3 Flash</td><td>3.0BIT</td><td>59-66</td><td>820</td></tr>
<tr><td>GLM 5.3 Flash</td><td>UD-IQ2_XXS</td><td>50-52</td><td>1,060-1,080</td></tr>
<tr><td>GLM 5.3 Flash</td><td>REAP50-Q3_K_M</td><td>59-66</td><td>1,300-1,310</td></tr>
<tr><td>Qwen3.6-35B-A3B</td><td>UD-Q3_K_M</td><td>394-395</td><td>6,350-6,550</td></tr>
<tr><td>Qwen3.6-35B-A3B</td><td>UD-Q2_K_XL</td><td>450</td><td>6,400-6,590</td></tr>
<tr><td>Qwen3.6-35B-A3B</td><td>UD-Q4_K_M</td><td>363-401</td><td>6,030-6,690</td></tr>
<tr><td>Ornith 1.5</td><td>APEX-I-COMPACT</td><td>316-319</td><td>6,830-6,860</td></tr>
<tr><td>Ornith 1.5</td><td>APEX-I-MINI</td><td>289-300</td><td>6,300-6,380</td></tr>
<tr><td>Ornith 1.5</td><td>APEX-I-QUALITY</td><td>289-298</td><td>6,040-6,390</td></tr>
<tr><td>Gemma 4 26B-A4B</td><td>UD-Q3_K_M</td><td>353-356</td><td>7,880-8,230</td></tr>
<tr><td>Gemma 4 26B-A4B</td><td>UD-Q2_K_XL</td><td>327-340</td><td>7,380-8,000</td></tr>
<tr><td>Gemma 4 26B-A4B</td><td>UD-Q4_K_M</td><td>335-346</td><td>7,040-8,050</td></tr>
<tr><td>Tiel Coder 35B-A3B</td><td>UD-Q3_K_XL</td><td>337-345</td><td>6,570-6,830</td></tr>
<tr><td>Tiel Coder 35B-A3B</td><td>UD-IQ3_XXS</td><td>339-351</td><td>6,090-6,520</td></tr>
<tr><td>Tiel Coder 35B-A3B</td><td>UD-Q4_K_XL</td><td>316-322</td><td>6,750-6,870</td></tr>
</table>
<p class="sub">GLM 5.3 Flash is a 79 to 114 GB file and needs about 100 GB of RAM beside a 32 GB card. Qwen3.8-Flash-Next runs on Strata's own engine; its numbers are in the README.</p>
<h2>Works with Claude Code</h2>
<p>The server speaks the OpenAI and Anthropic APIs on localhost, so Claude Code, Codex and other apps run against the local model. This is Claude Code on the laptop model, in real time:</p>
<img src="claude-code-demo.gif" alt="Claude Code working against the local model">
<h2>Install</h2>
<p>Windows: download the source from the <a href="https://github.com/VecSzn/PolyStrata/releases/latest">latest release</a> and run <code>START-HERE.bat</code>; it installs a ready-made engine and the model. Linux: <code>./setup.sh</code> builds the engine. AMD cards work on Linux as a beta. Details, every option and the community results are in the README inside the zip (docs/ has the rest).</p>
</main></body></html>