Spaces:
Running
Running
docs: rename cost efficiency paper to Hybrid LLM, add BibTeX references section, and update PDF paths to docs/article/
Browse files- README.md +2 -2
- cost efficienct.pdf +0 -3
- docs/{A_Benchmark_for_Multi_LLM_Routing_System.pdf β article/A_Benchmark_for_Multi_LLM_Routing_System.pdf} +0 -0
- docs/{How_to_Use_Large_Language_Models.pdf β article/How_to_Use_Large_Language_Models.pdf} +0 -0
- docs/{cost_efficiency.pdf β article/cost_efficiency.pdf} +0 -0
- docs/index.html +63 -5
README.md
CHANGED
|
@@ -64,7 +64,7 @@ InferRoute is designed for high-throughput, low routing overhead, and self-heali
|
|
| 64 |
|
| 65 |
InferRoute integrates the core routing methodologies and trade-off evaluation principles from the research paper:
|
| 66 |
> **ROUTERBENCH: A Benchmark for Multi-LLM Routing System** (by Martian / [withmartian/routerbench](https://github.com/withmartian/routerbench)).
|
| 67 |
-
> π **[Download RouterBench Paper PDF](docs/A_Benchmark_for_Multi_LLM_Routing_System.pdf)**
|
| 68 |
|
| 69 |
### 1. Mathematical Scoring & Optimization
|
| 70 |
Predictive routing is formulated as choosing a backend $m$ that maximizes the utility score for a prompt $x$:
|
|
@@ -96,7 +96,7 @@ A smart, content-aware learned router (MLP/KNN) will push the Pareto frontier to
|
|
| 96 |
|
| 97 |
InferRoute integrates the cost-performance optimization concepts from the paper:
|
| 98 |
> **FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance** (by Stanford University / arXiv:2305.05196).
|
| 99 |
-
> π **[Download FrugalGPT Paper PDF](docs/How_to_Use_Large_Language_Models.pdf)** | π **[Download
|
| 100 |
|
| 101 |
FrugalGPT identifies three main classes of cost-saving methods:
|
| 102 |
1. **Prompt Adaptation**: Reduces input tokens dynamically. In InferRoute, when routing to cheap local backends (Ollama/vLLM), the **Prompt Adapter** (`prompt_adapter.py`) automatically compresses prompt sizes by pruning few-shot examples, restoring full prompts when upgrading to premium models.
|
|
|
|
| 64 |
|
| 65 |
InferRoute integrates the core routing methodologies and trade-off evaluation principles from the research paper:
|
| 66 |
> **ROUTERBENCH: A Benchmark for Multi-LLM Routing System** (by Martian / [withmartian/routerbench](https://github.com/withmartian/routerbench)).
|
| 67 |
+
> π **[Download RouterBench Paper PDF](docs/article/A_Benchmark_for_Multi_LLM_Routing_System.pdf)**
|
| 68 |
|
| 69 |
### 1. Mathematical Scoring & Optimization
|
| 70 |
Predictive routing is formulated as choosing a backend $m$ that maximizes the utility score for a prompt $x$:
|
|
|
|
| 96 |
|
| 97 |
InferRoute integrates the cost-performance optimization concepts from the paper:
|
| 98 |
> **FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance** (by Stanford University / arXiv:2305.05196).
|
| 99 |
+
> π **[Download FrugalGPT Paper PDF](docs/article/How_to_Use_Large_Language_Models.pdf)** | π **[Download Hybrid LLM Routing Paper PDF](docs/article/cost_efficiency.pdf)** | π **[Download InferRoute Project Plan PDF](docs/InferRoute_Project_Plan.pdf)**
|
| 100 |
|
| 101 |
FrugalGPT identifies three main classes of cost-saving methods:
|
| 102 |
1. **Prompt Adaptation**: Reduces input tokens dynamically. In InferRoute, when routing to cheap local backends (Ollama/vLLM), the **Prompt Adapter** (`prompt_adapter.py`) automatically compresses prompt sizes by pruning few-shot examples, restoring full prompts when upgrading to premium models.
|
cost efficienct.pdf
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:c0ebc21073112b8f8e51c7ea279eeac4d5b62be88c69a5cd2bccc2d58ec28cbb
|
| 3 |
-
size 4833670
|
|
|
|
|
|
|
|
|
|
|
|
docs/{A_Benchmark_for_Multi_LLM_Routing_System.pdf β article/A_Benchmark_for_Multi_LLM_Routing_System.pdf}
RENAMED
|
File without changes
|
docs/{How_to_Use_Large_Language_Models.pdf β article/How_to_Use_Large_Language_Models.pdf}
RENAMED
|
File without changes
|
docs/{cost_efficiency.pdf β article/cost_efficiency.pdf}
RENAMED
|
File without changes
|
docs/index.html
CHANGED
|
@@ -785,22 +785,22 @@
|
|
| 785 |
<h2>π Original Research Papers & Download Hub</h2>
|
| 786 |
<p>Read or download the full research publications and project blueprints associated with this routing engine:</p>
|
| 787 |
<div class="paper-grid" style="display: grid; grid-template-columns: repeat(2, 1fr); gap: 1rem; margin-top: 1.5rem;">
|
| 788 |
-
<a href="A_Benchmark_for_Multi_LLM_Routing_System.pdf" download class="sub-card" style="display: flex; flex-direction: column; text-decoration: none; transition: all 0.3s; padding: 1.5rem; background: rgba(0, 0, 0, 0.02); border: 1px solid var(--border-color); border-radius: 12px; color: var(--text-primary);">
|
| 789 |
<span style="font-size: 1.5rem; margin-bottom: 0.5rem;">π</span>
|
| 790 |
<span style="font-family: var(--font-outfit); font-weight: 700; font-size: 1.1rem; color: var(--text-primary); margin-bottom: 0.25rem;">RouterBench Paper</span>
|
| 791 |
<span style="font-size: 0.8rem; color: var(--text-secondary); margin-bottom: 0.75rem;">A Benchmark for Multi-LLM Routing System (Martian, 2024)</span>
|
| 792 |
<span style="align-self: flex-start; background: rgba(139, 92, 246, 0.15); border: 1px solid rgba(109, 40, 217, 0.2); color: var(--accent-violet); font-size: 0.75rem; font-weight: 600; padding: 0.25rem 0.65rem; border-radius: 6px;">Download PDF</span>
|
| 793 |
</a>
|
| 794 |
-
<a href="How_to_Use_Large_Language_Models.pdf" download class="sub-card" style="display: flex; flex-direction: column; text-decoration: none; transition: all 0.3s; padding: 1.5rem; background: rgba(0, 0, 0, 0.02); border: 1px solid var(--border-color); border-radius: 12px; color: var(--text-primary);">
|
| 795 |
<span style="font-size: 1.5rem; margin-bottom: 0.5rem;">π</span>
|
| 796 |
<span style="font-family: var(--font-outfit); font-weight: 700; font-size: 1.1rem; color: var(--text-primary); margin-bottom: 0.25rem;">FrugalGPT Paper</span>
|
| 797 |
<span style="font-size: 0.8rem; color: var(--text-secondary); margin-bottom: 0.75rem;">How to Use Large Language Models While Reducing Cost and Improving Performance (Stanford, 2023)</span>
|
| 798 |
<span style="align-self: flex-start; background: rgba(139, 92, 246, 0.15); border: 1px solid rgba(109, 40, 217, 0.2); color: var(--accent-violet); font-size: 0.75rem; font-weight: 600; padding: 0.25rem 0.65rem; border-radius: 6px;">Download PDF</span>
|
| 799 |
</a>
|
| 800 |
-
<a href="cost_efficiency.pdf" download class="sub-card" style="display: flex; flex-direction: column; text-decoration: none; transition: all 0.3s; padding: 1.5rem; background: rgba(0, 0, 0, 0.02); border: 1px solid var(--border-color); border-radius: 12px; color: var(--text-primary);">
|
| 801 |
<span style="font-size: 1.5rem; margin-bottom: 0.5rem;">π</span>
|
| 802 |
-
<span style="font-family: var(--font-outfit); font-weight: 700; font-size: 1.1rem; color: var(--text-primary); margin-bottom: 0.25rem;">
|
| 803 |
-
<span style="font-size: 0.8rem; color: var(--text-secondary); margin-bottom: 0.75rem;">Cost-
|
| 804 |
<span style="align-self: flex-start; background: rgba(139, 92, 246, 0.15); border: 1px solid rgba(109, 40, 217, 0.2); color: var(--accent-violet); font-size: 0.75rem; font-weight: 600; padding: 0.25rem 0.65rem; border-radius: 6px;">Download PDF</span>
|
| 805 |
</a>
|
| 806 |
<a href="InferRoute_Project_Plan.pdf" download class="sub-card" style="display: flex; flex-direction: column; text-decoration: none; transition: all 0.3s; padding: 1.5rem; background: rgba(0, 0, 0, 0.02); border: 1px solid var(--border-color); border-radius: 12px; color: var(--text-primary);">
|
|
@@ -811,6 +811,64 @@
|
|
| 811 |
</a>
|
| 812 |
</div>
|
| 813 |
</div>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 814 |
</div>
|
| 815 |
|
| 816 |
<!-- Tab 2: Architecture -->
|
|
|
|
| 785 |
<h2>π Original Research Papers & Download Hub</h2>
|
| 786 |
<p>Read or download the full research publications and project blueprints associated with this routing engine:</p>
|
| 787 |
<div class="paper-grid" style="display: grid; grid-template-columns: repeat(2, 1fr); gap: 1rem; margin-top: 1.5rem;">
|
| 788 |
+
<a href="article/A_Benchmark_for_Multi_LLM_Routing_System.pdf" download class="sub-card" style="display: flex; flex-direction: column; text-decoration: none; transition: all 0.3s; padding: 1.5rem; background: rgba(0, 0, 0, 0.02); border: 1px solid var(--border-color); border-radius: 12px; color: var(--text-primary);">
|
| 789 |
<span style="font-size: 1.5rem; margin-bottom: 0.5rem;">π</span>
|
| 790 |
<span style="font-family: var(--font-outfit); font-weight: 700; font-size: 1.1rem; color: var(--text-primary); margin-bottom: 0.25rem;">RouterBench Paper</span>
|
| 791 |
<span style="font-size: 0.8rem; color: var(--text-secondary); margin-bottom: 0.75rem;">A Benchmark for Multi-LLM Routing System (Martian, 2024)</span>
|
| 792 |
<span style="align-self: flex-start; background: rgba(139, 92, 246, 0.15); border: 1px solid rgba(109, 40, 217, 0.2); color: var(--accent-violet); font-size: 0.75rem; font-weight: 600; padding: 0.25rem 0.65rem; border-radius: 6px;">Download PDF</span>
|
| 793 |
</a>
|
| 794 |
+
<a href="article/How_to_Use_Large_Language_Models.pdf" download class="sub-card" style="display: flex; flex-direction: column; text-decoration: none; transition: all 0.3s; padding: 1.5rem; background: rgba(0, 0, 0, 0.02); border: 1px solid var(--border-color); border-radius: 12px; color: var(--text-primary);">
|
| 795 |
<span style="font-size: 1.5rem; margin-bottom: 0.5rem;">π</span>
|
| 796 |
<span style="font-family: var(--font-outfit); font-weight: 700; font-size: 1.1rem; color: var(--text-primary); margin-bottom: 0.25rem;">FrugalGPT Paper</span>
|
| 797 |
<span style="font-size: 0.8rem; color: var(--text-secondary); margin-bottom: 0.75rem;">How to Use Large Language Models While Reducing Cost and Improving Performance (Stanford, 2023)</span>
|
| 798 |
<span style="align-self: flex-start; background: rgba(139, 92, 246, 0.15); border: 1px solid rgba(109, 40, 217, 0.2); color: var(--accent-violet); font-size: 0.75rem; font-weight: 600; padding: 0.25rem 0.65rem; border-radius: 6px;">Download PDF</span>
|
| 799 |
</a>
|
| 800 |
+
<a href="article/cost_efficiency.pdf" download class="sub-card" style="display: flex; flex-direction: column; text-decoration: none; transition: all 0.3s; padding: 1.5rem; background: rgba(0, 0, 0, 0.02); border: 1px solid var(--border-color); border-radius: 12px; color: var(--text-primary);">
|
| 801 |
<span style="font-size: 1.5rem; margin-bottom: 0.5rem;">π</span>
|
| 802 |
+
<span style="font-family: var(--font-outfit); font-weight: 700; font-size: 1.1rem; color: var(--text-primary); margin-bottom: 0.25rem;">Hybrid LLM Routing Paper</span>
|
| 803 |
+
<span style="font-size: 0.8rem; color: var(--text-secondary); margin-bottom: 0.75rem;">Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing (IBM / Tsinghua, 2024)</span>
|
| 804 |
<span style="align-self: flex-start; background: rgba(139, 92, 246, 0.15); border: 1px solid rgba(109, 40, 217, 0.2); color: var(--accent-violet); font-size: 0.75rem; font-weight: 600; padding: 0.25rem 0.65rem; border-radius: 6px;">Download PDF</span>
|
| 805 |
</a>
|
| 806 |
<a href="InferRoute_Project_Plan.pdf" download class="sub-card" style="display: flex; flex-direction: column; text-decoration: none; transition: all 0.3s; padding: 1.5rem; background: rgba(0, 0, 0, 0.02); border: 1px solid var(--border-color); border-radius: 12px; color: var(--text-primary);">
|
|
|
|
| 811 |
</a>
|
| 812 |
</div>
|
| 813 |
</div>
|
| 814 |
+
|
| 815 |
+
<!-- Academic Bibliography & References -->
|
| 816 |
+
<div class="glass-card" style="margin-top: 2rem;">
|
| 817 |
+
<h2>π Academic Bibliography & References</h2>
|
| 818 |
+
<p>Formal scientific citations for the core research papers referenced during the design and optimization of the InferRoute gateway:</p>
|
| 819 |
+
|
| 820 |
+
<div style="display: flex; flex-direction: column; gap: 1.5rem; margin-top: 1.5rem;">
|
| 821 |
+
|
| 822 |
+
<div class="sub-card" style="background: rgba(0,0,0,0.01);">
|
| 823 |
+
<h4 style="font-family: var(--font-outfit); font-weight: 700; color: var(--text-primary); margin-bottom: 0.5rem;">1. FrugalGPT (Stanford University)</h4>
|
| 824 |
+
<p style="font-size: 0.9rem; color: var(--text-secondary); margin-bottom: 0.75rem; font-style: italic;">
|
| 825 |
+
Chen, L., Zaharia, M., & Zou, J. (2023). FrugalGPT: How to use large language models while reducing cost and improving performance. <em>arXiv preprint arXiv:2305.05196</em>.
|
| 826 |
+
</p>
|
| 827 |
+
<details style="cursor: pointer; font-size: 0.85rem;">
|
| 828 |
+
<summary style="color: var(--accent-violet); font-weight: 600; outline: none; margin-bottom: 0.5rem;">Show BibTeX Citation</summary>
|
| 829 |
+
<pre style="background: rgba(0,0,0,0.03); padding: 0.75rem; border-radius: 6px; border: 1px solid var(--border-color); overflow-x: auto; font-family: monospace; color: var(--text-secondary);">@article{chen2023frugalgpt,
|
| 830 |
+
title={FrugalGPT: How to use large language models while reducing cost and improving performance},
|
| 831 |
+
author={Chen, Lingjiao and Zaharia, Matei and Zou, James},
|
| 832 |
+
journal={arXiv preprint arXiv:2305.05196},
|
| 833 |
+
year={2023}
|
| 834 |
+
}</pre>
|
| 835 |
+
</details>
|
| 836 |
+
</div>
|
| 837 |
+
|
| 838 |
+
<div class="sub-card" style="background: rgba(0,0,0,0.01);">
|
| 839 |
+
<h4 style="font-family: var(--font-outfit); font-weight: 700; color: var(--text-primary); margin-bottom: 0.5rem;">2. RouterBench (Martian)</h4>
|
| 840 |
+
<p style="font-size: 0.9rem; color: var(--text-secondary); margin-bottom: 0.75rem; font-style: italic;">
|
| 841 |
+
Li, T., Martian Team, et al. (2024). RouterBench: A Benchmark for Multi-LLM Routing System. <em>arXiv preprint arXiv:2403.11164</em>.
|
| 842 |
+
</p>
|
| 843 |
+
<details style="cursor: pointer; font-size: 0.85rem;">
|
| 844 |
+
<summary style="color: var(--accent-violet); font-weight: 600; outline: none; margin-bottom: 0.5rem;">Show BibTeX Citation</summary>
|
| 845 |
+
<pre style="background: rgba(0,0,0,0.03); padding: 0.75rem; border-radius: 6px; border: 1px solid var(--border-color); overflow-x: auto; font-family: monospace; color: var(--text-secondary);">@article{li2024routerbench,
|
| 846 |
+
title={RouterBench: A Benchmark for Multi-LLM Routing System},
|
| 847 |
+
author={Li, Teh-Hsien and others},
|
| 848 |
+
journal={arXiv preprint arXiv:2403.11164},
|
| 849 |
+
year={2024}
|
| 850 |
+
}</pre>
|
| 851 |
+
</details>
|
| 852 |
+
</div>
|
| 853 |
+
|
| 854 |
+
<div class="sub-card" style="background: rgba(0,0,0,0.01);">
|
| 855 |
+
<h4 style="font-family: var(--font-outfit); font-weight: 700; color: var(--text-primary); margin-bottom: 0.5rem;">3. Hybrid LLM Routing (IBM / Tsinghua)</h4>
|
| 856 |
+
<p style="font-size: 0.9rem; color: var(--text-secondary); margin-bottom: 0.75rem; font-style: italic;">
|
| 857 |
+
Ding, J., et al. (2024). Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing. <em>arXiv preprint arXiv:2404.14944</em>.
|
| 858 |
+
</p>
|
| 859 |
+
<details style="cursor: pointer; font-size: 0.85rem;">
|
| 860 |
+
<summary style="color: var(--accent-violet); font-weight: 600; outline: none; margin-bottom: 0.5rem;">Show BibTeX Citation</summary>
|
| 861 |
+
<pre style="background: rgba(0,0,0,0.03); padding: 0.75rem; border-radius: 6px; border: 1px solid var(--border-color); overflow-x: auto; font-family: monospace; color: var(--text-secondary);">@article{ding2024hybrid,
|
| 862 |
+
title={Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing},
|
| 863 |
+
author={Ding, Jiayi and others},
|
| 864 |
+
journal={arXiv preprint arXiv:2404.14944},
|
| 865 |
+
year={2024}
|
| 866 |
+
}</pre>
|
| 867 |
+
</details>
|
| 868 |
+
</div>
|
| 869 |
+
|
| 870 |
+
</div>
|
| 871 |
+
</div>
|
| 872 |
</div>
|
| 873 |
|
| 874 |
<!-- Tab 2: Architecture -->
|