Download docs/benchmarking/getting-started.mdx from shimen/yapapaway: direct link, hf CLI and curl.
- Browser
- Download file 3.32 kB
-
https://huggingface.co/spaces/shimen/yapapaway/resolve/main/docs/benchmarking/getting-started.mdx
- Command line
-
hf download hf://spaces/shimen/yapapaway/docs/benchmarking/getting-started.mdx
-
curl -L -o getting-started.mdx https://huggingface.co/spaces/shimen/yapapaway/resolve/main/docs/benchmarking/getting-started.mdx
3.32 kB
| --- | |
| title: "Getting Started" | |
| description: "Introduction to Bifrost's performance capabilities and how to choose the right instance size for your workload." | |
| icon: "rocket" | |
| --- | |
| ## Overview | |
| Bifrost has been rigorously tested under high load conditions to ensure optimal performance for production deployments. Our benchmark tests demonstrate exceptional performance characteristics at **5,000 requests per second (RPS)** across different AWS EC2 instance types. | |
| **Key Performance Highlights:** | |
| - **Perfect Success Rate**: 100% request success rate under high load | |
| - **Minimal Overhead**: Less than 15µs added latency per request on average | |
| - **Efficient Queue Management**: Sub-microsecond queue wait times on optimized instances | |
| - **Fast Key Selection**: Near-instantaneous weighted API key selection (~10 ns) | |
| --- | |
| ## Test Environment Summary | |
| Bifrost was benchmarked on two primary AWS EC2 instance configurations: | |
| ### **t3.medium (2 vCPUs, 4GB RAM)** | |
| - **Buffer Size**: 15,000 | |
| - **Initial Pool Size**: 10,000 | |
| - **Use Case**: Cost-effective option for moderate workloads | |
| ### **t3.xlarge (4 vCPUs, 16GB RAM)** | |
| - **Buffer Size**: 20,000 | |
| - **Initial Pool Size**: 15,000 | |
| - **Use Case**: High-performance option for demanding workloads | |
| --- | |
| ## Performance Comparison at a Glance | |
| | Metric | t3.medium | t3.xlarge | Improvement | | |
| |--------|-----------|-----------|-------------| | |
| | **Success Rate @ 5k RPS** | 100% | 100% | No failed requests | | |
| | **Bifrost Overhead** | 59 µs | 11 µs | **-81%** | | |
| | **Average Latency** | 2.12s | 1.61s | **-24%** | | |
| | **Queue Wait Time** | 47.13 µs | 1.67 µs | **-96%** | | |
| | **JSON Marshaling** | 63.47 µs | 26.80 µs | **-58%** | | |
| | **Response Parsing** | 11.30 ms | 2.11 ms | **-81%** | | |
| | **Peak Memory Usage** | 1,312.79 MB | 3,340.44 MB | +155% | | |
| > **Note**: t3.xlarge tests used significantly larger response payloads (~10 KB vs ~1 KB), yet still achieved better performance metrics. | |
| <Note> | |
| All benchmarks are on mocked OpenAI calls, whose latency and payload size are mentioned in the respective analysis pages. | |
| </Note> | |
| --- | |
| ## Configuration Flexibility | |
| One of Bifrost's key strengths is its **configuration flexibility**. You can fine-tune the speed ↔ memory trade-off based on your specific requirements: | |
| | Configuration Parameter | Effect | | |
| |------------------------|--------| | |
| | `initial_pool_size` | Higher values = faster performance, more memory usage | | |
| | `buffer_size` & `concurrency` | Controls queue depth and max parallel workers (per provider) | | |
| | `retry` & `timeout` | Tune aggressiveness for each provider to meet your SLOs | | |
| **Configuration Philosophy:** | |
| - **Higher settings** (like t3.xlarge profile) prioritize raw speed | |
| - **Lower settings** (like t3.medium profile) optimize for memory efficiency | |
| - **Custom tuning** lets you find the sweet spot for your specific workload | |
| --- | |
| ## Next Steps | |
| ### **Detailed Performance Analysis** | |
| - **[t3.medium Performance](./t3.medium)** - Deep dive into cost-effective performance | |
| - **[t3.xlarge Performance](./t3.xl)** - High-performance configuration analysis | |
| ### **Run Your Own Tests** | |
| - **[Run Your Own Benchmarks](./run-your-own-benchmarks)** - Step-by-step guide to benchmark Bifrost in your environment | |
| Ready to dive deeper? Choose your instance type above or learn how to run your own performance tests. | |