|
Download plans/performance-agentic-roadmap.md from shimen/API: direct link, hf CLI and curl.
- Browser
- Download file 3.41 kB
-
https://huggingface.co/spaces/shimen/API/resolve/main/plans/performance-agentic-roadmap.md
- Command line
-
hf download hf://spaces/shimen/API/plans/performance-agentic-roadmap.md
-
curl -L -o performance-agentic-roadmap.md https://huggingface.co/spaces/shimen/API/resolve/main/plans/performance-agentic-roadmap.md
3.41 kB
| # Performance & Agentic Roadmap for CLIProxyAPI | |
| This roadmap focuses on optimizing the server for high-performance single-user usage (low latency, high throughput) and enhancing agentic capabilities (tool use, reasoning, debugging). | |
| ## Phase 1: Core Performance Optimization | |
| ### 1.1 HTTP Client Reuse (Critical) | |
| **Problem:** Currently, handlers create a new `http.Client` for every request. This disables TCP connection pooling (Keep-Alive), causing unnecessary TLS handshakes and increasing latency. | |
| **Solution:** | |
| - Create a global `*http.Client` in `main.go` with optimized transport settings. | |
| - Inject this client into all handlers. | |
| ```go | |
| // Recommended Transport Settings | |
| t := &http.Transport{ | |
| MaxIdleConns: 100, | |
| MaxIdleConnsPerHost: 20, | |
| IdleConnTimeout: 90 * time.Second, | |
| DisableCompression: true, // Proxy should often pass through raw bytes | |
| } | |
| ``` | |
| ### 1.2 Memory Optimization (sync.Pool) | |
| **Problem:** JSON translation involves allocating new byte slices for every request/response body. | |
| **Solution:** | |
| - Implement `sync.Pool` for byte buffers used in the `translator` package. | |
| - Reuse buffers for reading request bodies and constructing responses. | |
| ### 1.3 Asynchronous Logging | |
| **Problem:** Logging might be blocking the request path. | |
| **Solution:** | |
| - Ensure `logrus` or the custom logger is writing asynchronously or to a buffered channel to avoid I/O blocking on the main request thread. | |
| ## Phase 2: Agentic Capabilities & Tooling | |
| ### 2.1 Unified Tool Abstraction | |
| **Current State:** Tool translation is handled point-to-point (e.g., Gemini->OpenAI). | |
| **Goal:** Create a standardized `ToolDefinition` struct in `sdk` that acts as an intermediate representation (IR). | |
| **Benefit:** | |
| - Easier to add new providers (Ollama, DeepSeek, etc.). | |
| - Write tools once, run on any provider. | |
| ### 2.2 "Agentic Trace" Debugging | |
| **Goal:** When using CLI tools (like Cline/RooCode), it's hard to see *why* a tool call failed. | |
| **Solution:** | |
| - Add a generic `X-Agent-Trace-ID` header. | |
| - Create a specific "Trace" log level that captures the *exact* JSON sent to and from the upstream provider for tool calls. | |
| - Expose a simple `/v1/trace/{id}` endpoint to view the "thought process" and tool outputs. | |
| ### 2.3 Enhanced "Thinking" Support | |
| **Goal:** Maximize the reasoning capabilities of models like Claude 3.7 and OpenAI o1. | |
| **Actions:** | |
| - Ensure `internal/thinking` supports "Budget" parameters for all providers (currently seems focused on specific ones). | |
| - Add support for "Thought Blocks" parsing in the stream to separate "reasoning" from "final answer" for clients that don't support it natively. | |
| ## Phase 3: Architecture & Maintainability | |
| ### 3.1 Refactor `main.go` | |
| **Problem:** The entry point is too complex (`God Function`). | |
| **Solution:** | |
| - Extract server initialization into `internal/bootstrap`. | |
| - Move configuration loading to `internal/config/loader.go`. | |
| ### 3.2 Security Hardening (Architectural) | |
| **Action:** | |
| - Externalize all hardcoded OAuth secrets to `config.yaml`. | |
| - Implement a simple "Allowlist" for the Management API's `APICall` to prevent SSRF, even in a private network (defense in depth). | |
| ## Implementation Priority | |
| 1. **Refactor HTTP Client** (Highest Impact/Effort ratio). | |
| 2. **Refactor `main.go`** (Makes future changes easier). | |
| 3. **Agentic Trace Logging** (High value for debugging "smart" agents). | |
| 4. **Buffer Pools** (Micro-optimization, do last). | |