Spaces:
Paused
Paused
Commit History
-
refactor(quota-viewer): 🔨 enhance credential sorting and cooldown display
MirrowelAdd natural/numeric sorting for credentials to ensure proper ordering (e.g., proj-1, proj-2, proj-10 instead of proj-1, proj-10, proj-2). Improve cooldown display in quota viewer by grouping cooldowns by quota groups when available, providing clearer visibility into rate limiting status for grouped models. Also in this commit: - refactor(client): improve model stats lookup with alias support - feat(usage-manager): add quota display formatting for logging1af1879 -
refactor(usage): 🔨 sync request_count across quota group models
MirrowelRefactor quota group handling to synchronize request_count across all models in a quota group instead of recalculating it on each access. This improves performance and ensures consistency. The previous implementation iterated through all models in a group to sum request counts and find baselines on each quota check. The new implementation proactively syncs these values when requests are recorded. Changes: - Sync request_count to all models in quota group when recording successes, failures, or updating quota baselines - Quota estimation now reads values from any representative model since all models in the group are guaranteed to have synchronized data - Add credential filename parsing support for OAuth files without paths Also in this commit: - chore(gitignore): ignore quota_viewer_config.json1ac7bd0 -
feat(usage): ✨ use request count for credential selection when failed requests consume quota
MirrowelFor providers like antigravity where failed requests also consume quota, credential selection now uses `request_count` instead of `success_count`. This ensures proper credential rotation when quota is consumed by failed requests. - Add `_REQUEST_COUNT_PROVIDERS` set to identify affected providers - Update `_get_grouped_usage_count` to dynamically select usage field based on provider - Extend `_get_usage_count` signature to support configurable usage fields48b6b15 -
feat(quota-viewer): ✨ add quota and usage statistics viewer system
MirrowelThis introduces a comprehensive quota and usage statistics monitoring system with API endpoints and a rich terminal UI for real-time credential consumption visibility. Key components: - Add GET/POST /v1/quota-stats API endpoints to main proxy for retrieving and refreshing statistics - Implement full-featured TUI viewer (quota_viewer.py) with provider summaries, credential details, progress bars, and remote management - Add configuration system (quota_viewer_config.py) supporting multiple remote proxies with API key management - Extend RotatingClient with get_quota_stats(), reload_usage_from_disk(), and force_refresh_quota() methods - Enhance usage_manager with get_stats_for_endpoint() for aggregated statistics - Integrate viewer into launcher TUI as new menu option 5 Features include toggle between current and global/lifetime stats, token formatting, cooldown timers, quota group visualization, and estimated cost tracking. Supports Antigravity quota group enrichment and live API refre…8b4ff52 -
feat(antigravity): ✨ add credential availability tracking for models
MirrowelAdd get_available_credentials_for_model method to filter credentials not on cooldown. Update client logging to display available credentials count alongside total count.818ba14 -
feat(antigravity): ✨ add quota tracking with independent background jobs
MirrowelImplement comprehensive quota tracking for the Antigravity provider with automated background refresh. The system fetches quota baselines from the API, tracks request counts locally, and estimates remaining quota per credential/model. - Add AntigravityQuotaTracker mixin for quota management, cost learning, and verification methods - Extend BackgroundRefresher to support provider-specific background jobs with independent asyncio task timers - Add provider interface methods get_background_job_config() and run_background_job() for periodic task integration - Integrate UsageManager with quota baseline storage and aggregated tracking fields (request_count, failure_count) - Implement quota cost learning that persists empirical costs to learned_quota_costs.json - Define model quota groups for thinking variants (claude, gemini-2.5-flash, gemini-3-pro-preview) - Disable flawed key lockout logic that incorrectly triggered when a single quota group was exhausted2be7535
-
refactor(core): 🔨 centralize path management for PyInstaller compatibility
MirrowelIntroduce a new `paths.py` utility module that provides centralized path resolution for all data files (logs, cache, OAuth credentials, usage data) with support for both PyInstaller EXE and script/library runtime modes. - Add `get_default_root()` to auto-detect EXE directory when frozen, otherwise uses CWD - Add `get_logs_dir()`, `get_cache_dir()`, `get_oauth_dir()`, and `get_data_file()` helpers - Update `RotatingClient` to accept optional `data_dir` parameter for override capability - Refactor all hardcoded path calculations to use centralized utilities - Update `CredentialManager` to accept `oauth_dir` parameter - Convert `failure_logger` to lazy initialization with configurable logs directory - Migrate `UsageManager` to accept optional `file_path` with automatic defaults - Update all provider-specific logging directories to use centralized helpers - Refactor credential tool and settings tool to use path utilities - Update main.py to load .env files from correct root directory in b…467f294
-
feat(io): ✨ add fault-tolerant file operations with automatic recovery
MirrowelEnhances application reliability by introducing a comprehensive I/O abstraction layer that eliminates crashes from filesystem issues. The system distinguishes between critical state files (credentials, usage data) that require memory buffering with retry logic, and disposable logs that can be safely dropped on failure. Key improvements: - New `ResilientStateWriter` class maintains in-memory state for critical files with background retry mechanism on disk failure - Introduced `safe_write_json`, `safe_log_write`, and `safe_mkdir` utility functions for one-shot operations with graceful degradation - Logging subsystems (`DetailedLogger`, `failure_logger`) now drop data on disk failure to prevent memory exhaustion during streaming - Authentication providers (`GoogleOAuthBase`, `IFlowAuthBase`, `QwenAuthBase`) preserve credentials in memory when filesystem becomes unavailable - `UsageManager` delegates persistence to `ResilientStateWriter` for automatic recovery from transient failures - `…7f148b3 -
Merge origin/main into feat/runtime-resilience - Resolve conflict in usage_manager.py by integrating runtime resilience with human-readable timestamps
Mirrowel01932f9 -
feat(concurrency): ✨ add priority-based concurrency multipliers for credential tiers
MirrowelThis commit introduces a flexible priority-based concurrency multiplier system that allows higher-priority credentials (e.g., paid tiers) to handle more concurrent requests than lower-priority credentials, regardless of rotation mode. Key changes: - Added `default_priority_multipliers` and `default_sequential_fallback_multiplier` to `ProviderInterface` for provider-level configuration - Implemented multiplier lookup with mode-specific overrides via environment variables (format: `CONCURRENCY_MULTIPLIER_<PROVIDER>_PRIORITY_<N>[_<MODE>]=<multiplier>`) - Modified `UsageManager` to calculate effective concurrency limits by applying multipliers to base `MAX_CONCURRENT_REQUESTS_PER_KEY` values - Added `PriorityMultiplierManager` to `settings_tool.py` for runtime configuration and display of multipliers - Configured default multipliers for Antigravity (P1: 5x, P2: 3x, sequential fallback: 2x) and Gemini CLI (P1: 5x, P2: 3x) - Introduced `model_usage_weights` to account for models with differ…aefb706
-
feat(usage): ✨ add human-readable timestamp fields to usage data for debugging
MirrowelThis commit introduces helper methods to automatically generate and persist human-readable timestamp fields alongside Unix timestamps in the usage tracking data. - Add `_format_timestamp_local()` method to convert Unix timestamps to local time strings with timezone offset - Add `_add_readable_timestamps()` method to enrich usage data with 'window_started' and 'quota_resets' fields - Integrate timestamp formatting into the save flow, automatically updating readable fields before persisting to disk - Set `quota_reset_ts` when initializing new model windows based on provider's window configuration The readable timestamps improve observability and debugging by making it easier to understand when quota windows started and when they will reset, without requiring manual timestamp conversion.4cdd261 -
fix(resilience): complete circuit breaker patterns per PR review
Address bot review feedback on PR #32: - Add _disk_available flag update in _write_json exception handler - Add _disk_available flag update in log_stream_chunk (critical for streams) - Document intentional no-memory-fallback design for streams - Add _fallback_mode update in failure_logger exception handler - Add complete circuit breaker pattern to usage_manager5e42536
-
feat: add runtime resilience for file deletion survival
Implement graceful degradation patterns that allow the proxy to continue running even if core files are deleted during runtime. Changes only take effect on restart, enabling safe development while the proxy is serving. ## Changes by Component ### Usage Manager (usage_manager.py) - Wrap `_save_usage()` in try/except with directory auto-recreation - Enhanced `_load_usage()` with explicit error handling - In-memory state continues working if file operations fail ### Failure Logger (failure_logger.py) - Add module-level `_file_handler` and `_fallback_mode` state - Create `_create_file_handler()` with directory auto-recreation - Create `_ensure_handler_valid()` for handler recovery - Use NullHandler as fallback when file logging fails ### Detailed Logger (detailed_logger.py) - Add class-level `_disk_available` and `_console_fallback_warned` flags - Add instance-level `_in_memory_logs` list for fallback storage - Skip disk writes gracefully when filesystem unavailable ### Google OAuth B…31c3d36 -
refactor(usage): 🔨 cache provider plugin instances to reduce redundant instantiation
MirrowelIntroduced a provider instance cache (`_provider_instances`) to store and reuse provider plugin instances across multiple method calls. - Added `_get_provider_instance()` helper method to centralize provider plugin instantiation logic with caching support - Refactored `_get_usage_reset_config()`, `_get_model_quota_group()`, `_get_models_in_quota_group()`, `_get_usage_field_name()`, and cost calculation logic to use the cached provider instances - Eliminated redundant provider plugin instantiation that occurred on every method call - Simplified error handling by consolidating null checks in the helper method This change improves performance by avoiding repeated instantiation of the same provider plugin objects and reduces code duplication across provider plugin access patterns.fd01482 -
feat(usage): ✨ implement per-model quota tracking with authoritative reset timestamps
MirrowelThis commit introduces granular per-model quota tracking that supports provider-specific reset timestamps from quota exhausted errors. Key changes: - Add `quota_reset_timestamp` field to `ClassifiedError` to capture authoritative Unix timestamp from provider's quota exhausted responses - Implement per-model usage tracking mode where each model maintains its own window with `window_start_ts` and `quota_reset_ts` - Add quota group support for models that share quota limits (e.g., Claude Sonnet and Opus on Antigravity) - Parse Antigravity's `quotaResetTimeStamp` ISO format to Unix timestamp for precise reset timing - Update reset logic to prioritize authoritative `quota_reset_ts` over fallback window calculations - Distinguish between quota exhausted (sets authoritative reset time) and rate limit (transient cooldown only) - Migrate Antigravity provider to per-model tracking with 5-hour windows for paid tier and 7-day windows for free tier The per-model mode enables more accurate quota …0ca1651 -
feat(usage): ✨ add provider-specific rolling window usage tracking
MirrowelImplement flexible per-provider usage reset configurations to support different quota windows (5h rolling for Antigravity paid tiers, 7-day for free tier) instead of universal daily resets. - Add `get_usage_reset_config()` and `get_default_usage_field_name()` methods to ProviderInterface for provider-specific configuration - Implement Antigravity-specific reset config returning different windows based on credential tier (5h for paid, 7-day for free) - Refactor UsageManager to support custom usage field names ("5h_window", "weekly") instead of hardcoded "daily" - Add window start timestamp tracking that begins on first request and resets after window expiration - Extract reset logic into separate methods (`_check_window_reset`, `_check_daily_reset`) for cleaner separation - Add credential-to-provider mapping via regex pattern matching for OAuth credential paths - Archive expired window stats to "global" field matching existing daily reset behavior - Preserve unexpired cooldowns during …98f6823 -
feat(rotation): ✨ add sequential rotation mode with provider-specific quota parsing
MirrowelIntroduces a new credential rotation mode system that allows providers to choose between "balanced" (distribute load evenly) and "sequential" (use until exhausted) strategies. Sequential mode is particularly beneficial for providers with cache-preserving features like Antigravity's thinking signature caches. Key changes: - Added ROTATION_MODE_{PROVIDER} environment variable support with comprehensive documentation in .env.example - Implemented provider-specific quota error parsing for Antigravity and Gemini CLI providers, extracting retry_after from Google RPC error format (handles compound durations like "143h4m52.73s") - Extended ProviderInterface with rotation mode configuration and parse_quota_error() method - Updated UsageManager to support sequential credential selection that preserves sticky credential usage until quota exhaustion - Enhanced error_handler.py classify_error() to attempt provider-specific parsing before falling back to generic classification - Added rotation mode…bd84d38 -
feat(providers): ✨ implement credential tier initialization and persistence system
MirrowelThis commit introduces a comprehensive credential tier management system across the library, enabling automatic tier detection, persistence, and intelligent credential prioritization at startup. - Add `initialize_credentials()` method to `ProviderInterface` for startup credential loading - Add `get_credential_tier_name()` method to expose human-readable tier names for logging - Implement tier persistence in credential files via `_proxy_metadata` field - Add lazy-loading fallback for tier data when not in memory cache - Introduce `BackgroundRefresher._initialize_credentials()` to pre-load all provider tiers before refresh loop - Pass `credential_tier_names` map through client to usage_manager for enhanced logging - Update `UsageManager.acquire_key()` to display tier information in acquisition logs - Make `ModelDefinitions` a singleton to prevent duplicate loading across providers - Add comprehensive 3-line startup summary showing provider counts, credentials, and tier breakdown - Imple…4dfb828
-
Merge branch 'Antigravity' of https://github.com/Mirrowel/LLM-API-Key-Proxy into fix/quota-exceeded-global-cooldown
5a29725 -
fix(rotator): prevent quota errors from global cooldown
Resolves a critical issue where a single credential hitting quota limits triggered a provider-wide cooldown, causing denial of service for all remaining healthy credentials. **Problem:** When any credential encountered a 429 "Quota Exceeded" error, the system applied a global 60-second cooldown to the entire provider. This blocked ALL credentials from being used, even though quota errors are credential-specific, not provider-wide. **Root Cause:** The error classification system did not distinguish between: - Rate limit errors (IP/provider throttling) → affects all credentials - Quota errors (account/key limits) → affects only that credential Both were treated as `rate_limit` and triggered `cooldown_manager`, which pauses the entire provider. **Solution:** 1. **Enhanced Error Classification** (error_handler.py): - Parse RateLimitError messages for "quota"/"resource_exhausted" - Classify as `quota_exceeded` (not `rate_limit`) - Preserves retry_after headers for both types …0dd6d21 -
fix(logging): preserve full credential filenames in logs
Resolved logging truncation issue where OAuth credential filenames were being aggressively abbreviated (e.g., `...6.json` instead of `antigravity_oauth_16.json`), causing ambiguity when debugging or auditing specific credentials. **Changes:** - Enhanced `mask_credential()` utility in error_handler.py: - Now explicitly detects `.json` file extensions - Returns full basename for file paths (e.g., `antigravity_oauth_16.json`) - Maintains security by masking API keys to last 6 characters (`...xyz123`) - Replaced all manual credential truncation with centralized `mask_credential()`: - client.py: 15 instances (stream handling, retry logging, model discovery) - usage_manager.py: 16 instances (key acquisition, release, cooldown tracking) - failure_logger.py: 2 instances (failure logging and summaries) - Code quality improvements: - Fixed indentation error in client.py during refactoring - Ensured consistent, safe credential logging across entire application - Configuration…fce1762
-
feat(rotation): ✨ add configurable weighted random credential selection
MirrowelIntroduces a new `rotation_tolerance` parameter to enable weighted random credential selection as an alternative to deterministic least-used rotation. This enhancement addresses potential fingerprinting vulnerabilities while maintaining load balance. - Add `rotation_tolerance` parameter to both `RotatingClient` and `UsageManager` (default: 0.0 for backward compatibility) - Implement `_select_weighted_random()` method using weight formula: `(max_usage - credential_usage) + tolerance + 1` - Support three recommended tolerance levels: - 0.0: Deterministic least-used (existing behavior) - 3.0-4.0: Balanced randomness with good load distribution - 5.0+: High randomness for maximum unpredictability - Update credential acquisition logic to apply weighted selection within tier-based priority system - Enhance logging to indicate selection method (weighted-random vs least-used) and include usage counts - Add comprehensive docstrings explaining rotation strategy and tolerance impact - Impo…f35e0e7 -
feat(client): ✨ add credential prioritization system for tier-based model access
MirrowelImplements a comprehensive credential prioritization system that enables providers to enforce tier-based access controls and optimize credential selection based on account types. Key changes: - Added `get_credential_priority()` and `get_model_tier_requirement()` methods to ProviderInterface, allowing providers to define credential tiers and model restrictions - Enhanced UsageManager.acquire_key() to respect credential priorities, always attempting highest-priority credentials first before falling back to lower tiers - Implemented Gemini-specific tier detection in GeminiCliProvider, mapping paid tier credentials to priority 1, free tier to priority 2, and unknown to priority 10 - Added model-based filtering in RotatingClient to exclude incompatible credentials before acquisition (e.g., Gemini 3 models require paid-tier credentials) - Improved logging to show priority-aware credential selection and tier compatibility warnings The system gracefully handles unknown credential tiers by t…b5da45c
-
fix(auth): 🔨 tighten oauth refresh intervals, expand refresh buffers, improve logging and add iFlow docs
Mirrowel- lower default OAUTH_REFRESH_INTERVAL from 3600s to 600s (updated .env.example, README, and background refresher) - move background refresher sleep to after proactive refresh loop and suppress noisy info log to avoid unnecessary startup delay/noise - increase REFRESH_EXPIRY_BUFFER_SECONDS for Gemini (30 minutes) and Qwen (3 hours) - convert several lib_logger.info calls to debug and add richer error logging for HTTP failures and missing refresh tokens - add Content-Type/Accept headers for Qwen token requests and surface HTTP error bodies for easier troubleshooting - add extracted documentation: iFlow OAuth flow and iFlow JS bundle summary (docs/iflow_auth_flow_extracted.md, docs/iflow_js_bundle_summary.md)abbdf2a
-
fix(core): improve error classification and prevent false key penalization
MirrowelThis commit refines the failure handling mechanism to ensure keys are only penalized for errors directly attributable to them (e.g., authentication, sustained client errors). Key changes: - Enhanced retry logic to reliably extract cooldown durations from HTTP headers (Retry-After, X-RateLimit-Reset) and duration strings (e.g., "60s", "2m"). - Introduced logic to differentiate between key-specific failures and transient provider-level errors (server_error, api_connection). - Provider-level errors now trigger a short cooldown but skip consecutive failure incrementation, preserving the key's standing. - Rate limit errors now default to a 60s cooldown if no explicit retry duration is detected.20944a5 -
feat(core): enable configuration of maximum concurrent requests per key
MirrowelThis introduces functionality to allow multiple concurrent requests to utilize the same API key, which is necessary when a provider's capacity allows for parallel usage (e.g., modern OpenAI tiers). The `UsageManager` is updated to track concurrent request counts per model per key, moving from a simple busy/idle state to a counter. - New environment variables (`MAX_CONCURRENT_REQUESTS_PER_KEY_<PROVIDER>`) define the maximum concurrency limit for keys of a specific provider. - The default limit is 1, maintaining the previous behavior (no concurrency). - Updates provider endpoint resolution to support loading custom API bases via environment variables (e.g., `CUSTOM_API_BASE`) if the provider is not hardcoded.79e83ae
-
refactor(core): 🔨 use static class attribute for provider skipping flag
MirrowelConvert `DynamicOpenAICompatibleProvider.skip_cost_calculation` from an instance method to a static class attribute. This allows the `UsageManager` to check the property directly against the provider class definition via `getattr`, avoiding unnecessary method overhead and potential instantiation checks for a simple flag.3962356 -
feat(core): enable static model configuration and LiteLLM compatibility for custom providers
Mirrowelintroduce static model definitions and runtime parameter conversion for custom endpoints. This change significantly improves compatibility with self-hosted or dynamically configured OpenAI-compatible APIs: - **Model Definitions:** Adds a new `ModelDefinitions` utility to load static model configurations (IDs, options like `reasoning_effort`) from environment variables (e.g., `PROVIDER_MODELS`). - **Dynamic Providers:** Extends provider discovery to dynamically register new providers whenever an `_API_BASE` environment variable is detected. - **Client Conversion:** Implements `_convert_model_params_for_litellm` in the RotatingClient to rewrite the model argument (to `openai/{model_id}`) and inject the necessary `api_base` and `custom_llm_provider` kwargs right before calling LiteLLM. - **Option Application:** Ensures that model options loaded from the static definitions are merged into the LiteLLM request arguments. - **Cost Management:** Configures custom providers to skip cost calcu…39e01ca
-
refactor(core): review fixes + improve error handling and execution logic
MirrowelReorganized request handling in proxy_app/main.py for better logging and error management. Updated launcher.bat to streamline execution modes and remove redundant code. Enhanced condition checks in gemini_cli_provider.py for more reliable processing. Improved cost calculation in usage_manager.py for embeddings.19af906
-
fix(core): stabilize streaming for custom providers and refine usage recording
MirrowelRefactor the streaming logic in the RotatingClient to add retry mechanisms for custom providers, preventing failures on recoverable errors like rate limits and server issues. Update usage recording to capture cumulative usage from stream chunks instead of the final stream object, avoiding potential double-counting and ensuring accurate logging even on early disconnections. Minor adjustments to logging in the usage manager and provider for better debugging. - Introduce retry loop with exponential backoff for custom provider streams on errors such as RateLimitError, ServiceUnavailableError, and InternalServerError. - Modify usage tracking to accumulate the latest usage chunk during streaming and record it upon completion, eliminating reliance on the stream object's final state. - Add debug logging in Gemini CLI provider for chunk conversion. - Enhance error classification and key rotation logic to handle various exception types more robustly.79c9227 -
fix(core): stabilize streaming for custom endpoints and eliminate initialization conflicts
MirrowelThis patch improves the resilience of the request rotator and streamlines provider startup by removing manual, conflict-prone model cost registrations. - Enforce graceful key rotation when custom providers return HTTP 429 (Rate Limit) errors during stream setup, preventing request failure and silently rotating the credential. - Remove redundant `litellm.register_model` calls from Qwen, NVIDIA, and Gemini CLI providers to avoid internal litellm conflicts and duplicate model definitions. - Update the provider system to allow specific endpoints to flag themselves as excluded from the Litellm-based cost calculation pipeline. - Centralize Gemini reasoning parameter parsing, ensuring `reasoning_effort` and `custom_reasoning_budget` are correctly translated and injected, strictly targeting the `gemini-2.5` model lineage. - Inject necessary client metadata headers (User-Agent, X-Goog-Api-Client) into Gemini CLI requests for API compliance. - Configure logging environment early in the applicat…63418af -
refactor: improve model registration and usage manager robustness
MirrowelRegister hardcoded and discovered models with LiteLLM across providers (Gemini, Qwen, Nvidia). This ensures the rotator library sets model metadata, even if costs are zero, which prevents potential runtime errors during cost calculation. Further improvements to robustness and logging: - Explicitly pass the model ID to `litellm.completion_cost()` to ensure accurate cost lookup, mitigating potential calculation fallbacks. - Increase credential obfuscation in `UsageManager` logs from the last 4 characters to the last 6 characters for better security. - Implement a type-lenient check for the `thought` flag in the Gemini stream parser to handle potential type inconsistencies.0a3b5b1
-
fix(usage): initialize daily reset for new usage keys
MirrowelEnsures that when a new usage key is created, its 'last_daily_reset' timestamp is correctly initialized to the current date. This prevents an immediate, unintended daily usage reset for newly created keys.c036ceb -
refactor(rotator): randomize key selection and streamline daily stats reset
MirrowelRandomize API key selection lists in `RotatingClient` to ensure a fair distribution of keys, especially when multiple keys have similar usage statistics. This prevents deterministic selection biases and promotes better load balancing. Consolidate the daily usage statistics reset logic into a dedicated `_reset_daily_stats_if_needed` method within the `UsageManager`. This centralizes and clarifies the reset mechanism, removing inline and potentially inconsistent checks.aaab6f8
-
feat(core): implement deadline-driven request execution and resilient error handling
MirrowelIntroduce a `global_timeout` parameter to `RotatingClient` and refactor the request lifecycle to operate within a strict time budget. - All key acquisition, rotation, and retry mechanisms now adhere to this global deadline, preventing indefinite hangs. - Redesign error propagation to handle transient provider failures (e.g., rate limits, 5xx errors) internally. These errors now trigger key rotation or deadline-aware retries without immediately raising exceptions to the caller. - Provide a more stable client experience by shielding consumers from intermittent backend issues, returning a failure only when the global timeout is exceeded or all keys are exhausted. - Update documentation to reflect the new `global_timeout` and improved error handling. BREAKING CHANGE: The client's error propagation for transient failures has changed. `acompletion` and `aembedding` methods in `RotatingClient` no longer raise exceptions for intermittent issues like rate limits or server errors; instead, the…62ed41b -
Fix(logging): enhance logging configuration and control
MirrowelTLDR loggin is unfucked, mostly83934f3
-
perf(rotator): prevent indefinite waits for API key acquisition
MirrowelIntroduces a timeout for API key acquisition and enhances streaming error handling. - Implements a `wait_timeout` in `UsageManager.acquire_key` to prevent indefinite waiting when all keys are busy or on cooldown. - Introduces `NoAvailableKeysError` to explicitly signal when no key can be acquired within the timeout. - Modifies `RotatingClient.acompletion` to gracefully handle `NoAvailableKeysError` and other general exceptions by yielding structured error chunks for streaming clients instead of raising exceptions directly. - Adjusts key release logic in `RotatingClient` to ensure proper resource management. - Refines logging for key acquisition wait re-evaluations.f8d59cd
-
fix: Type import oopsie
Mirrowel065b08b -
feat: Add support for embeddings endpoint
MirrowelThis commit introduces a new OpenAI-compatible `/v1/embeddings` endpoint to the proxy, enabling the creation of text embeddings. Key changes include: - A new `aembedding` method in `RotatingClient` to handle API calls, key rotation, and rate limit errors for embedding requests. - The `UsageManager` is updated to correctly track token usage from embedding responses. - Request logging is enhanced to categorize logs into `completions` and `embeddings` subdirectories for better organization. - A sanitizer for embedding requests is added to filter unsupported parameters before calling the OpenAI API.64e385a -
feat: Handle API errors within streams and record usage
MirrowelThis commit introduces robust error handling for streaming API responses, allowing the client to detect, record usage for, and retry on failures that occur mid-stream. Previously, if a provider sent a JSON error object after a stream had started (e.g., for a content filter violation), the client would not recognize it as a retryable failure. This resulted in failed requests not being retried and token usage for the attempt not being recorded. Changes: - A new `StreamedAPIError` exception is introduced to signal an API error discovered within a stream. - The `_safe_streaming_wrapper` in `client.py` now inspects reassembled JSON chunks. If an error object is found, it raises `StreamedAPIError`, propagating the failure to the main retry logic. - `UsageManager.record_usage` is updated to accept an optional `error_data` payload. This allows for accurate token counting using data from the provider's error response, even on failed requests. This ensures that the rotator can correctly retry…dbb8b44
-
refactor: Remove file locking mechanism from usage data saving
Mirrowel1faf5c2 -
feat: Enhance daily reset logic to ensure timezone-aware date handling
Mirrowel33e95e6
-
fix: Handle missing cooldown values in UsageManager to prevent key lockout type errors
Mirrowel5fb237f -
feat: Update key release logic in RotatingClient and UsageManager to support model-specific tracking
Mirrowelbab8bf0
-
feat: Enhance error classification and handling in UsageManager and error_handler for improved retry logic and cooldown management
Mirrowel27b342a -
feat: Enhance asynchronous handling in RotatingClient and UsageManager for improved error management and usage tracking
Mirrowel4bbfff4
-
feat: Improve key acquisition logic in RotatingClient and UsageManager for better async handling and error management
Mirrowelaa8035e -
feat: Enhance error handling in UsageManager to extract retry delay from multiple error message formats
Mirrowelbf79e30 -
feat: Refactor logging to use a dedicated logger across providers and enhance model retrieval with httpx
Mirroweld51ae99
-
feat: Implement per-model API key cooldowns
MirrowelPreviously, when an API key encountered a rate limit or authentication error for any model, it was put on a global cooldown, preventing its use across all models. This change introduces per-model cooldowns. If a key fails for a specific model, it is now only put on cooldown for that particular model. This allows the key to continue being used with other models, significantly improving key utilization and overall system resilience. Updated: - README.md and src/rotator_library/README.md to reflect the new per-model cooldown behavior. - src/rotator_library/usage_manager.py to manage cooldowns on a per-model basis.c2eea0c