Spaces:
Paused
Paused
Commit History
-
feat(antigravity): ✨ add TransientQuotaError for bare 429 responses
MirrowelIntroduces a new TransientQuotaError exception to handle 429 responses without retry timing information. The antigravity provider now detects bare 429 errors and retries them internally like empty responses. After exhausting retries, it raises TransientQuotaError to trigger credential rotation. 429 responses with retry information continue to be treated as quota exhaustion for cooldown periods. This distinction improves handling of transient rate limits versus true quota exhaustion.543f871 -
fix(retry): 🐛 parse millisecond durations and avoid immediate retry loops
Mirrowel- Add explicit support for millisecond format (e.g., '290.979975ms') in duration parsing - Fix misinterpretation where 'ms' suffix was incorrectly parsed as minutes using negative lookahead - Ensure sub-second duration values round up to at least 1 second to prevent immediate retry loops - Properly handle zero-second delays and distinguish them from missing valuesbb812a7
-
feat(antigravity): ✨ add automatic retry logic for empty API responses
MirrowelImplements robust handling for cases where the Antigravity provider returns empty responses (no content, no tool calls) due to transient server-side issues. - Introduces new `EmptyResponseError` exception class to represent empty response failures - Classifies empty responses as rotatable server errors (HTTP 503 equivalent) to trigger credential rotation - Adds configurable retry mechanism with `ANTIGRAVITY_EMPTY_RESPONSE_RETRIES` (default: 3) and `ANTIGRAVITY_EMPTY_RESPONSE_RETRY_DELAY` (default: 2s) - Implements separate retry logic for streaming and non-streaming requests to preserve true streaming behavior - Refactors request handling to use a URL fallback loop with nested empty response retry logic - Tracks whether any chunks were yielded in streaming mode to avoid emitting synthetic final chunks for empty responses - Moves helper functions (`_env_bool`, `_env_int`) earlier in the file for use in configuration constants - Stores last received usage metadata for proper final chunk…c5dfecd
-
fix(oauth): 🐛 prevent deadlock and token desync for rotating refresh tokens
MirrowelThis commit fixes critical issues in OAuth providers (Google, iFlow, Qwen) that use rotating refresh tokens, where each token refresh invalidates the previous token. **Problems Fixed:** 1. **Deadlock Prevention**: Removed inline re-authentication from `refresh_token()` method that was called while holding a lock. When refresh failed with HTTP 400/401/403, the method would call `initialize_token()` directly, which would try to acquire the same lock, causing a deadlock. Now, invalid token errors are caught and queued for background re-authentication via `asyncio.create_task()`. 2. **Token Desync**: Changed `_save_credentials()` to write to disk FIRST, then update cache. Previously, cache was updated first with `buffer_on_failure=True`, which could leave stale tokens on disk if the write failed. For rotating tokens, this caused the old refresh_token on disk to become invalid after a successful API call, requiring re-auth on restart. 3. **Stale Cache Usage**: Modified `refresh_token()`…8ed4f52
-
feat(usage): ✨ implement per-model quota tracking with authoritative reset timestamps
MirrowelThis commit introduces granular per-model quota tracking that supports provider-specific reset timestamps from quota exhausted errors. Key changes: - Add `quota_reset_timestamp` field to `ClassifiedError` to capture authoritative Unix timestamp from provider's quota exhausted responses - Implement per-model usage tracking mode where each model maintains its own window with `window_start_ts` and `quota_reset_ts` - Add quota group support for models that share quota limits (e.g., Claude Sonnet and Opus on Antigravity) - Parse Antigravity's `quotaResetTimeStamp` ISO format to Unix timestamp for precise reset timing - Update reset logic to prioritize authoritative `quota_reset_ts` over fallback window calculations - Distinguish between quota exhausted (sets authoritative reset time) and rate limit (transient cooldown only) - Migrate Antigravity provider to per-model tracking with 5-hour windows for paid tier and 7-day windows for free tier The per-model mode enables more accurate quota …0ca1651 -
feat(rotation): ✨ add sequential rotation mode with provider-specific quota parsing
MirrowelIntroduces a new credential rotation mode system that allows providers to choose between "balanced" (distribute load evenly) and "sequential" (use until exhausted) strategies. Sequential mode is particularly beneficial for providers with cache-preserving features like Antigravity's thinking signature caches. Key changes: - Added ROTATION_MODE_{PROVIDER} environment variable support with comprehensive documentation in .env.example - Implemented provider-specific quota error parsing for Antigravity and Gemini CLI providers, extracting retry_after from Google RPC error format (handles compound durations like "143h4m52.73s") - Extended ProviderInterface with rotation mode configuration and parse_quota_error() method - Updated UsageManager to support sequential credential selection that preserves sticky credential usage until quota exhaustion - Enhanced error_handler.py classify_error() to attempt provider-specific parsing before falling back to generic classification - Added rotation mode…bd84d38
-
fix(error-handler): 🔨 extract JSON retry parsing into dedicated function
MirrowelExtracted retry delay parsing logic from `get_retry_after()` into a new `_extract_retry_from_json_body()` helper function to improve code organization and maintainability. - New `_extract_retry_from_json_body()` function handles parsing of Antigravity/Google API JSON error responses - Preserves case-sensitive key handling for API responses (RetryInfo, ErrorInfo) - Prioritizes response body JSON parsing over HTTP headers for httpx errors - Maintains backward compatibility with all existing retry delay extraction patterns - Improves code readability by separating JSON parsing concerns from the main retry extraction logicabdc406 -
fix(error-handler): 🐛 handle compound duration formats in retry-after parsing
MirrowelAdded support for parsing complex duration strings commonly returned by Antigravity and Google APIs, such as "156h14m36.752463453s" or "562476.752463453s". - Introduced `_parse_duration_string()` helper to parse compound duration formats (hours, minutes, seconds with decimals) - Updated `extract_retry_after_from_body()` to handle both simple and compound duration strings - Enhanced `get_retry_after()` to iterate through all error details (not just first item) and check both RetryInfo and ErrorInfo metadata - Added `httpx.HTTPStatusError` to exception handling in client retry logic - Fixed formatting inconsistencies in conditional statements for rate limit handlingfa51b1a
-
Merge branch 'Antigravity' of https://github.com/Mirrowel/LLM-API-Key-Proxy into fix/quota-exceeded-global-cooldown
5a29725 -
fix(rotator): prevent quota errors from global cooldown
Resolves a critical issue where a single credential hitting quota limits triggered a provider-wide cooldown, causing denial of service for all remaining healthy credentials. **Problem:** When any credential encountered a 429 "Quota Exceeded" error, the system applied a global 60-second cooldown to the entire provider. This blocked ALL credentials from being used, even though quota errors are credential-specific, not provider-wide. **Root Cause:** The error classification system did not distinguish between: - Rate limit errors (IP/provider throttling) → affects all credentials - Quota errors (account/key limits) → affects only that credential Both were treated as `rate_limit` and triggered `cooldown_manager`, which pauses the entire provider. **Solution:** 1. **Enhanced Error Classification** (error_handler.py): - Parse RateLimitError messages for "quota"/"resource_exhausted" - Classify as `quota_exceeded` (not `rate_limit`) - Preserves retry_after headers for both types …0dd6d21 -
fix(logging): preserve full credential filenames in logs
Resolved logging truncation issue where OAuth credential filenames were being aggressively abbreviated (e.g., `...6.json` instead of `antigravity_oauth_16.json`), causing ambiguity when debugging or auditing specific credentials. **Changes:** - Enhanced `mask_credential()` utility in error_handler.py: - Now explicitly detects `.json` file extensions - Returns full basename for file paths (e.g., `antigravity_oauth_16.json`) - Maintains security by masking API keys to last 6 characters (`...xyz123`) - Replaced all manual credential truncation with centralized `mask_credential()`: - client.py: 15 instances (stream handling, retry logging, model discovery) - usage_manager.py: 16 instances (key acquisition, release, cooldown tracking) - failure_logger.py: 2 instances (failure logging and summaries) - Code quality improvements: - Fixed indentation error in client.py during refactoring - Ensured consistent, safe credential logging across entire application - Configuration…fce1762 -
fix: improve error handling implementation based on code review
mirrobot-agent[bot]- Fix credential counting to track unique credentials (RequestErrorAccumulator) - Move import os to module level in mask_credential function - Fix status code check to use explicit 'is not None' comparison - Improve context window error detection with more specific patterns - Correct comment about server error classification - Remove redundant '1 *' in exponential backoff calculations - Add warning log for unreachable None return path - Remove redundant error_accumulator.model/provider assignments - Remove access to private _content attribute in failure_logger - Add circular reference detection in error chain loop - Reorder error recording to occur after should_rotate_on_error check These changes address issues identified in both mirrobot-agent and GitHub Copilot code reviews.a1cc875
-
feat(core): ✨ add structured error accumulator and consistent error handling/reporting
MirrowelIntroduce RequestErrorAccumulator and related helpers to aggregate and classify errors across credential rotation and provide structured client-facing error responses. - Add RequestErrorAccumulator to record per-credential errors (abnormal vs normal), build concise server log messages and structured client error payloads. - Add mask_credential, is_abnormal_error, should_rotate_on_error and should_retry_same_key helpers and extend classify_error to better handle httpx.HTTPStatusError and more error types (forbidden, quota_exceeded, context_window_exceeded, etc.). - Update RotatingClient (sync and streaming paths) to: - initialize and record errors into the accumulator during retries/rotation, - mask credentials in logs, - handle httpx.HTTPStatusError explicitly, - standardize cooldowns and retry-vs-rotate decisions, - return a structured error response (dict) for non-streaming failures and yield structured JSON error payloads for streaming failures. - Improve failure_logger: …7cb148b
-
fix(gemini): 🐛 consolidate parallel tool responses and improve rate limit handling
MirrowelThis commit addresses multiple issues with Gemini API providers related to parallel function calling and rate limit error handling: **Tool Response Consolidation:** - Parallel function responses are now consolidated into a single user message as required by Gemini API specification - Previously, consecutive tool responses were sent as separate messages, causing API errors - Implemented pending tool parts accumulation pattern in both GeminiCliProvider and AntigravityProvider - Tool responses are flushed when a non-tool message is encountered or at the end of message processing **Thought Signature Handling:** - Fixed parallel function call signature behavior to match Gemini 3 API requirements - Only the first parallel function call in a message receives a thoughtSignature field - Subsequent parallel calls no longer include thoughtSignature to prevent API validation errors - Removed `first_sig_seen` tracking flags since signatures are now stored per tool call **Rate Limit Error Handlin…d4593e5
-
fix(core): improve error classification and prevent false key penalization
MirrowelThis commit refines the failure handling mechanism to ensure keys are only penalized for errors directly attributable to them (e.g., authentication, sustained client errors). Key changes: - Enhanced retry logic to reliably extract cooldown durations from HTTP headers (Retry-After, X-RateLimit-Reset) and duration strings (e.g., "60s", "2m"). - Introduced logic to differentiate between key-specific failures and transient provider-level errors (server_error, api_connection). - Provider-level errors now trigger a short cooldown but skip consecutive failure incrementation, preserving the key's standing. - Rate limit errors now default to a 60s cooldown if no explicit retry duration is detected.20944a5
-
feat(core): enable static model configuration and LiteLLM compatibility for custom providers
Mirrowelintroduce static model definitions and runtime parameter conversion for custom endpoints. This change significantly improves compatibility with self-hosted or dynamically configured OpenAI-compatible APIs: - **Model Definitions:** Adds a new `ModelDefinitions` utility to load static model configurations (IDs, options like `reasoning_effort`) from environment variables (e.g., `PROVIDER_MODELS`). - **Dynamic Providers:** Extends provider discovery to dynamically register new providers whenever an `_API_BASE` environment variable is detected. - **Client Conversion:** Implements `_convert_model_params_for_litellm` in the RotatingClient to rewrite the model argument (to `openai/{model_id}`) and inject the necessary `api_base` and `custom_llm_provider` kwargs right before calling LiteLLM. - **Option Application:** Ensures that model options loaded from the static definitions are merged into the LiteLLM request arguments. - **Cost Management:** Configures custom providers to skip cost calcu…39e01ca -
feat(providers): ✨ enable dynamic support for custom openai-compatible providers
MirrowelThis change introduces a mechanism to automatically detect and register custom OpenAI-compatible providers based on environment variables (e.g., `CUSTOM_PROVIDER_API_BASE`). - Adds `OpenAICompatibleProvider` class for generic endpoint interactions. - Updates `AllProviders` (in `error_handler`) to dynamically discover and register these providers by scanning the environment variables. - Modifies `RotatingClient` to lazily instantiate the compatible provider when a provider name is used and its API base is configured. This allows users to integrate with any standards-compliant API without explicit internal registration.225c46e
-
feat(provider): introduce OAuth credential management and custom provider handling
Mirrowel- Implemented a new `CredentialManager` to discover and manage OAuth credential files from standard paths (`~/.gemini`, `~/.qwen`). - Added a `BackgroundRefresher` to proactively refresh OAuth tokens before they expire, ensuring continuous service. - Refactored `RotatingClient` to support both API keys and OAuth credentials for provider authentication. - Integrated `litellm_provider_params` in `RotatingClient` to allow specific LiteLLM configurations per provider (e.g., Google Cloud project ID for Gemini CLI). - Introduced a `has_custom_logic` flag and `acompletion` method in `ProviderInterface` to enable custom handling for providers like Gemini CLI and Qwen Code, which require specific request formats, authentication, or stream parsing not fully supported by LiteLLM's standard interface. - Updated `proxy_app/main.py` to utilize the new OAuth credential loading, provider-specific LiteLLM parameters, and the background token refresher. - Enhanced `error_handler.py` to classify `httpx` …d28e7c5
-
feat(client): add pre-request callback support to API client
MirrowelIntroduce optional asynchronous callbacks that execute before each API request attempt, allowing for custom preprocessing such as validation or logging. - Includes a new `PreRequestCallbackError` exception for handling callback failures. - Adds `abort_on_callback_error` parameter to control whether exceptions in callbacks stop the request or are logged as warnings. - Extends support to both completion and embedding methods, with detailed docstrings for usage. This feature enables users to inject custom logic into the request pipeline without modifying core client behavior.da8d5f4
-
perf(rotator): prevent indefinite waits for API key acquisition
MirrowelIntroduces a timeout for API key acquisition and enhances streaming error handling. - Implements a `wait_timeout` in `UsageManager.acquire_key` to prevent indefinite waiting when all keys are busy or on cooldown. - Introduces `NoAvailableKeysError` to explicitly signal when no key can be acquired within the timeout. - Modifies `RotatingClient.acompletion` to gracefully handle `NoAvailableKeysError` and other general exceptions by yielding structured error chunks for streaming clients instead of raising exceptions directly. - Adjusts key release logic in `RotatingClient` to ensure proper resource management. - Refines logging for key acquisition wait re-evaluations.f8d59cd
-
refactor: Enhance error handling and retry logic
MirrowelThis commit improves the robustness of the client by implementing more granular error handling and retry strategies. The changes introduce three distinct behaviors based on the classified error type: 1. **Fail Fast:** For unrecoverable errors like `InvalidRequestError` and the newly classified `ContextWindowExceededError`, the request now fails immediately. This prevents wasting time and keys on a request that cannot be fixed by rotation. 2. **Retry with Backoff:** For temporary issues like `server_error` and `api_connection`, the client will retry with the same key using exponential backoff. `APIConnectionError` and `Timeout` are now grouped and use a longer initial backoff period to better handle network instability. 3. **Rotate Key:** For key-specific issues like `rate_limit` or `authentication`, the client records the failure and immediately tries the next available key.a20e9d0
-
feat: enhance changelog generation and error handling in build process
Mirrowel5db0197
-
feat: Enhance error classification and handling in UsageManager and error_handler for improved retry logic and cooldown management
Mirrowel27b342a -
feat: Enhance asynchronous handling in RotatingClient and UsageManager for improved error management and usage tracking
Mirrowel4bbfff4
-
Inital commit
Mirrowelbf565ba