Live logs showed two failures:
- OpenRouter 404 "unavailable for free" — 9 of the pool's :free slugs were
retired (llama-3.1-8b, mistral-7b, gemma-3, llama-4, deepseek-v3, etc). Each
404 cooled the whole pool until exhaustion. Rebuilt both pools from the current
GET /api/v1/models free list (llama-3.2-3b, gpt-oss-20b/120b, gemma-4-31b,
qwen3-next, hermes-3-405b, qwen3-coder, nemotron-nano) — verified 200/429, no 404.
- GitHub "Bad credentials" — GITHUB_TOKEN unset on the host. Added _prune_unkeyed
so providers without a key are dropped from rotation instead of burning a 60s
cooldown every run; OpenRouter remains the working path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fix: working GitHub Models fallback + file limits that fit free-tier context
- Replace invalid GitHub Models IDs (Phi/Llama/Mistral/Gemma 400-fail on this
account) with the only enabled ones: gpt-4o-mini + gpt-4o. List them first so
rotation no longer burns attempts cooling dead models, and there is a real
fallback once OpenRouter's shared daily quota is spent.
- Add combined 15MB upload cap and truncate oversized text to ~160K chars before
the analyst reads it, so large files no longer overflow the model context and
trip the rotation loop's "dead model" path.
- CORS: default-allow the live Vercel frontend + preview deploys.
- Docs: align api.md base URL (Render), provider names, and limits.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fix: batch cooldown all OpenRouter models on rate limit, suppress verbose errors
- Reduce Groq smart model max_tokens 4096→1024 so a single LLM call
(input + max output) stays under the 12000 TPM/min org limit when
processing large file contents through a long prompt chain.
- Add "[RETRY]" to _NOISE filter — litellm's router prints this prefix
when it internally retries before raising the exception to our handler.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
fix: drop RAG search tools, fix rate limit detection
- Remove CSVSearchTool/JSONSearchTool/PDFSearchTool/XMLSearchTool/TXTSearchTool:
these require OpenAI embeddings which fail with an OpenRouter key (401).
FileReadTool reads any file directly without needing an embedding API.
- Fix is_rate_limit: Groq returns RateLimitError/rate_limit_exceeded in the
error string, not "429". Rotation never triggered for Groq TPM errors.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
chore: bust Docker layer cache to force full redeploy
All recent bots.py fixes (monkey-patch, cooldown system) were not
reaching the container because Docker was reusing a stale COPY layer.
Changing requirements.txt forces Docker to invalidate the pip install
and COPY layers, ensuring the latest code is deployed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
fix: force caching=False in litellm call to stop cache_breakpoint re-injection
The previous patch stripped cache_breakpoint from messages before calling
litellm, but litellm's internal client wrapper was re-adding it because
CrewAI passes caching=True in the completion kwargs. Setting caching=False
in kwargs prevents the wrapper from injecting cache_breakpoint at all.
Stripping from messages kept as a second layer of defense.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
fix: monkey-patch litellm.completion to strip cache_breakpoint from messages
litellm.cache=None only disables response caching. The cache_breakpoint
field is injected into message dicts by CrewAI's prompt-caching feature
before calling litellm.completion — a different code path. Wrap
litellm.completion to pop cache_breakpoint from every message at call
time, regardless of which component added it.
Also filter litellm's own "All providers rate-limited / Auto-retrying in"
noise lines from the SSE stream.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
fix: always scan from index 0 so Groq is re-prioritized after cooldown
_pick_model previously started from self._fast_idx, which drifted after
rotations. Once Groq's cooldown expired, OpenRouter was still picked
first because the search started mid-pool. Now _pick_model always scans
from index 0, guaranteeing Groq is chosen the moment it's available.
Also removed the post-failure index advance — cooldowns already prevent
re-picking a failed model, the advance just permanently shifted the
search window away from Groq.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
feat: module-level cooldown system for smart provider cycling
Replace per-request index rotation with a persistent module-level
cooldown dict so rate-limit state carries across HTTP requests:
- _set_cooldown(): marks a model unavailable until its quota refills.
Groq models all share one org-wide TPM quota so one 429 cools them all.
- _pick_model(): always returns the highest-priority (lowest index) model
not in cooldown, so Groq is used again the moment its limit refills.
- _wait_until_available(): sleeps only when every model in both pools is
still cooling; otherwise proceeds immediately to the next provider.
- Cooldown durations: rate-limit → parsed retry-after, 404/bad-request
→ 10 min, server error → 60s.
This eliminates the "all servers limited" waste at request start because
providers that are still cooling are skipped instantly rather than being
tried, failing, and then rotated.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
fix: drop unsupported params, rotate on BadRequestError
- litellm.drop_params=True: silently strips provider-unsupported params
(seed, logprobs, parallel_tool_calls, response_format on models that
reject it) instead of crashing with 400
- Rotation logic now catches BadRequestError/invalid_request_error so
provider-specific feature rejections rotate to the next model instead
of crashing the pipeline
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
fix: disable litellm cache to stop Groq cache_breakpoint error
Groq rejects messages with 'cache_breakpoint' property (unsupported).
Newer litellm injects this automatically when caching is enabled.
Setting litellm.cache = None prevents the injection.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Dockerfile: add CREWAI_TOOLS_ALLOW_UNSAFE_PATHS=true so tools can
read /tmp/ files (CrewAI added a security gate blocking non-/app/ paths)
- bots.py: lower fast-model max_tokens 2048→1024 to stay under Groq's
6000 TPM free-tier limit per minute
- main.py: add [TRACE], pip/upgrade banners, and update notices to
_NOISE filter so they don't reach the frontend stream
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
fix: install litellm explicitly before crewai to prevent resolver override
pip's dependency resolver was allowing crewai to install without litellm
even when crewai[litellm] was listed. Installing litellm in a separate
prior RUN step guarantees it is present when crewai initializes.
fix: add litellm to requirements for groq/ and non-OpenAI provider support
- groq/llama-3.1-70b-versatile (deprecated) → groq/llama3-70b-8192
- groq/llama-3.2-3b-preview (unstable) → groq/llama3-8b-8192
- _parse_retry_after now handles Groq's 'Please try again in Xs' format
so rate-limit sleeps are accurate (2-5s) instead of always defaulting to 35s
- 401 added to transient codes so a misconfigured key rotates to next
provider instead of hard-crashing the pipeline
- comment updated: Gemini no longer in pool
remove Gemini from pools; Groq + OpenRouter is sufficient
Groq free tier (14,400 req/day) covers all portfolio traffic.
Gemini removed since no key available. OpenRouter free stays as
deep fallback across 9 diverse models.
add Groq + Gemini as primary free providers ahead of OpenRouter fallback
Groq (14,400 req/day) and Gemini AI Studio (1,500 req/day) are genuinely
free with no credit card. Both are placed first in the pools so OpenRouter
shared rate limits are only hit as a last resort. _api_key_for() routes
the correct key per provider prefix.
1. LineCapture now strips box-drawing chars instead of filtering whole
lines. This preserves agent name content so the frontend pipeline
diagram can detect active agents. Previous |filter was silently
dropping all agent identification lines.
2. handleAnalyze guard was blocking auto-retry: status === 'running'
prevented isAutoRetry calls from ever executing. Fixed by exempting
auto-retry from the running-guard.
3. AbortController now created before the 10s retry delay so Stop
works correctly during the countdown.
expand model pools: 4→8 fast, 3→7 smart with diverse providers
Adds Google gemma-3-12b, Qwen qwen3-8b, Meta llama-4-scout, Microsoft phi-3-mini
to fast pool. Adds DeepSeek v3, Meta llama-4-maverick, Mistral small-3.1-24b,
Google gemma-3-12b to smart pool. More provider diversity means rate limits on
one provider are less likely to exhaust all fallbacks.
fix: promote gemma-3-27b-it to first SMART model; suppress CrewAI box-drawing noise
- qwen3-coder (Venice) rate-limits on nearly every first attempt, burning a full
RETRY cycle before rotation. gemma-3-27b-it (Google) is consistently more
reliable and should be tried first.
- Noise filter now suppresses box-drawing chars, tracing preference messages,
event ordering warnings, and [Finalize] lines from CrewAI verbose output.
fix: set max_retries=0 on LLMs to prevent litellm double-retry on 429
CrewAI agent restarts 3x per exception; litellm max_retries=1 added a
duplicate error per restart = 6 wasted 429 calls before our handler fires.
With max_retries=0, each 429 surfaces in one call and our rotation logic
triggers immediately.
fix: replace 404 model IDs with verified OpenRouter free-tier models
- main.py: remove redundant create_agents/create_tasks calls (create_crew
already calls them internally on every rotation attempt)
- bots.py: rotate log now shows actual error snippet not hardcoded "429"
- bots.py: remove unreachable RuntimeError after loop (last attempt always
raises inside the loop, never falls through)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
deepseek-v4-flash via Crucible returned 402 insufficient_quota — rotation
was not triggering because only 429 was caught. Now catches all recoverable
provider errors: 429 (rate limit), 402 (quota), 503 (unavailable), 529 (overloaded).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add model rotation — auto-fallback on 429 across 4 fast / 3 smart models
Each pool uses different upstream providers (DeepSeek, MiniMax, Z.ai, Qwen,
Nous) so a rate-limit on one provider doesn't kill the whole pipeline.
On any 429 the crew rebuilds with the next model pair and retries immediately.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove fastembed and pandas from requirements.txt (unused, bloat image)
- Strip apt-get libgomp1 install from Dockerfile (was only for fastembed)
- Add PYTHONDONTWRITEBYTECODE + PYTHONUNBUFFERED to Dockerfile
- Escape user context braces in bots.py to prevent CrewAI KeyError on {tokens}
- Frontend: parse server error body for real message on 400 responses
- .dockerignore: add .git/ and *.egg-info/
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
