feat(optimizations): reduce frames, add proactive spacing, caching, and project guide

QWEN.md

@@ -0,0 +1,233 @@
+# Insta-to-WP (Content Agent) — Project Guide
+
+## Overview
+
+A Python-based content automation platform that:
+1. **Social Media → WordPress**: Downloads videos from Instagram/TikTok/YouTube/X, analyzes frames with AI vision models, generates blog posts, and publishes to WordPress
+2. **Article Rewriter**: Scrapes articles from URLs, rewrites them with AI, finds related media, and publishes to WordPress
+3. **Web Dashboard**: Flask-based web app with job queue management, RSS feed monitoring, live logs, and a background content publishing agent
+
+**Deployment**: Hugging Face Spaces (Docker), Render.com, or local Flask dev server.
+
+---
+
+## Architecture
+
+### Three Main Entry Points
+
+| File | Lines | Purpose |
+|---|---|---|
+| `app.py` | ~670 | Flask web dashboard + background agent. Runs on port 5000 (dev) or 7860 (Docker). Web UI at `/` with job submission, queue, live logs, settings, RSS feeds |
+| `main.py` | ~1760 | CLI pipeline for social media video → WordPress. Handles download → frame extraction → AI vision analysis → content generation → featured image creation → WP publish |
+| `article_rewriter.py` | ~1750 | Standalone article pipeline. Handles scrape → AI rewrite → image filtering → related media embedding → featured image → WP publish |
+
+### Dual-Engine AI System
+
+- **Primary**: OpenRouter API (`https://openrouter.ai/api/v1`) using `qwen/qwen3.6-plus:free`
+- **Backup**: Groq API (`https://api.groq.com/openai/v1`) using `llama-3.3-70b-versatile` (text) / `llama-3.2-90b-vision-preview` (vision)
+- **Rate Limit Recovery**: All OpenRouter calls wrapped with `_openrouter_with_retry()` — on HTTP 429, cools down 90 seconds, retries once, then falls back to Groq
+
+### AI Call Sites (5 total)
+
+| Location | Function | Model | Purpose |
+|---|---|---|---|
+| `main.py` | `_analyze_frames()` | Qwen 3.6 Plus (vision) | Analyze up to 60 video frames, return JSON with description, subjects, mood, frame categories (backside/frontside/other) |
+| `main.py` | `generate_content()` | Qwen 3.6 Plus | Generate blog title + 2 paragraphs from video analysis |
+| `article_rewriter.py` | `rewrite_article()` | Qwen 3.6 Plus | Rewrite scraped article into unique blog post with title, paragraphs, SEO description, tags, image alt texts |
+| `article_rewriter.py` | `filter_irrelevant_images()` | Qwen 3.6 Plus | AI-filter article images for relevance (keep content, drop ads/logos/thumbnails) |
+| `app.py` | `screen_article()` | Qwen 3.6 Plus | Screen RSS feed articles against criteria before queuing |
+
+---
+
+## Environment Variables
+
+See `.env.template` for all variables. Key ones:
+
+```
+OPENROUTER_API_KEY=          # Primary AI API key
+GROQ_API_KEY=                # Backup AI API key
+LLM_MODEL=qwen/qwen3.6-plus:free
+GROQ_BACKUP_MODEL=llama-3.3-70b-versatile
+LLM_VISION_BACKUP_MODEL=meta-llama/llama-4-scout-17b-16e-instruct
+GROQ_VISION_MODEL=llama-3.2-90b-vision-preview
+WP_URL=                      # WordPress site URL
+WP_USERNAME=
+WP_APP_PASSWORD=
+WP_CATEGORY_ID=              # Default category for social media posts
+WP_SLAY_CATEGORY_ID=         # Category for social media posts
+WP_REWRITE_CATEGORY_ID=      # Category for rewritten articles
+WP_POST_STATUS=publish       # or "draft"
+IG_USERNAME=                 # Optional Instagram credentials
+IG_PASSWORD=
+FLASK_SECRET_KEY=            # Auto-generated if not set
+```
+
+---
+
+## Job System
+
+### Database: SQLite (`agent.db`)
+
+| Table | Purpose |
+|---|---|
+| `jobs` | Job queue: id, url, title, mode, status (pending/running/done/failed/review), error, result_json, timestamps |
+| `posts` | Published posts tracking |
+| `processed_urls` | URL deduplication tracker |
+| `feeds` | RSS feed subscriptions with last_checked, articles_found |
+| `settings` | Key-value store for agent_enabled, screening_enabled, check_interval_minutes, etc. |
+
+### Job Modes
+
+- `"article"` → calls `article_rewriter.process_article()`
+- `"instagram"` / `"social"` → calls `main.process_video()`
+- `"upload"` / `"upload_video"` → calls `main.process_uploaded_video()`
+- `"upload_image"` → calls `main.process_uploaded_image()`
+
+### Processing Flow
+
+1. Jobs created via `POST /api/jobs` or `POST /api/upload` → status `pending`
+2. Background `agent_loop()` runs every 10s, calls `process_pending_jobs()`
+3. Jobs processed **sequentially** (one at a time) with `threading.Lock()`
+4. 90-second sleep between jobs (strict sequential processing)
+5. Status transitions: `pending` → `running` → `done` | `failed` | `review`
+
+### Background Agent
+
+- `agent_loop()` runs continuously, checks `agent_enabled` setting
+- Every `check_interval_minutes` (default 30): checks RSS feeds for new articles
+- RSS articles pass through `screen_article()` AI filter before queuing
+- `review` status jobs appear in dashboard for manual approval before publishing
+
+---
+
+## Deployment
+
+### Docker (Hugging Face Spaces)
+
+```dockerfile
+# Python 3.10-slim base
+# Installs ffmpeg + system libs
+# Runs: gunicorn app:app --bind 0.0.0.0:7860 --timeout 120 --workers 1
+```
+
+### Render.com
+
+Config in `render.yaml` — Docker env, Frankfurt region, free plan, health check at `/api/health`
+
+### Local Development
+
+```bash
+# PowerShell (Windows)
+.\start_server.ps1
+# Or directly
+.venv\Scripts\python.exe app.py
+# Visit http://localhost:5000
+```
+
+### CI/CD
+
+GitHub Actions (`.github/workflows/sync_to_hf.yml`) — on push to `main`, force-pushes to Hugging Face Spaces.
+
+---
+
+## Frontend
+
+| File | Lines | Purpose |
+|---|---|---|
+| `templates/dashboard.html` | ~626 | Main dashboard UI with PWA support |
+| `templates/login.html` | — | Login page (credentials: `webmazta` / `2010`) |
+| `static/dashboard.css` | ~837 | "Premium Dark Mode" styling |
+| `static/dashboard.js` | ~775 | Dashboard frontend (job management, live logs, settings) |
+| `static/manifest.json` | — | PWA manifest (app name: "Content Agent", theme: #6366f1) |
+| `static/sw.js` | — | Service worker for offline support |
+
+---
+
+## Key Technologies
+
+- **Video**: `yt-dlp` (download from 1000+ sites), `instaloader` (Instagram), `apify-client` (web scraping)
+- **Vision**: `opencv-python-headless` (frame extraction), `Pillow` (image processing, featured image generation)
+- **AI**: `openai` 1.90.0 (LLM client), `pydantic` (structured output validation)
+- **Web**: `flask` 3.0.0, `gunicorn` 21.2.0, `feedparser` (RSS), `requests`, `aiohttp` (async HTTP)
+- **Scraping**: `beautifulsoup4`, `lxml`
+
+---
+
+## Runtime Directories
+
+| Directory | Purpose |
+|---|---|
+| `downloads/` | Downloaded media files (.mp4, .jpg, .txt metadata) |
+| `output/` | Generated featured images (`featured_<id>.jpg`) |
+| `uploads/` | User-uploaded media via dashboard |
+
+---
+
+## Development Guidelines
+
+### Adding New LLM Features
+
+1. Always use the **dual-engine pattern**: OpenRouter primary → Groq backup
+2. **Wrap all OpenRouter calls** with `_openrouter_with_retry()` — 90s cooldown on 429
+3. Use `response_format={"type": "json_object"}` for structured output
+4. Include `extra_body={"reasoning": {"max_tokens": N}}` for OpenRouter reasoning tokens
+5. Log reasoning tokens when available:
+   ```python
+   usage = getattr(completion, 'usage', None)
+   if usage and hasattr(usage, 'extra_fields') and 'reasoning_tokens' in usage.extra_fields:
+       print(f"      [INFO] Reasoning tokens used: {usage.extra_fields['reasoning_tokens']}")
+   ```
+
+### Code Style
+
+- Functions use docstrings with clear descriptions
+- Print-based logging with emoji prefixes: `[AI]`, `[ENGINE]`, `[WARN]`, `[RATE LIMIT]`, `[CONTENT]`
+- Network helpers force IPv4 via `socket.getaddrinfo` monkeypatch
+- Configuration loaded from `.env` via `load_dotenv()` at module top
+- Use `os.getenv()` for dynamic config (background worker sync)
+
+### Database Operations
+
+- Use `get_db()` helper (returns `sqlite3.Row` dict-like rows)
+- All writes should use parameterized queries (`?` placeholders)
+- Use `LOCK` threading.Lock for job processing serialization
+
+---
+
+## Testing
+
+12 test scripts in root directory:
+- `test_dual_engine.py` — OpenRouter + Groq + Vision testing
+- `test_groq_vision.py`, `test_vision_json.py` — Vision model tests
+- `test_local_llm.py` — LM Studio connectivity
+- `test_screening.py` — Article screening
+- Various Gemini model tests (legacy experiments)
+
+---
+
+## Sub-projects (git-ignored)
+
+- `insta-to-wp-android/` — Android companion app (Kotlin/Gradle)
+- `cobby/` — Minimal directory with `.gemini/` subfolder
+- `hf_sync/` — Duplicate copy for Hugging Face sync
+
+---
+
+## Quick Commands
+
+```bash
+# Run web dashboard
+python app.py
+
+# Process single video
+python main.py <video_url>
+
+# Rewrite single article
+python article_rewriter.py <article_url>
+
+# Batch process from file
+python main.py --file urls.txt
+
+# Stop background processing
+curl -X POST http://localhost:5000/api/jobs/stop
+```

app.py

@@ -47,11 +47,13 @@ def _openrouter_with_retry(api_call_fn, max_retries=1):
     """
     Retry an OpenRouter API call with 90-second cooldown on rate limit (HTTP 429).
     Falls through to caller's exception handling after max retries exhausted.
+    Returns (response, retries_used) tuple on success.
     """
     import time
     for attempt in range(max_retries + 1):
         try:
-            return api_call_fn()
+            result = api_call_fn()
+            return result, attempt  # Return response + retries used
         except Exception as e:
             error_str = str(e)
             if "429" in error_str and attempt < max_retries:
@@ -79,6 +81,7 @@ UPLOAD_DIR.mkdir(exist_ok=True)
 
 LOCK = threading.Lock()
 STOP_PROCESSING = threading.Event()  # Set this to halt queue processing
+LAST_LLM_JOB_TIME = 0  # Timestamp of last LLM-heavy job (for proactive rate-limit spacing)
 
 # ──────────────────────────────────────────────
 # Logging Setup
@@ -296,11 +299,24 @@ def process_pending_jobs():
         print(f"[AGENT] ❌ Job #{job_id} failed: {e}")
         send_notification("failed", url, str(e)[:200], mode)
 
+    # Track completion time for proactive rate-limit spacing
+    global LAST_LLM_JOB_TIME
+    LAST_LLM_JOB_TIME = time.time()
+
     return True
 
 def _process_all_pending():
+    global LAST_LLM_JOB_TIME
     while True:
         with LOCK:
+            # Proactive rate-limit spacing: wait if last LLM job was too recent
+            MIN_JOB_GAP = 120  # 2 minutes between LLM-heavy jobs
+            elapsed = time.time() - LAST_LLM_JOB_TIME
+            if LAST_LLM_JOB_TIME > 0 and elapsed < MIN_JOB_GAP:
+                wait = MIN_JOB_GAP - elapsed
+                print(f"[AGENT] ⏳ Proactive cooldown: waiting {wait:.0f}s before next job...")
+                # Release lock while sleeping to let other threads run
+                time.sleep(wait)
             if not process_pending_jobs():
                 break
         time.sleep(90)
@@ -353,7 +369,9 @@ def screen_article(title, summary, criteria):
         def _do_screen():
             return client.chat.completions.create(model="qwen/qwen3.6-plus:free", messages=[{"role":"user","content":prompt}], response_format={"type":"json_object"})
 
-        resp = _openrouter_with_retry(_do_screen)
+        resp, retries = _openrouter_with_retry(_do_screen)
+        if retries > 0:
+            print(f"[AGENT] [RETRY] Screening succeeded after {retries} retry(ies)")
         result = json.loads(resp.choices[0].message.content)
         print(f"[AGENT] Screening: {result.get('publish')} - {result.get('reason')}")
         return result.get("publish", True)
@@ -390,6 +408,7 @@ def check_rss_feeds():
     return new_articles
 
 def agent_loop():
+    global LAST_LLM_JOB_TIME
     print("[AGENT] 🤖 Agent thread started")
     last_rss = 0
     while True:
@@ -397,7 +416,13 @@ def agent_loop():
             if get_setting("agent_enabled") == "true":
                 if time.time() - last_rss > int(get_setting("check_interval_minutes", "30")) * 60:
                     check_rss_feeds(); last_rss = time.time()
-                with LOCK: process_pending_jobs()
+                with LOCK:
+                    # Proactive rate-limit spacing in agent loop too
+                    MIN_JOB_GAP = 120
+                    elapsed = time.time() - LAST_LLM_JOB_TIME
+                    if LAST_LLM_JOB_TIME > 0 and elapsed < MIN_JOB_GAP:
+                        time.sleep(MIN_JOB_GAP - elapsed)
+                    process_pending_jobs()
             time.sleep(10)
         except Exception as e: print(f"[AGENT] Loop error: {e}"); time.sleep(30)
 
@@ -659,6 +684,11 @@ def _start_agent():
 print("[STARTUP] Initializing database...")
 init_db()
 
+# Warn if backup engine has no API key
+if not os.getenv("GROQ_API_KEY"):
+    print("[WARN] GROQ_API_KEY is not set. If OpenRouter exhausts retries, jobs will fail with no fallback.")
+    print("[WARN] Set GROQ_API_KEY in .env or environment to enable the backup AI engine.")
+
 @app.before_request
 def _ensure_agent(): _start_agent()
 

main.py

@@ -715,10 +715,12 @@ def _openrouter_with_retry(api_call_fn, max_retries=1):
     """
     Retry an OpenRouter API call with 90-second cooldown on rate limit (HTTP 429).
     Falls through to caller's exception handling after max retries exhausted.
+    Returns (response, retries_used) tuple on success.
     """
     for attempt in range(max_retries + 1):
         try:
-            return api_call_fn()
+            result = api_call_fn()
+            return result, attempt  # Return response + retries used
         except Exception as e:
             error_str = str(e)
             if "429" in error_str and attempt < max_retries:
@@ -738,11 +740,51 @@ def analyze_video(video_path: Path, frames: list[Image.Image] = None) -> dict:
     If frames are provided, use them; otherwise extract from video_path.
     """
     if frames is None:
-        frames = extract_frames(video_path, count=60)
+        frames = extract_frames(video_path, count=30)
     return _analyze_frames(frames)
 
+
+# ──────────────────────────────────────────────
+# 2c. Vision Result Cache
+# ──────────────────────────────────────────────
+
+_VISION_CACHE_PATH = Path("vision_cache.json")
+
+def _get_vision_cache_key(frames: list[Image.Image]) -> str:
+    """Create a content hash from frames using first, middle, and last frame fingerprints."""
+    hasher = hashlib.md5()
+    for idx in [0, len(frames) // 2, -1]:
+        frame = frames[idx]
+        # Hash a downsampled 64x64 grayscale version for content similarity
+        thumb = frame.resize((64, 64)).convert("L").tobytes()
+        hasher.update(thumb)
+    return hasher.hexdigest()
+
+def _load_vision_cache() -> dict:
+    if _VISION_CACHE_PATH.exists():
+        try:
+            with open(_VISION_CACHE_PATH, "r") as f:
+                return json.load(f)
+        except (json.JSONDecodeError, IOError):
+            return {}
+    return {}
+
+def _save_vision_cache(cache: dict):
+    try:
+        with open(_VISION_CACHE_PATH, "w") as f:
+            json.dump(cache, f)
+    except IOError:
+        pass
+
 def _analyze_frames(frames: list[Image.Image]) -> dict:
     """Analyze provided frames using the dual-engine setup (OpenRouter Primary, Groq Backup)."""
+    # Check cache first
+    cache_key = _get_vision_cache_key(frames)
+    cache = _load_vision_cache()
+    if cache_key in cache:
+        print(f"   [CACHE] Vision result found for this video content (skip AI call)")
+        return cache[cache_key]
+
     print(f"   [AI] Analyzing {len(frames)} frames (Vision task)...")
 
     prompt = textwrap.dedent("""
@@ -781,14 +823,14 @@ def _analyze_frames(frames: list[Image.Image]) -> dict:
     
     # --- Step 1: Try OpenRouter (Primary) ---
     print(f"      [DEBUG] OpenAI available: {bool(OpenAI)}, OPENROUTER_API_KEY available: {bool(openrouter_key)}")
-    print(f"      [DEBUG] Model: {LLM_MODEL}, Frames to analyze: {len(frames[:60])}")
+    print(f"      [DEBUG] Model: {LLM_MODEL}, Frames to analyze: {len(frames[:30])}")
     if OpenAI and openrouter_key:
         try:
             print(f"   [ENGINE] Trying OpenRouter ({LLM_MODEL})...")
             client = OpenAI(base_url="https://openrouter.ai/api/v1", api_key=openrouter_key)
 
             messages_content = [{"type": "text", "text": prompt}]
-            for frame in frames[:60]:
+            for frame in frames[:30]:
                 b64_image = _image_to_base64(frame)
                 messages_content.append({
                     "type": "image_url",
@@ -808,8 +850,11 @@ def _analyze_frames(frames: list[Image.Image]) -> dict:
                     }
                 )
 
-            completion = _openrouter_with_retry(_do_analyze)
+            completion, retries = _openrouter_with_retry(_do_analyze)
             raw_text = completion.choices[0].message.content.strip()
+
+            if retries > 0:
+                print(f"      [RETRY] OpenRouter succeeded after {retries} retry(ies)")
             
             # Log reasoning tokens if available (OpenRouter specific)
             usage = getattr(completion, 'usage', None)
@@ -817,7 +862,13 @@ def _analyze_frames(frames: list[Image.Image]) -> dict:
                 print(f"      [INFO] Reasoning tokens used: {usage.extra_fields['reasoning_tokens']}")
             
             print(f"      [DEBUG] OpenRouter returned {len(raw_text)} chars, starts with: {raw_text[:100]}")
-            return json.loads(raw_text)
+            result = json.loads(raw_text)
+            # Save to cache for future reuse
+            cache = _load_vision_cache()
+            cache[cache_key] = result
+            _save_vision_cache(cache)
+            print(f"   [CACHE] Saved vision result to cache")
+            return result
         except Exception as e:
             error_type = type(e).__name__
             error_str = str(e)
@@ -841,7 +892,7 @@ def _analyze_frames(frames: list[Image.Image]) -> dict:
             client = OpenAI(base_url="https://api.groq.com/openai/v1", api_key=groq_key)
             
             messages_content = [{"type": "text", "text": prompt}]
-            for frame in frames[:60]:
+            for frame in frames[:30]:
                 b64_image = _image_to_base64(frame)
                 messages_content.append({
                     "type": "image_url",
@@ -856,7 +907,13 @@ def _analyze_frames(frames: list[Image.Image]) -> dict:
             )
             raw_text = completion.choices[0].message.content.strip()
             print(f"      [DEBUG] Groq returned {len(raw_text)} chars, starts with: {raw_text[:100]}")
-            return json.loads(raw_text)
+            result = json.loads(raw_text)
+            # Save to cache for future reuse
+            cache = _load_vision_cache()
+            cache[cache_key] = result
+            _save_vision_cache(cache)
+            print(f"   [CACHE] Saved vision result to cache")
+            return result
         except Exception as e:
             error_type = type(e).__name__
             error_str = str(e)
@@ -935,8 +992,11 @@ def generate_content(analysis: any, source_url: str) -> dict:
                     }
                 )
 
-            completion = _openrouter_with_retry(_do_generate)
+            completion, retries = _openrouter_with_retry(_do_generate)
             raw_text = completion.choices[0].message.content.strip()
+
+            if retries > 0:
+                print(f"      [RETRY] OpenRouter succeeded after {retries} retry(ies)")
             
             # Log reasoning tokens (OpenRouter specific)
             usage = getattr(completion, 'usage', None)
@@ -1443,7 +1503,7 @@ def process_video(url_input: any) -> dict:
         is_video = ext in [".mp4", ".webm", ".avi", ".mov"]
         
         # Extract frames
-        frames = extract_frames(video_path, count=60 if i == 0 else 5)
+        frames = extract_frames(video_path, count=30 if i == 0 else 5)
         all_frames.extend(frames)
 
         # Upload original to WP
@@ -1459,8 +1519,8 @@ def process_video(url_input: any) -> dict:
     # If grouped, all_frames might have frames from multiple videos.
     # We analyze the primary video's frames specifically or use the global pool.
     # Use the first downloaded video's path as the reference path
-    primary_video_path = DOWNLOAD_DIR / f"{safe_name}.mp4" 
-    analysis = analyze_video(primary_video_path, frames=all_frames[:60])
+    primary_video_path = DOWNLOAD_DIR / f"{safe_name}.mp4"
+    analysis = analyze_video(primary_video_path, frames=all_frames[:30])
     categories = analysis.get("frame_categories", {}) if isinstance(analysis, dict) else {}
 
     # Step: Generate Content
@@ -1469,7 +1529,7 @@ def process_video(url_input: any) -> dict:
 
     # Step: Create Collages
     # UNIFIED: Use the same frames that were sent to the AI
-    processing_frames = all_frames[:60]
+    processing_frames = all_frames[:30]
     featured_path = create_featured_image(title, processing_frames, safe_name, categories)
     flyer_path = create_flyer(title, processing_frames, safe_name, categories)
 
@@ -1541,7 +1601,7 @@ def process_uploaded_video(video_path_input: any) -> dict:
         wp_extra_media.append({"id": wp_item["id"], "url": wp_item["url"], "type": "video" if is_video else "image"})
         
         if is_video:
-            frames = extract_frames(p, count=60)
+            frames = extract_frames(p, count=30)
             all_frames.extend(frames)
         else:
             try:
@@ -1550,13 +1610,13 @@ def process_uploaded_video(video_path_input: any) -> dict:
             except Exception as e:
                 print(f"[WARN] Could not open image {p}: {e}")
 
-    # Use first 60 frames for analysis/featured
-    frames_for_processing = all_frames[:60]
+    # Use first 30 frames for analysis/featured
+    frames_for_processing = all_frames[:30]
 
     # Step 2: Analyze primary media with AI
     # UNIFIED: Use the same frames we extracted above
     # frames_for_processing = all_frames[:60]
-    analysis = analyze_video(primary_path, frames=all_frames[:60])
+    analysis = analyze_video(primary_path, frames=all_frames[:30])
     categories = analysis.get("frame_categories", {}) if isinstance(analysis, dict) else {}
 
     # Step 3: Generate blog content
@@ -1568,10 +1628,10 @@ def process_uploaded_video(video_path_input: any) -> dict:
 
     # Step 4: Create featured image
     # UNIFIED: Use the same frames that were sent to the AI
-    featured_path = create_featured_image(title, all_frames[:60], safe_name, categories)
+    featured_path = create_featured_image(title, all_frames[:30], safe_name, categories)
 
     # Step 5: Create flyer
-    flyer_path = create_flyer(title, all_frames[:60], safe_name, categories)
+    flyer_path = create_flyer(title, all_frames[:30], safe_name, categories)
 
     # Step 6: Upload generated media
     featured_media = upload_media_to_wp(featured_path, f"featured-{safe_name}.jpg")