Spaces:
Sleeping
Sleeping
Commit History
5 days ago
-
fix: review findings in search, polling, clustering and ingestion
- HF Space queues pasted URLs for the local worker instead of downloading - Word search matches from the start of words ("ai" no longer hits "email") - Polling picks up new reels and drops queued reels deleted elsewhere - Recompute is claimed under a lock; reels saved mid-run get the new clusters - Bulk import drops repeated URLs; one key prompt for concurrent 401s - Connecting banner only appears if /health takes over 1.5s Claude-Session: https://claude.ai/code/session_01Y9VNLc1qrK3HkwUguu1p6s2dea10d -
chore: remove VBScript worker launcher (task runs pythonw.exe)
Claude-Session: https://claude.ai/code/session_01Y9VNLc1qrK3HkwUguu1p6s53a6927 -
feat: new reels inherit nearest cluster; drop unused fallbacks
- Save paths copy the cluster of the closest clustered reel (match_reels, >= 0.4) - Remove GET /reels, /reels/{id}/details and the no-Supabase frontend fallback - Remove init_local_db; reel load failures now show an error - Scheduled worker runs pythonw.exe directly instead of a VBScript Claude-Session: https://claude.ai/code/session_01Y9VNLc1qrK3HkwUguu1p6s737ff6a -
fix: API key handling, cheaper clustering and queue polling
- Wrong API key is cleared with a clear error; drop build-time VITE_API_KEY - Local backend binds 127.0.0.1 only - Cluster recompute reuses stored embeddings, repairs missing ones - Poll only queued reels; retry /health until the backend answers - Batch re-import counts reels by their real status Claude-Session: https://claude.ai/code/session_01Y9VNLc1qrK3HkwUguu1p6sf2e5f75
-
fix(backend): embed_reel never raises and never returns an unstorable vector
Claude-Session: https://claude.ai/code/session_01MV2z14HLUeGpAd4apqTpChe335b6a -
feat(backend): backfill script for reel search embeddings
Claude-Session: https://claude.ai/code/session_01MV2z14HLUeGpAd4apqTpChf6df0e9 -
feat(backend): embed every reel on save for semantic search
Claude-Session: https://claude.ai/code/session_01MV2z14HLUeGpAd4apqTpCh9a322cb -
feat(backend): store and query reel embeddings in the data layer
Claude-Session: https://claude.ai/code/session_01MV2z14HLUeGpAd4apqTpCha927b9c -
fix(backend): balanced clustering, retry failed transcriptions, pytest
- Clustering: switch from average-linkage agglomerative to k-means on normalized embeddings. On the real 740-reel library the old method put 96% of reels into two clusters (404/306) and left singletons; k-means gives 12 groups of 33-106. - Worker: a failed transcription now raises so the reel is retried with backoff; only the final attempt settles for a caption-only extraction. Previously any Whisper error silently produced a caption-only "done" row. - Tests: verify_pipeline.py's hand-rolled runner becomes pytest (backend/test_pipeline.py, monkeypatch fixtures). Drops two hasattr-only tests; adds clustering balance and transcription retry tests. Dev dependencies live in requirements-dev.txt. - Remove batch_import.py, superseded by POST /extract/batch. Claude-Session: https://claude.ai/code/session_0166W7S7XvHg5P2uyJtP4GiCf39e6ee -
chore: small review leftovers
- Recompute resets queued/failed reels to Unclustered instead of clustering their empty text as "Uncategorized content". - Log full LLM responses at debug, not info. - check_queue.py only inspects; it no longer deletes a hard-coded test row. - Dockerfile: drop unused curl. - Fix stale model names in .env.example, a docstring and two UI labels; give the page a real title. Claude-Session: https://claude.ai/code/session_0166W7S7XvHg5P2uyJtP4GiC9a7ec08 -
fix(api): return queued or failed rows from /extract/url instead of 500
Submitting a URL whose row already existed but was not finished (pending, processing or failed) returned 500: ExtractionResponse required a full ReelExtraction and a title, and it dropped status/error. The response model now matches what the table actually holds, so the dashboard shows the queue or failure banner for that reel. Claude-Session: https://claude.ai/code/session_0166W7S7XvHg5P2uyJtP4GiCd6c336d -
refactor(backend): one LLM failover helper, one worker loop
- _chat_json owns the model failover loop that extraction and cluster naming each carried a copy of. - run_worker() is the single queue loop; the in-process thread and run_worker.py both call it. A failed stale-claim recovery is logged instead of aborting startup. - Inline save_to_database into its three callers (the URL path keeps its re-check after the pipeline runs). - Drop the scikit-learn `affinity` fallback excluded by the >=1.5 pin and gather main.py's imports at the top. Claude-Session: https://claude.ai/code/session_0166W7S7XvHg5P2uyJtP4GiCfe69644 -
refactor(backend): route batch and cluster queries through db.py
- main.py no longer touches the Supabase client: batch status/enqueue use db.statuses_for_urls and db.insert_pending_reels. - Recompute writes one update per cluster (chunked) instead of one per reel. - reels_for_clustering pages past the 1000-row request cap. - Fold mark_failed_with_status into mark_failed and keep one FAILURE_STATUSES set in db.py. - Batch endpoints are sync so their blocking DB calls stay off the event loop. Claude-Session: https://claude.ai/code/session_0166W7S7XvHg5P2uyJtP4GiCef8a11d -
chore: delete retired clustering engine, uncalled endpoints, template leftovers
- Remove the old chunked-LLM clustering shims and the three tests that only exercised them; recompute calls cluster_topics_semantically directly. - Remove POST /reels/status and GET /clusters (+ db.cluster_counts): the frontend never calls them and derives clusters itself. - Remove the one-shot SQLite migration script (migration is done). - Remove Vite template assets, template README and the placeholder smoke test. Claude-Session: https://claude.ai/code/session_0166W7S7XvHg5P2uyJtP4GiCf94590b -
fix: load full reel library, unblock server during extraction, bulk drop
- Dashboard fetched only the newest 500 reels; page through Supabase so the whole library loads (754 rows today). db.list_reels pages too. - /extract/url, /extract/file and /extract/text were async but ran the blocking pipeline, freezing every other request. Make them sync so FastAPI runs them in its threadpool. - Bulk import drop zone called the audio drop handler and rejected saved_posts.json. Give it its own handler. - Drop the unused /reels search param, which interpolated raw text into a PostgREST filter. Claude-Session: https://claude.ai/code/session_0166W7S7XvHg5P2uyJtP4GiC82b2e56
-
fix(pipeline): add multi-model failover, URL normalization and recover failed reels
9209fc6 -
feat: improve semantic clustering and failed reel recovery
53163d6 -
fix(backend): make Groq LLM model configurable and default to qwen/qwen3.6-27b
ea39eba
-
fix(worker): add recovery and bounded retries
274b7fb -
fix: speed up local worker schedule to 5m and correct ReelModal queue/processing banner display
b3fa71d
-
feat: add self-healing pipeline error handling and yt-dlp auto-updates
210f9f6 -
fix: failing jobs
9e99b0b
-
fix(styling,worker): improve light mode logo visibility and auto-disable cloud worker on HF Spaces
fbf284c
-
implement plans 004-008: async cluster recompute, details payload optimization, env template, API access gate, and vitest act warning fix
c488f66 -
docs(backend): fix duplicate step numbering comment in extract_file
d7a5d75 -
chore(backend): drop unused import, use sys.exit in run_worker
d7f2eba -
fix(backend): drop allow_credentials with wildcard CORS origin
bc3052e -
fix(backend): guard row_to_record against malformed extracted_json
3721257 -
refactor(backend): migrate startup worker to FastAPI lifespan handler
1b28cc6 -
refactor(backend): extract shared _run_pipeline from url + worker paths
98a34cf -
refactor(backend): extract _chunked helper, replace 4 manual chunk loops
3a8b66b -
refactor(backend): hoist datetime/uuid imports, drop redundant local re-imports
a423a95
-
feat(backend): add scheduled background queue drain worker and silent runner scripts
0015145
-
feat(backend): add ffmpeg auto-detection path injection helper for Windows
f92fae9
-
chore: update MEMORY.md + add check_queue.py diagnostic script
- MEMORY.md: rewrote project snapshot to reflect fully live hybrid pipeline (Supabase, HF Space, Vercel, local worker); added 2026-06-07 session entry documenting the 3 iPhone Shortcut root causes fixed today: 1. id column needed gen_random_uuid() default for anon REST inserts 2. Shortcut URL/body were swapped (posting TO IG URL, not Supabase endpoint) 3. VITE_API_URL was empty in Vercel production - backend/check_queue.py: handy diagnostic to inspect and clean up pending/processing/failed rows in the Supabase queue - docs/superpowers/plans/2026-06-06-silent-windows-worker.md: commit previously untracked worker plan doc7ffeb0f -
feat(batch): add Instagram saved_posts.json bulk import to frontend and API
a9a360b -
fix(backend): pin httpx < 0.28.0 to resolve groq client crash on startup
79bc552
-
feat(backend): add standalone worker script and HF Space README config
0211181 -
feat(backend): background queue worker drains pending reels
5b6113e -
feat(backend): Supabase queue functions (claim/update/mark_failed)
9b5f12c -
chore(backend): pin websockets>=13,<16 for realtime asyncio compat
b7a2ba9 -
chore(backend): require supabase>=2.31.0 for new sb_secret_ key format
818cb11 -
feat(backend): one-time SQLite to Supabase migration script
ed01981 -
refactor(backend): route all persistence through Supabase db.py (drop SQLite)
Recompute now updates clusters per-row; delete + cache + reels/clusters all via db.12dc286 -
feat(backend): add Supabase db.py data layer (no wiring yet)
8f0c113 -
refactor(backend): extract process_reel_url from extract_url handler
Prereq for the Supabase queue worker — same pipeline, callable in-process.beedba5 -
feat(reels): delete endpoint + frontend delete buttons
DELETE /reels/{id} (404 when missing). Trash buttons on cards, table rows, and the detail modal with confirm. Also fix verify_pipeline polluting the real DB: recompute test now cleans up its seeded rows.412135f -
feat(clustering): merge pass consolidates fragmented chunk names + dedup
Chunked clustering named each batch independently, fragmenting clusters. Add _merge_cluster_names (one small names-only Groq call) to map duplicates to canonical names, plus keep-first dedup per id.d74a6e3 -
fix(clustering): chunk topics to stay under Groq TPM limit
All-in-one clustering of ~350 reels hit the 6000 TPM cap (413). Cluster in paced batches with index-shrunk payloads, threading discovered cluster names into later batches. Adds _cluster_one_chunk + chunking test.b26661a -
docs(spec): bulk import of saved reels via frontend + batch endpoint
Also add backend/batch_import.py CLI used to import the user's IG export now.9cd02e9