Spaces:
Sleeping
Sleeping
Commit History
5 days ago
-
fix: review findings in search, polling, clustering and ingestion
- HF Space queues pasted URLs for the local worker instead of downloading - Word search matches from the start of words ("ai" no longer hits "email") - Polling picks up new reels and drops queued reels deleted elsewhere - Recompute is claimed under a lock; reels saved mid-run get the new clusters - Bulk import drops repeated URLs; one key prompt for concurrent 401s - Connecting banner only appears if /health takes over 1.5s Claude-Session: https://claude.ai/code/session_01Y9VNLc1qrK3HkwUguu1p6s2dea10d -
feat: new reels inherit nearest cluster; drop unused fallbacks
- Save paths copy the cluster of the closest clustered reel (match_reels, >= 0.4) - Remove GET /reels, /reels/{id}/details and the no-Supabase frontend fallback - Remove init_local_db; reel load failures now show an error - Scheduled worker runs pythonw.exe directly instead of a VBScript Claude-Session: https://claude.ai/code/session_01Y9VNLc1qrK3HkwUguu1p6s737ff6a -
fix: API key handling, cheaper clustering and queue polling
- Wrong API key is cleared with a clear error; drop build-time VITE_API_KEY - Local backend binds 127.0.0.1 only - Cluster recompute reuses stored embeddings, repairs missing ones - Poll only queued reels; retry /health until the backend answers - Batch re-import counts reels by their real status Claude-Session: https://claude.ai/code/session_01Y9VNLc1qrK3HkwUguu1p6sf2e5f75
-
fix(backend): embed_reel never raises and never returns an unstorable vector
Claude-Session: https://claude.ai/code/session_01MV2z14HLUeGpAd4apqTpChe335b6a -
feat(backend): embed every reel on save for semantic search
Claude-Session: https://claude.ai/code/session_01MV2z14HLUeGpAd4apqTpCh9a322cb -
fix(backend): balanced clustering, retry failed transcriptions, pytest
- Clustering: switch from average-linkage agglomerative to k-means on normalized embeddings. On the real 740-reel library the old method put 96% of reels into two clusters (404/306) and left singletons; k-means gives 12 groups of 33-106. - Worker: a failed transcription now raises so the reel is retried with backoff; only the final attempt settles for a caption-only extraction. Previously any Whisper error silently produced a caption-only "done" row. - Tests: verify_pipeline.py's hand-rolled runner becomes pytest (backend/test_pipeline.py, monkeypatch fixtures). Drops two hasattr-only tests; adds clustering balance and transcription retry tests. Dev dependencies live in requirements-dev.txt. - Remove batch_import.py, superseded by POST /extract/batch. Claude-Session: https://claude.ai/code/session_0166W7S7XvHg5P2uyJtP4GiCf39e6ee -
chore: small review leftovers
- Recompute resets queued/failed reels to Unclustered instead of clustering their empty text as "Uncategorized content". - Log full LLM responses at debug, not info. - check_queue.py only inspects; it no longer deletes a hard-coded test row. - Dockerfile: drop unused curl. - Fix stale model names in .env.example, a docstring and two UI labels; give the page a real title. Claude-Session: https://claude.ai/code/session_0166W7S7XvHg5P2uyJtP4GiC9a7ec08 -
fix(api): return queued or failed rows from /extract/url instead of 500
Submitting a URL whose row already existed but was not finished (pending, processing or failed) returned 500: ExtractionResponse required a full ReelExtraction and a title, and it dropped status/error. The response model now matches what the table actually holds, so the dashboard shows the queue or failure banner for that reel. Claude-Session: https://claude.ai/code/session_0166W7S7XvHg5P2uyJtP4GiCd6c336d -
refactor(backend): one LLM failover helper, one worker loop
- _chat_json owns the model failover loop that extraction and cluster naming each carried a copy of. - run_worker() is the single queue loop; the in-process thread and run_worker.py both call it. A failed stale-claim recovery is logged instead of aborting startup. - Inline save_to_database into its three callers (the URL path keeps its re-check after the pipeline runs). - Drop the scikit-learn `affinity` fallback excluded by the >=1.5 pin and gather main.py's imports at the top. Claude-Session: https://claude.ai/code/session_0166W7S7XvHg5P2uyJtP4GiCfe69644 -
refactor(backend): route batch and cluster queries through db.py
- main.py no longer touches the Supabase client: batch status/enqueue use db.statuses_for_urls and db.insert_pending_reels. - Recompute writes one update per cluster (chunked) instead of one per reel. - reels_for_clustering pages past the 1000-row request cap. - Fold mark_failed_with_status into mark_failed and keep one FAILURE_STATUSES set in db.py. - Batch endpoints are sync so their blocking DB calls stay off the event loop. Claude-Session: https://claude.ai/code/session_0166W7S7XvHg5P2uyJtP4GiCef8a11d -
chore: delete retired clustering engine, uncalled endpoints, template leftovers
- Remove the old chunked-LLM clustering shims and the three tests that only exercised them; recompute calls cluster_topics_semantically directly. - Remove POST /reels/status and GET /clusters (+ db.cluster_counts): the frontend never calls them and derives clusters itself. - Remove the one-shot SQLite migration script (migration is done). - Remove Vite template assets, template README and the placeholder smoke test. Claude-Session: https://claude.ai/code/session_0166W7S7XvHg5P2uyJtP4GiCf94590b -
fix: load full reel library, unblock server during extraction, bulk drop
- Dashboard fetched only the newest 500 reels; page through Supabase so the whole library loads (754 rows today). db.list_reels pages too. - /extract/url, /extract/file and /extract/text were async but ran the blocking pipeline, freezing every other request. Make them sync so FastAPI runs them in its threadpool. - Bulk import drop zone called the audio drop handler and rejected saved_posts.json. Give it its own handler. - Drop the unused /reels search param, which interpolated raw text into a PostgREST filter. Claude-Session: https://claude.ai/code/session_0166W7S7XvHg5P2uyJtP4GiC82b2e56
-
fix(pipeline): add multi-model failover, URL normalization and recover failed reels
9209fc6 -
feat: improve semantic clustering and failed reel recovery
53163d6 -
fix(backend): make Groq LLM model configurable and default to qwen/qwen3.6-27b
ea39eba
-
fix(worker): add recovery and bounded retries
274b7fb
-
feat: add self-healing pipeline error handling and yt-dlp auto-updates
210f9f6 -
fix: failing jobs
9e99b0b
-
fix(styling,worker): improve light mode logo visibility and auto-disable cloud worker on HF Spaces
fbf284c
-
implement plans 004-008: async cluster recompute, details payload optimization, env template, API access gate, and vitest act warning fix
c488f66 -
docs(backend): fix duplicate step numbering comment in extract_file
d7a5d75 -
fix(backend): drop allow_credentials with wildcard CORS origin
bc3052e -
refactor(backend): migrate startup worker to FastAPI lifespan handler
1b28cc6 -
refactor(backend): extract shared _run_pipeline from url + worker paths
98a34cf -
refactor(backend): extract _chunked helper, replace 4 manual chunk loops
3a8b66b -
refactor(backend): hoist datetime/uuid imports, drop redundant local re-imports
a423a95
-
feat(backend): add ffmpeg auto-detection path injection helper for Windows
f92fae9
-
feat(batch): add Instagram saved_posts.json bulk import to frontend and API
a9a360b
-
feat(backend): background queue worker drains pending reels
5b6113e -
refactor(backend): route all persistence through Supabase db.py (drop SQLite)
Recompute now updates clusters per-row; delete + cache + reels/clusters all via db.12dc286 -
refactor(backend): extract process_reel_url from extract_url handler
Prereq for the Supabase queue worker — same pipeline, callable in-process.beedba5 -
feat(reels): delete endpoint + frontend delete buttons
DELETE /reels/{id} (404 when missing). Trash buttons on cards, table rows, and the detail modal with confirm. Also fix verify_pipeline polluting the real DB: recompute test now cleans up its seeded rows.412135f -
feat(clustering): merge pass consolidates fragmented chunk names + dedup
Chunked clustering named each batch independently, fragmenting clusters. Add _merge_cluster_names (one small names-only Groq call) to map duplicates to canonical names, plus keep-first dedup per id.d74a6e3 -
fix(clustering): chunk topics to stay under Groq TPM limit
All-in-one clustering of ~350 reels hit the 6000 TPM cap (413). Cluster in paced batches with index-shrunk payloads, threading discovered cluster names into later batches. Adds _cluster_one_chunk + chunking test.b26661a -
feat(ui): tabular insights view, cluster/tool/date filters, recompute
Tasks 6-9: load /clusters, in-memory filteredReels, cards|table toggle, InsightsTable, filter bar, Recompute clusters button. Also nudge the clustering prompt to merge near-duplicate themes (verified live: 7->4 clusters on the same data).3647e83 -
feat(clusters): models, recompute/list endpoints, cluster in /reels
Tasks 2-5: ClusterAssignment(s) models + cluster_topics_with_llm helper, POST /clusters/recompute, GET /clusters, cluster field in /reels rows. All mocked tests pass. Fixed recompute test to assign clusters by id (DB holds pre-existing rows, so positional mock was unreliable).9d9e576 -
feat(db): add idempotent cluster column migration
df3cc32 -
Initial commit: Instagram Reels structured-insight extractor
FastAPI backend (yt-dlp + Groq Whisper/Llama -> SQLite) and React/Vite dashboard. Verified end-to-end against a live reel. Fixes applied during verification: - action_items dict->List[str] 500 (field validator + prompt example) - silent-hook guard factored out and added to /extract/file - bump yt-dlp 2024.5.27 -> >=2026.3.17 (old pin broke IG extraction) - frontend: lucide-react Instagram icon removed -> Clapperboard Secrets (.env, cookies.txt), local db, and logs are gitignored.ad6f345