Plan: Add multi-company document linking via mentioned_company_ids
Add company extraction strategy to multi-company linking plan
Claude
Unverified signature
e8eea15
Adds section 6 detailing how ingestion flows will detect primary vs
mentioned companies:
- Primary: folder structure, metadata, or LLM fallback
- Mentioned: fast string matching + LLM extraction during claim extraction
- Integration point with existing claim extraction to get mentions "for free"
https://claude.ai/code/session_01UikxPeNTorkkCPaA5mhJAY
Design document for extending the KG schema to support documents that
mention multiple companies (e.g., expert interview transcripts discussing
competitors). Approach: keep existing company_id as primary, add new
mentioned_company_ids JSONB array column.
https://claude.ai/code/session_01UikxPeNTorkkCPaA5mhJAY
simplify plan: one class per platform, backend via env
Claude
Unverified signature
deee3f2
Simplified architecture:
- Each scraper is self-contained (CrunchbaseScraper, YouTubeScraper, etc.)
- Scraping backend (ScrapingDog/ScraperAPI) configured via SCRAPING_BACKEND env var
- No --provider/--backend CLI flags, just --source
- CLI: `scrape --source crunchbase --company "Anthropic"`
https://claude.ai/code/session_01RPiNB6tC5gyNrUDNz4F2in
refactor plan: separate classes per API/platform
Claude
Unverified signature
8b147a8
Architecture now has clear separation:
- ScrapingBackend ABC: ScrapingDogBackend, ScraperAPIBackend, DirectFetchBackend
- NewsSearchProvider ABC: CrunchbaseProvider, NewsAPIProvider, GoogleNewsProvider
- YouTube: YouTubeSearchClient, YouTubeTranscriptClient
- Podcast: ListenNotesClient, SpotifyPodcastClient, WhisperTranscriber
Sources compose these clients via dependency injection for better
testability and maintainability.
https://claude.ai/code/session_01RPiNB6tC5gyNrUDNz4F2in
Simplified approach:
- No RSS/pub-sub, just periodic CLI-triggered batch runs
- Use ScrapingDog/ScraperAPI for web content
- Company-centric: scrape for companies already in KG
- Same pipeline as GDrive: Source → GCS → Supabase
https://claude.ai/code/session_01RPiNB6tC5gyNrUDNz4F2in
add plan for public scraping ingestion routes
Claude
Unverified signature
46e2946
Comprehensive plan for adding WebScraperSource, YouTubeSource, and
PodcastSource to ingest news coverage, founder interviews, and podcast
transcripts into the knowledge graph.
https://claude.ai/code/session_01RPiNB6tC5gyNrUDNz4F2in