# BookmarkBrain — Local-First Semantic Bookmark Manager ## Vision 591+ Bookmarks in chaotischen Ordnern (15 Ebenen tief!) werden durchsuchbar per natuerlicher Sprache: "der Bash-Cheatsheet von neulich", "AWS Tutorials", "was hatte ich zu Vertical Farming gespeichert?" **Local-first.** Keine Cloud. Keine Accounts. Deine Daten bleiben bei dir. --- ## Architecture Overview ``` +------------------+ HTTP (localhost:7471) +------------------+ | Browser Extension| <--------------------------> | Python Backend | | (Chrome + Firefox)| | (FastAPI) | | | | | | - Popup UI | | - REST API | | - Quick Search | | - Import/Export | | - Save Bookmark | | - Auto-Tagger | | - Content Extract | | - Embedding Gen | +------------------+ +------------------+ | +------+------+ | SQLite | | + Embeddings| | (numpy .npy)| +-------------+ ``` --- ## Tech Stack | Component | Technology | Why | |-----------|-----------|-----| | Backend | Python 3.11+ / FastAPI | Async, schnell, Samuels Hauptsprache | | Database | SQLite (via sqlite3) | Local-first, zero config, single file | | Embeddings | `paraphrase-multilingual-MiniLM-L12-v2` | 384 dims, DE+EN, CPU-fast | | Vector Search | sqlite-vec (SQLite Extension) | Vektor-Suche + SQL in einer DB | | Frontend | htmx + Jinja2 (Web UI) | Wie StudyAgent, bewaehrter Stack | | Browser Extension | WXT Framework | Ein Codebase fuer Chrome + Firefox | | Build | uv (Python), pnpm (Extension) | Schnell, modern | ### Warum sqlite-vec statt ChromaDB/FAISS/NumPy? - **Single File:** Bookmarks + Vektoren in einer SQLite-DB, kein separates .npy - **SQL + Vektor kombinierbar:** `WHERE domain = 'github.com' AND embedding MATCH ?` - **Zero Dependencies:** Pure C Extension, `pip install sqlite-vec` - ChromaDB hat 100+ transitive Dependencies, FAISS ist overkill bei 2000 Bookmarks ### Warum `paraphrase-multilingual-MiniLM-L12-v2`? - 384 Dimensionen, 118M Parameter, ~450 MB - 50+ Sprachen inkl. Deutsch und Englisch - ~2000-4000 Sätze/Sekunde auf CPU - ONNX-Beschleunigung moeglich (2-3x Speedup) - Deine 591 Bookmarks: Embedding in <1 Sekunde --- ## Data Model ```sql CREATE TABLE bookmarks ( id INTEGER PRIMARY KEY AUTOINCREMENT, url TEXT NOT NULL UNIQUE, title TEXT NOT NULL, description TEXT DEFAULT '', -- User-supplied or auto-extracted context TEXT DEFAULT '', -- "Was wollte ich damit?" - User note folder_path TEXT DEFAULT '', -- Original browser folder path domain TEXT DEFAULT '', -- Extracted from URL favicon_url TEXT DEFAULT '', -- Auto-generated auto_tags TEXT DEFAULT '[]', -- JSON array of AI-generated tags user_tags TEXT DEFAULT '[]', -- JSON array of user-set tags content_snippet TEXT DEFAULT '', -- First ~500 chars of page content -- Metadata created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, last_visited_at TIMESTAMP, visit_count INTEGER DEFAULT 0, is_dead BOOLEAN DEFAULT FALSE, -- Link check: 404? -- Search embedding_text TEXT DEFAULT '' -- Combined text used for embedding ); CREATE VIRTUAL TABLE bookmarks_fts USING fts5( title, description, context, auto_tags, user_tags, content_snippet, content='bookmarks', content_rowid='id' ); CREATE INDEX idx_bookmarks_domain ON bookmarks(domain); CREATE INDEX idx_bookmarks_created ON bookmarks(created_at); ``` Embeddings als sqlite-vec Virtual Table: ```sql CREATE VIRTUAL TABLE vec_bookmarks USING vec0( id INTEGER PRIMARY KEY, embedding FLOAT[384] ); ``` Alles in einer einzigen SQLite-Datei: `data/bookmarks.db` --- ## Features (nach Prioritaet) ### Phase 1: Core (Woche 1-2) 1. **Bookmark Import** — Netscape HTML Parser (deine Datei), Chrome/Firefox JSON 2. **SQLite Storage** — Schema wie oben, FTS5 fuer Volltextsuche 3. **Embedding Generation** — Alle Bookmarks embedden bei Import 4. **Semantic Search API** — `GET /api/search?q=bash+cheatsheet` → Top-10 Ergebnisse 5. **Web UI** — FastAPI + htmx, Suchfeld + Ergebnisliste + Tag-Filter 6. **Auto-Tagging** — Clustering + Keyword-Extraction aus Titel/URL/Domain ### Phase 2: Browser Extension (Woche 3) 7. **Extension Popup** — Suchfeld, zeigt Ergebnisse aus Backend 8. **Quick Save** — Aktuelle Seite speichern mit einem Klick 9. **Content Extraction** — Content Script extrahiert Meta-Description, OG-Tags, ersten Absatz 10. **Context Note** — "Was wollte ich damit?" Textfeld beim Speichern ### Phase 3: Intelligence (Woche 4) 11. **Dead Link Checker** — Async HEAD requests, markiert tote Links 12. **Duplicate Detection** — Embedding-Similarity > 0.95 → wahrscheinlich Duplikat 13. **Smart Collections** — Auto-gruppierung via Embedding-Cluster (K-Means) 14. **"Related Bookmarks"** — Klicke einen Bookmark → zeige aehnliche 15. **Bookmark Health Dashboard** — Stats, tote Links, Duplikate, ungenutztes ### Phase 4: Polish (Woche 5+) 16. **Export** — Netscape HTML, JSON, Markdown 17. **Keyboard Shortcuts** — Cmd+K Style Quick-Search 18. **Dark Mode** 19. **Backup/Restore** 20. **Browser Bookmark Sync** — Periodic import from browser --- ## Project Structure ``` ~/Projekte/BookmarkBrain/ ├── PLAN.md # Dieser Plan ├── FOR_SMLFLG.md # Architektur-Doku ├── pyproject.toml # Python project config (uv) ├── backend/ │ ├── __init__.py │ ├── main.py # FastAPI app, routes │ ├── database.py # SQLite connection, schema, migrations │ ├── models.py # Pydantic models │ ├── search.py # Semantic + FTS search engine │ ├── embeddings.py # Model loading, embedding generation │ ├── importer.py # Netscape HTML / JSON import │ ├── tagger.py # Auto-tag generation │ ├── health.py # Dead link checker, duplicate detection │ └── templates/ │ ├── base.html │ ├── index.html # Main search UI │ ├── bookmark.html # Single bookmark detail/edit │ └── components/ │ ├── search_results.html │ ├── tag_cloud.html │ └── stats.html ├── extension/ │ ├── wxt.config.ts │ ├── package.json │ ├── entrypoints/ │ │ ├── popup/ # Search popup │ │ │ ├── App.tsx │ │ │ └── main.tsx │ │ ├── background.ts # Service worker │ │ └── content.ts # Page content extraction │ └── utils/ │ └── api.ts # Backend API client ├── data/ # Gitignored │ └── bookmarks.db # SQLite + sqlite-vec (alles in einer Datei) ├── scripts/ │ ├── import_bookmarks.py # CLI import tool │ └── reindex.py # Rebuild embeddings └── tests/ ├── test_importer.py ├── test_search.py └── test_tagger.py ``` --- ## API Endpoints ``` GET / # Web UI (htmx) GET /api/search?q=...&limit=10 # Semantic + FTS hybrid search GET /api/bookmarks # List all (paginated) GET /api/bookmarks/:id # Single bookmark POST /api/bookmarks # Create bookmark PUT /api/bookmarks/:id # Update (tags, context, etc.) DELETE /api/bookmarks/:id # Delete POST /api/import # Import from file (Netscape HTML / JSON) GET /api/export?format=html # Export GET /api/tags # All tags with counts GET /api/stats # Dashboard stats GET /api/health # Dead link report GET /api/duplicates # Suspected duplicates POST /api/reindex # Rebuild all embeddings ``` --- ## Search Algorithm (Hybrid) ```python import struct def search(query: str, limit: int = 10) -> list[SearchResult]: query_vec = model.encode(query) blob = struct.pack(f"{len(query_vec)}f", *query_vec) # 1. Semantic Search (sqlite-vec) semantic = db.execute(""" SELECT b.*, v.distance FROM vec_bookmarks v JOIN bookmarks b ON b.id = v.id WHERE v.embedding MATCH ? AND k = ? ORDER BY v.distance """, (blob, limit * 2)).fetchall() # 2. Full-Text Search (SQLite FTS5) fts = db.execute( "SELECT rowid, rank FROM bookmarks_fts WHERE bookmarks_fts MATCH ?", (query,) ).fetchall() # 3. Hybrid Score: 0.7 * semantic + 0.3 * fts_normalized return merge_and_rerank(semantic, fts)[:limit] ``` --- ## Auto-Tagging Strategy Kein LLM noetig fuer 500-2000 Bookmarks. Stattdessen: 1. **Domain-basierte Tags** — `github.com` → "github", `youtube.com` → "video" 2. **URL-Path Keywords** — `/docs/python/` → "python", "docs" 3. **TF-IDF auf Titel** — Haeufige Woerter ueber alle Bookmarks = schlechte Tags, seltene aber wiederkehrende Woerter = gute Tags 4. **Folder-Path Tags** — `Software Arc > Linux > Code` → "software-architecture", "linux", "code" 5. **Embedding Cluster Labels** — K-Means Cluster, repraesentativstes Wort pro Cluster --- ## Browser Extension ↔ Backend Communication **Localhost HTTP** (nicht Native Messaging): - Einfacher zu implementieren - Funktioniert identisch in Chrome + Firefox - CORS Headers im Backend setzen - Backend muss laufen (Systemd Service oder manuell) ```typescript // extension/utils/api.ts const API_BASE = "http://localhost:7471/api"; export async function search(query: string) { const res = await fetch(`${API_BASE}/search?q=${encodeURIComponent(query)}`); return res.json(); } export async function saveBookmark(data: BookmarkData) { const res = await fetch(`${API_BASE}/bookmarks`, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify(data), }); return res.json(); } ``` --- ## Dependencies ### Python (backend) ``` fastapi>=0.115 uvicorn[standard]>=0.30 sentence-transformers>=3.0 sqlite-vec>=0.1 numpy>=1.26 jinja2>=3.1 python-multipart>=0.0.9 httpx>=0.27 # For dead link checking beautifulsoup4>=4.12 # For HTML bookmark parsing scikit-learn>=1.4 # For K-Means clustering, TF-IDF ``` ### Node (extension) ``` wxt # Browser extension framework typescript ``` --- ## Differenzierung vs. Prior Art | Feature | Raindrop | Memex | Linkding | Hoarder | BookmarkBrain | |---------|----------|-------|----------|---------|---------------| | Local-first | No | Yes | Self-host | Self-host | **Yes, zero infra** | | Semantic Search | No | No | No | No (Meilisearch) | **Yes, local embeddings** | | AI Tags | Pro ($3/mo) | No | No | LLM-based | **Local, free** | | "Was wollte ich?" | No | Annotations | No | Notes | **First-class feature** | | Browser Extension | Yes | Yes | Yes | Yes | **Yes (Chrome+FF)** | | Setup Complexity | None (cloud) | Extension only | Docker | Docker+4 containers | **`pip install` + start** | | Multilingual | N/A | N/A | N/A | N/A | **DE+EN nativ** | **Kern-Differenzierung:** 1. Semantic Search LOKAL ohne Cloud/LLM-API — unique 2. "Was wollte ich damit?" als First-Class-Feature 3. Zero-Setup: kein Docker, kein Cloud-Account, `pip install && run` 4. Chaotische Ordner → saubere AI-Tags automatisch --- ## Run Commands ```bash # Setup cd ~/Projekte/BookmarkBrain uv venv && source .venv/bin/activate uv pip install -e . # Import existing bookmarks python scripts/import_bookmarks.py ~/Downloads/bookmarks_05.03.26.html # Start backend uvicorn backend.main:app --reload --port 7471 # Extension dev (separate terminal) cd extension && pnpm install && pnpm dev ``` --- ## Open Decisions (fuer Samuel) 1. **Port:** 7471 OK? (Mnemonic: "BRAIN" auf dem Telefon) 2. **Extension zuerst oder Web UI zuerst?** Plan sagt Web UI first. 3. **ONNX-Optimierung sofort oder spaeter?** Plan sagt spaeter (Phase 4). 4. **Soll der Backend-Server als Systemd-Service laufen?** Oder manuell starten? 5. **Projektname:** BookmarkBrain? Oder was anderes?