INTERVIEW_BOOK.md

SimPPL Research Assignment — Interview Master Guide

This document is a single-source interview guide for the SimPPL research assignment you submitted. It explains the project end-to-end, the design choices you made, alternatives you considered, and includes demo steps, talking points, and likely interview questions with concise answers. Use this to prepare for the interview and to narrate your demo confidently.

---

1. One-sentence summary

A reproducible analysis and interactive exploration system for an 8.8k Reddit-post dataset that provides timelines, semantic search, clustering, and network insights via precomputed artifacts, a Flask API, and a React frontend.

2. Goals and success criteria

• Make the dataset explorable and explainable for a demo (fast visualizations, evidence-backed claims).
• Provide semantic search and cluster summaries to support discoverability and narrative explanations.
• Precompute expensive artifacts offline so the runtime API is fast and reliable for interviews.
• Keep the system simple and reproducible (small infra footprint) while demonstrating production-like design tradeoffs.

Success = reproducible artifacts (CSV/JSON/NumPy files), running Flask demo locally, and a frontend that visualizes timelines, clusters, and the network.

3. High-level architecture (plain words)

1. Input: hf-clean-worktree/data.jsonl — wrapper-structured Reddit posts (some records are nested under data).
2. Preprocessing: parse JSONL → normalize fields → export metadata_full.csv / subreddit_timeline.csv and ingest into SQLite for query convenience.
3. Embeddings: compute 384-d sentence embeddings for title + selftext using all-MiniLM-L6-v2 and save embeddings.npy.
4. Reduce dims with UMAP → umap_coords.npy for visualization.
5. Cluster embeddings (KMeans) → cluster IDs + TF-IDF top terms per cluster for labels.
6. Build graph (author/post crosslinks) and compute PageRank and Louvain communities → graph.json.
7. Serve via a read-only Flask API that loads these artifacts at startup; frontend (React+Vite) queries the API to render interactive visualizations.
8. LLM service is used only for optional human-readable summaries and is backed by a caching layer.

The system prioritizes precompute-first design for demo stability and speed.

4. Data & preprocessing (in-depth)

4.1 Input format

• The dataset is JSONL. Each line may be a plain object or a wrapper like { "kind": "t3", "data": { ... } }.
• Important fields: id, subreddit, author, title, selftext, created_utc, score, num_comments, domain.

4.2 Cleaning steps (exact operations)

• Unwrap records when data exists.
• Normalize created_utc to timezone-aware created_dt (UTC) and derive date and hour columns.
• Coerce numeric columns: score, num_comments via pd.to_numeric(errors='coerce').
• Derive text = title + " \n\n " + selftext for embedding.
• Compute title_len, selftext_len, and engagement = score + num_comments as simple features.
• Deduplicate by id if duplicates present.

Why these steps? They ensure downstream models and visualizations have consistent, typed inputs and allow temporal analyses and simple heuristics for spike detection.

4.3 Exports

• metadata_full.csv: cleaned table of posts.
• subreddit_timeline.csv: per-subreddit min/max dates and counts (used for Gantt/timeline view).
• story_pack.csv: aggregated story candidates (day-level spikes, significant increases) to seed demo narratives.

Example code snippet (paraphrased):

# unwrap
obj = json.loads(line)
rec = obj.get('data', obj)
# normalize
rec['created_dt'] = pd.to_datetime(rec['created_utc'], unit='s', utc=True)
rec['text'] = (rec.get('title','') or '') + '\n\n' + (rec.get('selftext','') or '')

5. Embeddings: choice, process, and alternatives

Chosen: all-MiniLM-L6-v2 (sentence-transformers)

• Why: small, fast, good semantic quality for retrieval; open-source; 384-d embeddings are compact and efficient to store and query on a laptop.
• Practical: embedding 8.8k posts is fast (< 10 minutes on a modest CPU with batching; much faster with GPU) and results fit comfortably in memory.

How they were computed

• Embed title + selftext with batching (e.g., batch_size=64).
• L2-normalize vectors and save embeddings.npy (shape: [8799, 384]).
• Validate: check norms ~1 and shape matches number of posts.

Alternatives considered and why not chosen

• OpenAI embeddings (text-embedding-3-small): higher quality sometimes, but costs and API dependency make it worse for reproducible interviews.
• Larger SBERT models (e.g., all-mpnet-base-v2): better quality but bigger model size and slower inference; unnecessary for demo scale.
• TF-IDF or BM25: good for exact lexical matches, cheap, and deterministic. Kept as a fallback for exact-match queries, but embeddings enable semantic matches across phrasing.

Analogy: embeddings are like mapping every sentence into a point in meaning-space; nearby points have similar semantic content even if words differ.

6. Dimensionality reduction: UMAP

• Purpose: create 2D coords (umap_coords.npy) for scatter plots that preserve neighborhood structure for visual clusters.
• Why UMAP: faster than t-SNE, better preserves both global and local structure for exploratory visuals, deterministic with fixed seed (good for reproducible demos).
• Alternatives: PCA (linear, fast but poor for semantic geometry), t-SNE (good local structure but slower and unstable for reproducible demos).

Note: UMAP is for visualization only — all retrieval/clustering uses full-dim embeddings.

7. Clustering & labeling

Clustering method: KMeans (primary) with optional HDBSCAN

• KMeans chosen for simplicity, deterministic behavior, and interpretability when using a fixed k tuned for the dataset.
• HDBSCAN considered for discovering variable-sized clusters and noise handling; good alternative if cluster counts are unknown.

How cluster labels are generated:

• For each cluster, compute TF-IDF across text of cluster members.
• Pick top N terms as human-readable cluster label(s) and store representative post IDs for examples.

Why TF-IDF for labels? It's simple, transparent, and produces interpretable keywords that reflect cluster content.

8. Graph construction and network metrics

Graph types built

• Author → Post edges (author wrote post).
• Co-activity edges: edges between subreddits/accounts that co-occur in the dataset (same external domain or cross-post patterns).

Metrics computed

• PageRank: importance score of nodes (authors or posts).
• Louvain community detection: unsupervised module detection.

Why graphs? They reveal structural relationships (who/which community drives conversations) and surface interesting nodes for demonstration.

Alternatives: pure co-occurrence matrices or bipartite-projection heuristics; the chosen approach balances interpretability and visual appeal.

9. Storage & runtime choices

• SQLite: compact, zero-ops server, perfect for local demos and quick indexing. Chosen because it's simple and reproducible for interview demos.
• Alternatives: Postgres (stronger concurrency & SQL features) — not necessary for local demo.
• Embedding storage: embeddings.npy (NumPy) — fast to load in Python and memory efficient for tens of thousands of vectors.
• Caching: lightweight file-based cache or Redis for the LLM summaries in production.

Tradeoffs: keep infra light for interview; prioritize reproducibility over heavy infra.

10. API design (Flask) — endpoints and behavior

Design principle: read-only endpoints, load artifacts at startup, return JSON for the frontend.

Core endpoints (implementations in backend/routes):

• /search?q=: embed query (server-side or client-provided embedding), compute cosine similarity over normalized embeddings, return top-k posts with cluster context and TF-IDF snippets.
• /clusters: returns cluster metadata, top terms, sizes, example posts.
• /network: returns graph.json or graph slice (subgraph) for a subreddit or node.
• /timeseries: returns daily counts and spike markers for a subreddit or global view.
• /summary?id=: returns cached LLM summary for a post or cluster.

Why Flask: simple, lightweight, widely understood; enough for demo and easy to containerize.

11. Frontend (React + Vite) — pages and UX

Main pages: Landing, Overview, Search, Clusters, Network, TimeSeries, Embeddings Viewer. Key components: Metric cards, ForceGraph, Gantt timeline, UMAP scatter with hover details, Search results with evidence.

UX principle: show the evidence (actual posts, dates, and counts) supporting any claims made by summaries.

12. LLM usage & safety

• Purpose: cluster/cluster-summary generation and optional query summarization.
• Controlled use: kept separate from retrieval and graph logic, used to produce human-readable narratives only.
• Caching policy: summaries cached with prompt+artifact fingerprint. Versioned invalidation when artifacts change.
• Safety: sanitize inputs, avoid hallucination by including evidence in prompts (top-k posts as context) and keep LLM outputs always accompanied by direct citations.

13. Demo script (10–12 minute walkthrough)

1. Start: one-line project summary and goals.
2. Data facts: show metadata_full.csv counts, date range, number of subreddits, authors. (Invoke a simple command or show dashboard metric card.)
3. Timeline: open Timeline/Gantt and explain how per-subreddit min/max dates were computed and what they show.
4. Spike detection: show a spike day, open Story Pack entry, read the top posts and explain why the spike occurred.
5. Search: type a query, show semantic matches vs TF-IDF fallback; highlight how embeddings enable matches despite lexical differences.
6. Clusters: open cluster view, read TF-IDF label, expand example posts and show LLM summary with citations.
7. Network: display force graph around a high-PageRank node and explain PageRank and Louvain communities.
8. Wrap-up: summary of architectures, tradeoffs, and next steps if given more time.

Timing tips: practice transitions, have 2–3 example queries/spikes you can jump to.

14. Questions you're likely asked — concise answers

Q: Why precompute embeddings? A: Precompute keeps runtime fast and deterministic; computing embeddings on each query adds latency and cost.

Q: Why all-MiniLM-L6-v2 instead of OpenAI? A: It's open-source (reproducibility), fast, and cost-free. OpenAI may give higher quality but introduces cost and external dependency.

Q: How do you prevent LLM hallucinations? A: Provide context (top-k posts) in prompts and show evidence in UI. Cache summaries and present them with explicit citations.

Q: How would this scale? A: Replace SQLite with Postgres or vector DB (Milvus, Pinecone), use FAISS/Annoy for approximate nearest neighbors, and move LLM summarization to an async service with rate limits.

Q: Why KMeans? A: Deterministic and simple; HDBSCAN is better for non-spherical clusters — used if we detect noise or highly imbalanced cluster sizes.

15. Appendix — commands and quick-run guide

Prereqs: Python 3.10+, venv, node for frontend.

Create venv & install backend deps (example):

python -m venv .venv
source .venv/bin/activate   # or .venv\Scripts\Activate.ps1 on Windows
pip install -r backend/requirements.txt

Run preprocessing / EDA (SQLite queries):

This repo's EDA is based on querying backend/data/posts.db directly (no notebook required). The selective chunking approach (post-level by default, chunk long-tail posts for search) is justified with measured text-length distribution stats in MASTER_EXPLANATION.md (see the “EDA: Post length distribution (chunking justification)” section).

Compute embeddings (script example):

python backend/pipeline/embed.py --input metadata_full.csv --model all-MiniLM-L6-v2 --output backend/data/embeddings.npy

Start Flask API (dev):

cd backend
flask run --host=0.0.0.0 --port=5000

Start frontend (dev):

cd frontend
npm install
npm run dev

16. Appendix — glossary & quick concept primers

• Embedding: vector representation of text capturing semantic information.
• UMAP: manifold learning algorithm for projecting high-dimensional vectors to 2D/3D while preserving neighbor relations.
• KMeans: centroid-based clustering that partitions data into k clusters.
• HDBSCAN: density-based clustering that detects clusters of varying density and can mark noise.
• PageRank: algorithm that scores nodes by link structure; high PageRank nodes are 'important'.
• Louvain: community detection algorithm that groups nodes by modularity.

17. Final notes & preparation checklist

• Memorize 3 key demo stories (specific spike + explanation).
• Be ready to explain why each artifact was precomputed and how it maps to a UI element.
• Practice the 10–12 minute walkthrough and a few deeper-dive answers (embedding tuning, cluster selection, LLM prompt design).

---

If you want, I can now:

• Export this guide to research-engineering-intern-assignment/INTERVIEW_BOOK.md (done).
• Generate a printable PDF of the guide.
• Create a shorter one-page cheat-sheet and slide deck for interview use.