rules/do_not_commit.md

metadata

name: pedagogical_learning_and_concepts
description: >-
  Rules for leaving core logic blank for learning and documenting interview &
  real-world applications of applied concepts.

Pedagogical Learning & Applied Concepts Rule

🛑 Strict Zero-Solution Policy for Codebase Blanks

When implementing core algorithmic procedures, mathematical transformations, or diagnostic filters designated for learning:

1. NEVER Write the Working Solution in the Codebase:

   • The agent MUST NOT write out the completed algorithm or logic below the TODO/BLANK comment block.
   • The agent MUST NOT write "fallback implementations" or "reference solutions" in the file that defeat the purpose of the blank.
   • The function body for any [BLANK N] MUST strictly contain ONLY:
     1. Specification Docstring: Precise description of the task, input argument types, and expected output dictionary/list structure.
     2. Step-by-Step TODO Checklist: Algorithmic steps, formula hints, and constants to use.
     3. Non-Crashing Stub: A clean, minimal stub (pass with return None or return []) so the module compiles without syntax errors.

2. Where Solutions Belong:

   • The learner writes their own implementation directly in the file.
   • When the learner asks for review or guidance, the agent provides architectural guidance, code reviews, bug diagnoses, and algorithmic explanations in the chat dialogue.
   • The agent must NEVER overwrite or inject full code into a [BLANK N] file unless the user explicitly commands: "Apply this solution to the file" or "Fill in the blank for me".

3. Mandatory Deep-Dive Documentation for Every Concept: For every blank logic block and advanced concept, the agent MUST explicitly document:

   • Applied Concepts & Mathematical Foundations: Formula derivation, algorithmic complexity (Big-O time and space), and trade-offs.
   • Real-World Industry Applications: Exactly where and how this algorithmic pattern is used in modern production engineering (e.g., Google Search, Netflix/Spotify recommendation, Uber dispatch, distributed caching, LLM RAG pipelines).
   • Technical Interview Edge: How this concept appears in FAANG/tier-1 technical interviews, talking points to impress interviewers, edge cases to mention, and system design trade-offs.

---

Documented Concepts & Learning Algorithms

1. "Root-Cause" Prerequisite Back-Tracing Engine

• Concept: A student struggling in an advanced topic usually suffers from gaps in foundational prerequisites.
• Algorithm (DAG Traversal & Topological Sort):
  • Models the curriculum as a Directed Acyclic Graph (DAG) with courses/topics as nodes and dependency relations as directed edges.
  • Recursively traverses upstream dependencies using Depth-First Search (DFS) with cycle detection.
  • Applies Kahn's Algorithm or post-order DFS reverse sorting to topologically order prerequisites so fundamentals are studied first.
• Real-World Applications:
  • Build Systems (Bazel, Gradle, Webpack): Dependency resolution graphs to determine optimal compilation orders.
  • Task Schedulers (Apache Airflow, Celery): Orchestrating asynchronous DAG workflows without circular deadlocks.
  • Package Managers (npm, pip, pub): Resolving transitive dependency trees.
• Interview Talking Points:
  • Explain cycle detection using graph coloring (White/Gray/Black node states) or in-degree tracking.
  • Contrast BFS vs DFS traversal: DFS discovers deep dependency chains; BFS uncovers immediate co-requisites.

---

2. Peer-Validated "High-Yield" Heatmap (Wilson Score Confidence Interval)

• Concept: Average user ratings (e.g. 5/5 stars) fail when comparing an item with 1 positive review (100%) against an item with 98 positive reviews out of 100 (98%).
• Algorithm (Wilson Score Interval):
  • Calculates the statistically robust lower bound of a Bernoulli parameter confidence interval: $$\hat{p} = \frac{n_{\text{pos}}}{n}, \quad \text{Lower Bound} = \frac{\hat{p} + \frac{z^2}{2n} - z \sqrt{\frac{\hat{p}(1-\hat{p})}{n} + \frac{z^2}{4n^2}}}{1 + \frac{z^2}{n}}$$
  • Filters ratings specifically to the targeted cohort: students who initially failed or scored $< 40%$ and improved after studying the resource.
• Real-World Applications:
  • Reddit & Hacker News Ranking: Sorting comments and posts by "Best" using Wilson score intervals to prevent fresh single-upvoted comments from beating established top-quality comments.
  • Amazon / Yelp Review Sorting: Ranking verified purchaser reviews by helpfulness.
  • E-Commerce Fraud Detection: Identifying anomalous conversion spikes from low-sample sellers.
• Interview Talking Points:
  • Explain why naive Bayesian average or arithmetic mean fails on cold-start items with few reviews.
  • Discuss the $z$-score tradeoff (typically $z = 1.96$ for 95% confidence).

---

3. Dynamic Micro-Resource Bundling (Multi-Choice 0/1 Knapsack)

• Concept: Students have strict time budgets (e.g., 30 mins, 60 mins). A study bundle must pack the highest academic utility without exceeding the time limit, while ensuring multi-modal diversity (reading, video, flashcards, past-year problems).
• Algorithm:
  • Dynamic programming 0/1 Knapsack formulation where $W$ is the student's study duration budget, weights $w_i$ are resource durations in minutes, and values $v_i$ are utility scores derived from historical cohort success deltas.
• Real-World Applications:
  • Cloud Resource Allocation (AWS EC2 Spot Instances / Kubernetes Pod Scheduling): Packing workloads with CPU/RAM requirements onto hardware servers.
  • AdTech Bidding (Google Ads / Meta): Selecting the optimal set of auction ads to show in a fixed screen slot to maximize expected revenue.
• Interview Talking Points:
  • Compare DP table space complexity $O(n \cdot W)$ vs 1D array space optimization $O(W)$.
  • Discuss Fractional Knapsack (Greedy algorithm with value-to-weight ratio) vs 0/1 Knapsack (requires DP or Branch-and-Bound).

---

4. Rich Concept Explanations & Deep-Dive Research Links on Flashcards

• Concept: Transforming static flashcards into dynamic, multi-tier knowledge anchors with real-world paper citations (ArXiv, PubMed, OpenStax, Semantic Scholar).
• Algorithm:
  • Hybrid RAG (Retrieval-Augmented Generation) combining dense vector embeddings with cross-encoder re-ranking.
  • Generating LaTeX mathematical formulas and structured markdown citations dynamically.
• Real-World Applications:
  • Perplexity AI / Consensus: Academic search engines synthesizing direct answers backed by peer-reviewed research papers.
  • Medical Diagnostic Assistants (Epic Systems / Med-PaLM): Providing clinical recommendations with verbatim citations to clinical trial publications.
• Interview Talking Points:
  • Explain RAG retrieval vs parametric memory hallucination.
  • Discuss Reciprocal Rank Fusion (RRF) for merging dense vector searches with sparse BM25 keyword searches.

---

5. Live YouTube Search API Integration with Quota Caching

• Concept: Real-time retrieval of targeted educational video lectures with metadata caching and fallback heuristics.
• Algorithm:
  • Exponential backoff retry strategy with LRU memory caching to conserve YouTube Data API v3 quota (10,000 units/day limit).
  • Title relevance scoring via Levenshtein distance and token set similarity.
• Real-World Applications:
  • Video Streaming Aggregators: Integrating external media platforms without hitting third-party rate limits.
  • Content Distribution Networks (CDNs): Cache invalidation and cache-aside patterns.
• Interview Talking Points:
  • Discuss Cache-Aside vs Write-Through vs Write-Back caching strategies.
  • How to handle API rate limiting using token bucket and leaky bucket algorithms.

---

Pinned Master Roadmap & Priority Registry

#	Improvement	Origin Date	Complexity	Priority	Status
1	Micro-Diagnostic 2-Min Adaptive Radar & Risk MLP	2026-07-31	High	🔥 High	✅ COMPLETED
2	Live YouTube API Search & Cache-Aside Ranker ([BLANK 2])	2026-08-25	Medium	Medium	✅ COMPLETED
3	Automated CI/CD Pipeline & Hugging Face Deployment	2026-09-25	Medium	🔥 High	✅ COMPLETED
4	Study Bundler Zero-Topic Contamination & DAG Fallback	2026-09-26	High	🔥 High	✅ COMPLETED
5	Flashcard Deep-Dive Explanations, LaTeX Math & Papers	2026-08-17	Low – Medium	Medium	✅ COMPLETED
6	On-Demand Textbook RAG Fallback in Bundler (Tier 2)	2026-09-26	Medium	Medium	✅ COMPLETED
7	resources.db Hygiene Script & Cognitive/Relevance Filters	2026-09-26	Low	Medium	✅ COMPLETED
8	Real student_quiz_logs Table & Wilson Score Heatmap	2026-08-21	Medium	🔥 High	✅ COMPLETED
9	Native Interactive Mermaid Flowchart Render	2026-08-08	Medium	🔥 High	❌ REJECTED (Abandoned due to layout defects: text cut off, isolated corner nodes)
10	PDF.js Web Worker & Semantic Chunk Retrieval Fix	2026-09-14	Low – Medium	Medium	✅ COMPLETED
11	Multimodal PYQ Photo Scanner & Timestamp Mapper	2026-07-31	High	Medium	✅ COMPLETED
12	Bayesian / Deep Knowledge Tracing (BKT/DKT)	2026-07-31	High	Medium	📋 PENDING
13	Lens Transformation Cache Layer	2026-07-31	Low	Low	✅ COMPLETED

---

Pinned Architectural Blueprint: Flashcard Topic Relevance, External Question Datasets & Heatmap Integration

1. Root Cause of Topic Contamination

• Legacy Database Noise: Historic records in resources.db contained mismatched topic labels (e.g., OOP record #10212 labeled as "Computer Networks").
• Unconstrained Fallback: When candidates fell short for a time budget, app/bundler.py previously executed an unrestricted ORDER BY RANDOM() on General CS, leaking dynamic programming and recursion into unrelated networking decks.

2. The 4-Pillar Architectural Solution

1. Dual-Tier Generation Model:
   • Primary: Curated, high-yield university past-exam question banks (PYQs).
   • Dynamic: On-demand PDF RAG generation using authoritative uploaded textbooks (Topic01-Foundation.pdf, Algorithms-JeffE.pdf, DiscMathII.pdf).
2. Prerequisite-Aware Fallback (DAG Traversal):
   • If candidate resources for a specific topic run short, the bundler NEVER falls back to random General CS.
   • Instead, it traverses COURSE_PREREQUISITES to only source foundational prerequisite cards (e.g. WIA1005 (Computer Networks) falls back to its immediate ancestor WIA1003 (Computer Systems Architecture)).
3. Semantic Relevance Embedding Guard:
   • Computes cosine similarity between candidate flashcard embeddings and canonical syllabus topics: $$\text{CosineSim}(\vec{E}{\text{card}}, \vec{E}{\text{course}}) = \frac{\vec{E}{\text{card}} \cdot \vec{E}{\text{course}}}{|\vec{E}{\text{card}}| |\vec{E}{\text{course}}|}$$
   • Any candidate scoring $< 0.65$ is automatically purged from the study bundle.
4. Adaptive Wilson Score Heatmap Integration:
   • Integrates student_quiz_logs to adaptively boost high-heat flashcards (proven to help struggling students master threshold concepts).
   • Extensible to any newly registered course in kAvailableCourses.