IMPLEMENTATION_LOG.md

# Bio Nexus — Implementation Log

> Sprint-by-sprint build history. This file answers "what shipped and when."
> For *why* Bio Nexus exists and how it's architected, see `MASTER_PLAN.md`.
> For coding conventions, see `RULES.md`. For the job worker, see `durable-worker-design.md`. For the DB, see `schema.md`.

**Last Updated:** August 2026

---

## Track A — Prototype Sprint (18 days) ✅

Goal: one golden path — sequence in → BLAST → AI-interpreted report. Everything else deferred to Track B.

- Day 0: repo/infra pre-flight (Supabase project, storage bucket, API keys, NCBI access confirmed)
- Days 1–2: job CRUD spine (FastAPI + Next.js scaffolds, dummy data)
- Days 3–4: NCBI BLAST integration, demo-mode fallback for slow/queued searches
- Day 5: sequence input + validation, wizard shell
- Days 6–7: wizard → job creation → live processing screen (first end-to-end run)
- Days 8–9: raw results rendering (hits table, alignment view, no AI yet)
- Days 10–11: AI interpretation layer (Gemini), hedged-language prompt template
- Day 12: guest → account upgrade flow
- Day 13: dashboard (job list)
- Day 14: landing page polish
- Day 15: error states (invalid input, NCBI timeout, zero hits, rate-limit queueing)
- Day 16: deploy (Vercel + Railway/Render)
- Day 17: demo prep, cached demo sequences as backup
- Day 18: demo day

## Track B — Full Build (post-prototype)

| Sprint | Scope | Status |
|---|---|---|
| 1–2 | Pairwise alignment, `parent_job_id` pipeline chaining | ✅ |
| 3–4 | UniProt annotation, PDB structure fetch + 3Dmol.js viewer | ✅ |
| 5–6 | Multi-method MSA (ClustalOmega/MUSCLE/Kalign/MAFFT/T-Coffee), phylogenetic tree (NJ/UPGMA/ML) | ✅ |
| 7 | Pathway lookup (Reactome/WikiPathways) + diagram viewer | ✅ |
| 8 | Onboarding tutorial + `/learn` docs (10+ pages, glossary, inline help) | ✅ |
| 9–10 | PDF/JSON export, cache-hit tracking, Sentry monitoring | ✅ |
| 11 | Pipeline v2 engine (8-step BLAST→UniProt→MSA→Phylo→Domains→Pathway→AlphaFold→AI), pairwise/domain/structure depth | ✅ |
| 12 | Drug discovery compute: AutoDock Vina docking, MD simulation, ADMET, function prediction, protein interactions | ✅ |
| 13 | Sequencing MVP: FASTQ QC → trimming → assembly/consensus → variant calling → annotation (SARS-CoV-2 reference) | ✅ |
| 14 | Reliability: AI fallback chain (Groq→Gemini→Ollama), share links for all job types, wizard job persistence | ✅ |
| 15 | Design system v4.0: dark-only OLED theme, semantic tokens, Geist/Phosphor, landing rebuild | ✅ |

## Platform Hardening (ongoing, not tied to a single sprint)

- API key system (`sk_bio_` prefix, SHA-256 hashed)
- Share links (token-based, all job types)
- Guest → account upgrade via `linkIdentity`
- Enhanced dashboard/jobs/settings UI
- Durable job worker (see `durable-worker-design.md`) — replaced in-request job execution for docking/MD/function-predict/sequencing/pipeline

## Tool Verification Audit (Aug 2026)

Live audit of the four 🔴 tools from `FEATURE_VERIFICATION_CHECKLIST.md`; every fix is a `fix(...)` commit.

- `fix(pathways)` `bde6aea` — Pathway enrichment no longer depends on the Reactome `/token/{token}/pathways` endpoint (which 404s on the URL-encoded token). Reads the `pathways` array directly from the projection response, parses species, and surfaces `geneRatio` + per-pathway p-value to the UI and TSV export. Live-verified: 20 pathways for the TP53 gene set.
- `fix(interactions)` `d2ea003` — STRING-DB viewer upgraded: evidence-type filter (experimental / database / co-expression / text-mining, threshold 0.3), JSON export alongside PNG/TSV, and LearnPopover tooltips on the combined + evidence score headers linking to `learn/interactions`. Backend endpoint live-verified (TP53 → 5 partners).

## Structure Prep Hardening (Aug 2026) — pre-work for techspec.md §1–§3

All seven audit findings from `techspec.md` fixed; manifest rows added to `FEATURE_VERIFICATION_CHECKLIST.md` (Structure Prep / De Novo / Export / Page Capture sections).

- **A1** `fix(structure-prep)` — PyMOL cleanup now uses the importable open-source wheel (`pymol-open-source-whl`, provides the `pymol2` module — no binary, no X server) with a loudly-logged Biopython fallback. fpocket built from source (`Discngine/fpocket` v4.0.0, needs `libnetcdf-dev`) in both API Dockerfiles — it is *not* packaged in Debian repos (pool dir 404s). Note: spec's "pymol2 on PyPI" assumption corrected during implementation.
- **A2+A6** — new `backend/migrations/008_structure_prep_jobs.sql`: durable job table with `user_id` ownership + RLS (pattern-matched to ngs_jobs). Router persists state per step and scopes status reads to the owning user (`require_user_id`, matching every other job router).
- **A3** — broken chains that can't be repaired (no accession or no >80% template) proceed but are tagged `chain_integrity="broken_unrepaired"`; repaired runs get `"repaired"`; clean runs `"intact"`. ESMFold window-splicing deferred with reason in manifest.
- **A4** — PDB ID / UniProt accession (reuses `UNIPROT_RE` from identifier_resolution) / SMR template IDs validated by regex before any network call; sequence alphabet + 10–768 length checked at request time.
- **A5** — `swissmodel.expasy.org`, `cfold.bme.uic.edu`, `api-inference.huggingface.co` added to SSRF allowlist; every outbound call in `structure_prep.py` now routes through `validate_url()`.
- **A7** — CASTp polling timeout now sets explicit `castp_status: "timed_out"` (plus `skipped`/`running`/`complete`/`error`); fpocket gets a symmetric `fpocket_status` incl. `unavailable`.
- Tests: `test_structure_prep_validation.py` (25 cases) + existing identifier-resolution suite pass.

## De Novo Pipeline Branch — tier 6 (Aug 2026) — techspec.md §1

Unknown sequences that fail tiers 1–5 (no BLAST hit / resolution exhausted) now complete the pipeline as a first-class "de novo" run instead of erroring. Confidence tiers threaded through the whole context: `identified` (direct/xref/name) → `homolog` (sequence/idmapping similarity) → `de_novo`.

- **Backend** `feat(pipeline)` — `identifier_resolution.resolve_to_uniprot` returns an enriched result (`status`/`confidence`) and never `None`; `pipeline_v2._execute` branches at the BLAST step: zero-hit proteins set `denovo_mode`, mark blast complete with an explanatory note, and run `_run_denovo_steps()` instead of failing. Resolution-exhausted runs (hits exist, no accession) fall through to homolog-confidence gating: `_run_domains_or_denovo()` swaps InterProScan5 sequence-search for accession lookup; `_run_alphafold_or_esmfold()` swaps ESMFold HF API for the AlphaFold repository.
- **New service** `app/services/de_novo.py` — `interpro_sequence_search()` (EBI iprscan5 submit/poll/JSON → normalized domain shape), `esmfold_structure()` (predict + mean pLDDT from CA B-factors, alphafold-shaped card with inline `pdb_text`), `composition_stats()` + `function_hints()` (honest heuristic labeling). MSA/phylo/pathway steps are marked failed with "Unavailable for de novo sequences — no identified homolog" rather than silently empty.
- **Frontend** — new `ConfidenceBadge` (three states: cyan identified / amber homolog / dashed-purple de_novo), new `DeNovoPanel` (dashed-border composition + function-hints card replacing UniProt panel when the bundle carries `_de_novo`); job page renders de novo results where it previously showed a dead-end "No significant similarity found" card; `AlphaFoldViewer` gains a `pdbData` prop so inline ESMFold models render without a fetch (title: "Predicted structure (ESMFold)"); pathway card shows an explicit unavailable notice in de novo runs; docking button hidden when no receptor PDB URL exists.
- Tests: `test_de_novo_branch.py` (8 cases incl. pipeline-level zero-hit acceptance test with mocked EBI/ESMFold) — 63 backend tests green across the three touched suites; frontend `tsc --noEmit` + `next lint` clean.
- Pending live verification: real ESMFold/InterProScan E2E run; migration `008` application (pre-work §A2).

## Structure Export — techspec.md §2 (Aug 2026)

- **New router** `app/routers/structure_export.py` (`/api/structure-export`) — authenticated downloads keyed by UniProt accession (AlphaFold DB file pattern) or 4-char PDB ID (RCSB). Formats: `.pdb`, `.cif`, and a genuine PyMOL session `.pse` built server-side with the pymol-open-source wheel (cartoon + `spectrum b` pLDDT rainbow + transparent background pre-applied — no manual coloring, acceptance §2.5). Missing upstream models map to clean 404s; pymol2-less deployments get an explicit 503 instead of a broken file.
- **Honest-scoping rule honored (§2.3b)** — ChimeraX/VMD ship as the preferred coordinate format plus a client-generated command script (`.cxc` / `.tcl`), never fake session files. `.cxs`/VMD state exports remain documented exclusions.
- **Docking exports** — worker now persists `result_sdf` (docked poses) and `receptor_pdb`; new endpoints `GET /api/docking/result/{id}/complex.pdb` (receptor + docked ligand merged) and `/ligand.sdf`. Legacy rows pre-dating persistence return an actionable "re-run" message rather than junk.
- **Frontend** — new `StructureExportMenu.tsx` dropdown: PDB/mmCIF/PyMOL-session items on `AlphaFoldViewer`'s toolbar (hidden without an accession — de novo models export via the existing PDB button); Complex-PDB/Ligand-SDF items on the docking results page. Downloads go through the authed axios instance as blobs.
- Tests: `test_structure_export.py` (12 cases — routing, URL patterns, format gate, 404 mapping, empty-pymol-output guard).

## Page Capture + Final Synthesis — techspec.md §3 (Aug 2026)

- **New table** `migrations/009_page_captures.sql` — one row per external source queried during a run, keyed `(job_id, source)` with RLS; stores the human-facing page URL, title, extracted text sections, figure image URLs, and an honest `fetch_status` (`captured`/`failed`/`skipped`). Must be applied in Supabase SQL Editor.
- **New service** `app/services/page_capture.py` — stdlib-only HTML extraction (no new deps), per-host rate limiter (1.5 s minimum interval), every fetch through `ssrf.validate_url()` (§3.2: no new SSRF surface). Captures are strictly best-effort fire-and-forget; failures still record a row with `fetch_status="failed"` so coverage is auditable.
- **Wiring** — `pipeline_v2._finalize_context()` queues captures for NCBI top hit, UniProt entry, RCSB structure, InterPro entry, AlphaFold DB page, and Reactome pathway browser — derived from actual run results; de novo runs correctly record no annotation-source pages.
- **New service** `app/services/final_synthesis.py` — deterministic findings assembly from real step results, each tagged with the run's confidence tier (identified/homolog/de_novo) and source-tool page link, plus tier-appropriate caveats. An optional LLM pass polishes wording only (`_mode: llm_polished|deterministic`); it can never invent findings or block the pipeline.
- **Frontend** — new `FinalReport.tsx` panel rendered above the AI interpretation card on the job page: headline, summary, per-finding rows with confidence badges and source-page links, caveats footer.
- Tests: `test_page_capture_synthesis.py` (9 cases — extraction, failure-honesty, rate limiting, tier threading, de novo caveats, deterministic fallback). Suite total: 84 tests green across the five touched files.

## Open / Next

- Phase 3 depth: RNA-seq differential expression, larger file storage and compute
- Phase 4 (not started): lab workspaces, custom pipeline builder, institution licensing, public API access

---

*When a sprint or phase ships, add one row here — do not restate the shipped-feature list in `MASTER_PLAN.md`.*