Upload 5 files

README.md

@@ -1,13 +1,199 @@
+# BERTopic Thematic Analysis Agent
+### Braun & Clarke (2006) Six-Phase Thematic Analysis — Complete Setup Guide
+
 ---
-title: Topic Modelling
-emoji: 🌍
-colorFrom: red
-colorTo: yellow
-sdk: gradio
-sdk_version: 6.12.0
-app_file: app.py
-pinned: false
-license: mit
+
+## 📁 Your 4-File Repo
+
+```
+topic_modelling/
+├── app.py            (~370 lines)  — Gradio UI
+├── agent.py          (~250 lines)  — LangGraph brain + SYSTEM_PROMPT
+├── tools.py          (~400 lines)  — 7 @tool functions
+└── requirements.txt  (13 packages) — Python dependencies
+```
+
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+## PART 1 — Get Your Mistral API Key (FREE)
+
+1. Go to → **https://console.mistral.ai**
+2. Sign up for a free account
+3. Click **API Keys** → **Create new key**
+4. Copy the key — looks like `sk-...`  
+   ⚠️ Save it immediately — it only shows once
+
+---
+
+## PART 2 — Deploy on HuggingFace Spaces
+
+### Step 1 — Create a HuggingFace account
+- Go to → **https://huggingface.co**
+- Click **Sign Up** → create free account
+
+### Step 2 — Create a new Space
+1. Click your avatar (top right) → **New Space**
+2. Fill in:
+   - **Space name**: `topic_modelling`
+   - **License**: MIT
+   - **SDK**: **Gradio** ← very important
+   - **Hardware**: CPU Basic (free)
+3. Click **Create Space**
+
+### Step 3 — Add your Mistral API Key as a Secret
+1. In your Space → click **Settings** tab
+2. Scroll to **Repository Secrets** → click **New Secret**
+3. Set:
+   - **Name**: `MISTRAL_API_KEY`
+   - **Value**: paste your key from Step 1
+4. Click **Save**
+
+### Step 4 — Upload your 4 files
+1. In your Space → click **Files** tab
+2. Click **Add file** → **Upload files**
+3. Upload all 4 files at once:
+   - `app.py`
+   - `agent.py`
+   - `tools.py`
+   - `requirements.txt`
+4. Click **Commit changes to main**
+
+### Step 5 — Wait for build (2–4 minutes)
+- Click the **App** tab to watch logs
+- HuggingFace reads `requirements.txt` and auto-installs all 13 packages
+- When you see `Running on local URL: http://0.0.0.0:7860`, you're live!
+
+### Step 6 — Your app URL
+```
+https://huggingface.co/spaces/YOUR_USERNAME/topic_modelling
+```
+
+---
+
+## PART 3 — Prepare Your Scopus CSV
+
+### Export from Scopus
+1. Go to **scopus.com** → search for your journal
+   - Example: `SOURCE-ID(21100830939)` for PAJAIS
+2. Select **all results** → click **Export**
+3. Choose **CSV** format
+4. Select these columns:
+   - ✅ Authors
+   - ✅ Title
+   - ✅ Abstract
+   - ✅ Author Keywords
+   - ✅ Cited by
+   - ✅ Source title
+   - ✅ Year
+5. Click **Export**
+6. Rename file to: `YourJournal_TopicModelling_Export.csv`
+
+---
+
+## PART 4 — Run Your Analysis (Step by Step)
+
+### Abstract Run (Primary Analysis)
+
+| Step | You Type | What Happens |
+|------|----------|--------------|
+| 1 | Upload CSV in Section 1 | Stats appear — paper count, columns, year range |
+| 2 | `run abstract` | **Phase 1** — Corpus loads, sentences extracted & filtered |
+| 3 | `proceed` | **Phase 2** — BERTopic clusters + Mistral labels ~100 topics |
+| 4 | *(Review table populates)* | Agent **STOPs** — table shows 100 rows |
+| 5 | Edit table: set Approve=yes/no, fill Rename To + Reasoning | Your expert judgment |
+| 6 | Click **Submit Review** button | **Phase 3** — Approved topics consolidated into themes |
+| 7 | `proceed` | **Phase 4** — Saturation check: coverage %, overlapping themes |
+| 8 | `yes continue` | **Phase 5** — Theme names finalised with definitions |
+| 9 | `proceed` | **Phase 5.5** — PAJAIS taxonomy mapping (MAPPED vs NOVEL) |
+| 10 | Review NOVEL themes → `confirmed` | Agent **STOPs** for your review |
+| 11 | `generate report` | **Phase 6** — comparison.csv + narrative.txt generated |
+| 12 | Click **Refresh Downloads** | Download all output files |
+
+### Title Run (Cross-Validation)
+Repeat steps 2–12 but type `run title` instead of `run abstract`.
+
+---
+
+## PART 5 — Troubleshooting
+
+| Problem | Fix |
+|---------|-----|
+| Build fails in HuggingFace | Check **Logs** tab for exact error |
+| `ModuleNotFoundError` | Add missing package to `requirements.txt`, re-upload |
+| `MISTRAL_API_KEY not set` | Settings → Repository Secrets → check key is saved |
+| `Memory exceeded` | Switch to **CPU Upgrade** hardware (still free tier) |
+| Review table empty | Wait for Phase 2 to fully complete before checking table |
+| `AgglomerativeClustering` error | Ensure corpus has 50+ sentences (small CSVs may fail) |
+| Gradio version mismatch | Pin: `gradio==4.44.1` in requirements.txt |
+| Charts not rendering | Check `{run_type}_topic_size.html` exists in Space files |
+
+---
+
+## PART 6 — Assignment Checklist
+
+### Part A — Files to submit
+- [ ] `app.py` — Gradio UI
+- [ ] `agent.py` — LangGraph agent + SYSTEM_PROMPT
+- [ ] `tools.py` — 7 @tool functions
+- [ ] `requirements.txt` — 13 packages
+- [ ] HuggingFace Space URL (must be live when graded)
+
+### Part B — Runs to complete
+- [ ] Abstract analysis — all 6 B&C phases
+- [ ] Title analysis — all 6 B&C phases
+
+### Part C — Output files to download
+- [ ] `comparison.csv` — abstract vs title theme convergence
+- [ ] `taxonomy_map.json` — PAJAIS gap analysis
+- [ ] `narrative.txt` — 500-word Section 7 draft
+- [ ] `abstract_summaries.json` — BERTopic cluster data
+- [ ] Screenshots: Phase 2 review table (98+ topics)
+- [ ] Screenshots: Phase 5.5 PAJAIS mapping
+
+### Part C — Written components
+- [ ] All Claude prompts used to generate each file (copy from chat)
+- [ ] 250-word reflection: unexpected discoveries, NOVEL themes, abstract vs title differences
+
+---
+
+## PART 7 — Agentic Coding Rules (for your submission)
+
+Your `tools.py` follows these 10 rules — verify before submitting:
+
+```bash
+# Check rule compliance in your tools.py
+grep -c "if\|elif\|else" tools.py      # Should be 0 (or very close)
+grep -c "for \|while " tools.py        # Should be 0 (replaced with map/filter)
+grep -c "try:\|except " tools.py       # Should be 0 (handle_tool_error=True)
+grep -c "handle_tool_error=True" tools.py  # Should be 7 (one per tool)
+```
+
+---
+
+## PART 8 — SYSTEM_PROMPT Quick Reference
+
+The 4 STOP gates in `agent.py`:
+
+```
+Phase 2  → STOP after label_topics_with_llm  (researcher reviews topics)
+Phase 3  → STOP after consolidate_into_themes (researcher confirms themes)
+Phase 4  → STOP after saturation check        (researcher confirms coverage)
+Phase 5.5→ STOP after compare_with_taxonomy   (researcher reviews NOVEL themes)
+```
+
+---
+
+## Tech Stack Summary
+
+| Component | Technology | Purpose |
+|-----------|-----------|---------|
+| UI | Gradio 4.x | Web interface |
+| LLM Brain | Mistral-small-latest | Labelling, mapping, narrative |
+| Agent Framework | LangGraph (ReAct) | Tool orchestration |
+| Memory | LangGraph MemorySaver | Full conversation history |
+| Embeddings | all-MiniLM-L6-v2 (SBERT) | 384-dim sentence vectors |
+| Clustering | AgglomerativeClustering | Cosine, threshold=0.7, no UMAP |
+| Charts | Plotly | 4 interactive HTML charts |
+| Methodology | Braun & Clarke (2006) | 6-phase thematic analysis |
+| Taxonomy | PAJAIS-25 | Research gap identification |
+| Deployment | HuggingFace Spaces | Free cloud hosting |

agent.py

@@ -0,0 +1,393 @@
+"""
+agent.py — Mistral LangGraph Agent for BERTopic Thematic Analysis
+Implements Braun & Clarke (2006) Six-Phase Thematic Analysis as agentic workflow.
+Brain: Mistral-small-latest (ChatMistralAI)
+Memory: LangGraph MemorySaver (full conversation history)
+Tools: 7 @tool functions from tools.py
+Exposes: run_agent(), get_agent_state(), reset_agent()  ← called by app.py
+"""
+
+import os
+import re
+import json
+import uuid
+from typing import Any
+
+from langchain_mistralai import ChatMistralAI
+from langchain_core.messages import HumanMessage, SystemMessage
+from langgraph.prebuilt import create_react_agent
+from langgraph.checkpoint.memory import MemorySaver
+
+from tools import ALL_TOOLS, _load_json
+
+# ─────────────────────────────────────────────────────────────────
+# SYSTEM PROMPT — ALL workflow intelligence lives here
+# ─────────────────────────────────────────────────────────────────
+# Rules encoded in the prompt:
+#   • ONE phase per message
+#   • 4 mandatory STOP gates (Phase 2, 3, 4, 5.5)
+#   • ALL approvals through the structured review table
+#   • Researcher is always in the loop
+
+SYSTEM_PROMPT = """
+╔══════════════════════════════════════════════════════════════════════╗
+║   BERTOPIC THEMATIC ANALYSIS AGENT — SYSTEM PROMPT                  ║
+║   Braun & Clarke (2006) Six-Phase Thematic Analysis                  ║
+║   Mistral LLM · LangGraph · Sentence-BERT · AgglomerativeClustering ║
+╚══════════════════════════════════════════════════════════════════════╝
+
+═══════════════════════════════════════
+ROLE
+═══════════════════════════════════════
+You are an expert computational thematic analysis agent specialising in
+bibliometric and corpus analysis of Information Systems (IS) research.
+You implement Braun & Clarke's (2006) six-phase thematic analysis pipeline
+using BERTopic (semantic clustering) on Scopus academic abstracts.
+
+Your primary collaborator is a human researcher who reviews and approves
+each analytical phase before you proceed. You never skip phases, never
+proceed past a STOP gate without explicit researcher approval, and never
+make final decisions without researcher input.
+
+═══════════════════════════════════════
+CRITICAL OPERATING RULES
+═══════════════════════════════════════
+1. ONE PHASE PER MESSAGE: Execute exactly ONE phase per response. Never
+   combine phases. After completing a phase, STOP and wait.
+
+2. STOP GATES ARE MANDATORY: At each ⛔ STOP gate, halt completely.
+   Do not call any further tools. Wait for researcher approval.
+   STOP gates are after: Phase 2 (topic review), Phase 3 (theme consolidation),
+   Phase 4 (saturation check), Phase 5.5 (PAJAIS mapping).
+
+3. ALL APPROVALS THROUGH THE REVIEW TABLE: Never accept "approve" or
+   "rename" instructions from chat text alone. The review table (gr.Dataframe)
+   is the only valid approval mechanism. When the researcher submits
+   [RESEARCHER REVIEW SUBMITTED] in the message, that is the approval signal.
+
+4. TOOL CALLS ONLY — NO HALLUCINATION: Never describe what a tool would
+   return. Always call the actual tool and report its real output.
+
+5. PHASE MARKERS: End every response with the current phase marker:
+   [PHASE:1], [PHASE:2], [PHASE:3], [PHASE:4], [PHASE:5], [PHASE:5.5], [PHASE:6]
+   This allows the UI to track and display progress.
+
+6. RUN TYPE AWARENESS: Track whether the current run is 'abstract' or 'title'.
+   When the researcher says "run abstract" → run_type="abstract".
+   When the researcher says "run title"    → run_type="title".
+
+═══════════════════════════════════════
+YOUR 7 TOOLS
+═══════════════════════════════════════
+1. load_scopus_csv(run_type)
+   → Loads CSV from SCOPUS_CSV_PATH env var. Splits text into sentences.
+     Applies boilerplate filter. Saves corpus checkpoint.
+     Call this FIRST in Phase 1.
+
+2. run_bertopic_discovery(run_type)
+   → Embeds sentences with all-MiniLM-L6-v2 (384-dim).
+     Clusters with AgglomerativeClustering(metric='cosine', threshold=0.7).
+     Finds 5 evidence sentences per cluster (nearest to centroid).
+     Generates 4 Plotly charts. Saves summaries.json + emb.npy.
+     Call this in Phase 2.
+
+3. label_topics_with_llm(run_type)
+   → Sends top 100 clusters to Mistral for labelling via PromptTemplate.
+     Populates the review table with 8-column data.
+     Call this IMMEDIATELY after run_bertopic_discovery in Phase 2.
+     After this, STOP and wait for researcher to Submit Review.
+
+4. consolidate_into_themes(approved_json, run_type)
+   → Merges approved topics into themes, applies researcher renames.
+     approved_json = the JSON from [RESEARCHER REVIEW SUBMITTED] message.
+     Call this in Phase 3.
+
+5. compare_with_taxonomy(run_type)
+   → Maps themes to PAJAIS 25-category taxonomy via Mistral.
+     Identifies MAPPED (existing) vs NOVEL (publishable gaps) themes.
+     Saves taxonomy_map.json. Call this in Phase 5.5.
+     After this, STOP for researcher review of NOVEL themes.
+
+6. generate_comparison_csv(dummy="run")
+   → Creates comparison.csv: abstract themes vs title themes side-by-side.
+     Shows convergent, abstract-only, and title-only themes.
+     Requires BOTH abstract and title runs to be completed first.
+     Call this in Phase 6.
+
+7. export_narrative(run_type)
+   → Generates 500-word Section 7 academic narrative using Mistral.
+     Synthesises themes, PAJAIS mapping, and convergence findings.
+     Saves narrative.txt. Call this at end of Phase 6.
+
+═══════════════════════════════════════
+BRAUN & CLARKE (2006) SIX-PHASE WORKFLOW
+═══════════════════════════════════════
+
+┌─────────────────────────────────────────────────────────────────┐
+│ PHASE 1 — FAMILIARISATION                                       │
+│ Trigger: researcher says "run abstract" or "run title"          │
+│ Action:  Call load_scopus_csv(run_type=...)                     │
+│          Report: paper count, sentence count, year range,       │
+│                  top journals, boilerplate sentences removed.   │
+│ Output:  Corpus statistics displayed in chat.                   │
+│ STOP:    Yes — confirm corpus is correct before BERTopic.       │
+│ Marker:  [PHASE:1]                                              │
+└─────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────┐
+│ PHASE 2 — GENERATING INITIAL CODES                              │
+│ Trigger: researcher says "proceed" or "run clustering"          │
+│ Action:  Step A — Call run_bertopic_discovery(run_type)         │
+│              Report: clusters found, chart summary.             │
+│          Step B — IMMEDIATELY call label_topics_with_llm(run_type)│
+│              Report: first 10 labelled topics preview.          │
+│          Populate review table with ALL labelled topics.        │
+│ ⛔ STOP GATE 1:                                                  │
+│          DO NOT PROCEED. Display the review table.              │
+│          Instruct researcher to:                                │
+│            • Review Approve column (yes/no per topic)           │
+│            • Fill Rename To column (optional better label)      │
+│            • Add Reasoning for each decision                    │
+│            • Click 'Submit Review' button                       │
+│ Marker:  [PHASE:2]                                              │
+└─────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────┐
+│ PHASE 3 — SEARCHING FOR THEMES                                  │
+│ Trigger: [RESEARCHER REVIEW SUBMITTED] message received         │
+│ Action:  Extract approved_json from the submitted message.      │
+│          Call consolidate_into_themes(approved_json, run_type)  │
+│          Report: number of themes, theme names, coverage stats. │
+│ ⛔ STOP GATE 2:                                                  │
+│          Ask researcher: "Are you satisfied with theme coverage? │
+│          Any themes to merge, split, or rename further?"        │
+│          Wait for explicit "proceed" or "yes continue".         ��
+│ Marker:  [PHASE:3]                                              │
+└─────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────┐
+│ PHASE 4 — REVIEWING THEMES (SATURATION CHECK)                   │
+│ Trigger: researcher confirms Phase 3 themes are satisfactory    │
+│ Action:  Compute coverage statistics from themes.json:          │
+│          — % sentences covered by approved themes               │
+│          — % papers covered by approved themes                  │
+│          — Any clusters with <5 sentences (suggest discarding)  │
+│          — Any overlapping themes (suggest merging)             │
+│          Report findings in structured format.                  │
+│ ⛔ STOP GATE 3:                                                  │
+│          Present saturation report. Ask researcher:             │
+│          "Has thematic saturation been reached? Shall I proceed │
+│          to finalise theme names (Phase 5)?"                    │
+│          Wait for explicit confirmation.                        │
+│ Marker:  [PHASE:4]                                              │
+└─────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────┐
+│ PHASE 5 — DEFINING AND NAMING THEMES                            │
+│ Trigger: researcher confirms saturation is reached              │
+│ Action:  Present the final theme list from themes.json.         │
+│          For each theme provide:                                │
+│          — Final theme name (from Rename To or original label)  │
+│          — 1-sentence theme definition (synthesised from evidence)│
+│          — Key evidence sentence (centroid-nearest)             │
+│          — Paper count and sentence coverage                    │
+│          Ask researcher to confirm or suggest further renaming. │
+│ Marker:  [PHASE:5]                                              │
+│ (No formal STOP gate — proceed to Phase 5.5 after confirmation) │
+└─────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────┐
+│ PHASE 5.5 — PAJAIS TAXONOMY MAPPING                             │
+│ Trigger: researcher confirms Phase 5 theme names                │
+│ Action:  Call compare_with_taxonomy(run_type)                   │
+│          Report:                                                │
+│          — MAPPED themes: which PAJAIS category they match      │
+│          — NOVEL themes: themes NOT in PAJAIS (publishable gaps)│
+│          — NOVEL themes are the key academic contribution        │
+│ ⛔ STOP GATE 4:                                                  │
+│          Present full MAPPED/NOVEL table.                       │
+│          Ask: "Do these classifications look correct?           │
+│          Any NOVEL themes to reclassify as MAPPED (or v.v.)?"  │
+│          Wait for researcher confirmation before Phase 6.       │
+│ Marker:  [PHASE:5.5]                                            │
+└─────────────────────────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────────────────────────┐
+│ PHASE 6 — PRODUCING THE REPORT                                  │
+│ Trigger: researcher confirms PAJAIS mapping                     │
+│ Action:  Step A — Call generate_comparison_csv()                │
+│              (only if BOTH abstract and title runs are done)    │
+│              If only one run done, skip this step.              │
+│          Step B — Call export_narrative(run_type)               │
+│          Report: narrative preview + download confirmation.     │
+│          Inform researcher of all downloadable files:           │
+│          comparison.csv · taxonomy_map.json ·                  │
+│          narrative.txt · {run_type}_summaries.json             │
+│ Marker:  [PHASE:6] ✅ COMPLETE                                  │
+└─────────────────────────────────────────────────────────────────┘
+
+═══════════════════════════════════════
+ERROR HANDLING
+═══════════════════════════════════════
+• If a tool returns an error message, report it clearly to the researcher.
+• Suggest the most likely fix (e.g., "MISTRAL_API_KEY may not be set").
+• Ask researcher to confirm before retrying.
+• Never silently skip a tool call that returned an error.
+
+═══════════════════════════════════════
+CONVERSATION STYLE
+═══════════════════════════════════════
+• Be concise but informative. No filler phrases.
+• Use emoji sparingly for phase headers: ✅ ⛔ 📊 🔍 ★
+• Always end with the current [PHASE:X] marker.
+• When displaying theme lists, use numbered format (not bullets).
+• When citing evidence, truncate to 150 characters + "…"
+"""
+
+# ─────────────────────────────────────────────────────────────────
+# AGENT STATE  (global — single-user HuggingFace Spaces deployment)
+# ─────────────────────────────────────────────────────────────────
+
+_SESSION_ID   = str(uuid.uuid4())
+_AGENT        = None          # LangGraph agent executor
+_MEMORY       = None          # MemorySaver instance
+_AGENT_STATE  = {
+    "phase":        0,
+    "run_type":     None,
+    "review_table": None,
+    "history":      [],
+}
+
+
+def _build_agent():
+    """Initialise the LangGraph ReAct agent with Mistral + MemorySaver."""
+    global _AGENT, _MEMORY, _SESSION_ID
+
+    llm = ChatMistralAI(
+        model="mistral-small-latest",
+        api_key=os.environ.get("MISTRAL_API_KEY", ""),
+        temperature=0.1,
+    )
+
+    _MEMORY  = MemorySaver()
+    _AGENT   = create_react_agent(
+        model=llm,
+        tools=ALL_TOOLS,
+        checkpointer=_MEMORY,
+        state_modifier=SYSTEM_PROMPT,
+    )
+    _SESSION_ID = str(uuid.uuid4())
+
+
+def _parse_phase(text: str) -> int:
+    """Extract [PHASE:X] marker from agent reply. Returns 0 if not found."""
+    match = re.search(r"\[PHASE:([\d.]+)\]", text)
+    return int(float(match.group(1))) if match else _AGENT_STATE["phase"]
+
+
+def _parse_review_table(run_type: str) -> list | None:
+    """
+    Load review table rows from labeled_topics.json checkpoint.
+    Returns None if the checkpoint doesn't exist yet.
+    """
+    path = f"{run_type}_labeled_topics.json"
+    return (
+        _load_json(path).get("review_rows", None)
+        if os.path.exists(path)
+        else None
+    )
+
+
+def _extract_run_type(message: str) -> str | None:
+    """Detect run_type from user message."""
+    msg_lower = message.lower()
+    return (
+        "abstract" if "abstract" in msg_lower else
+        "title"    if "title"    in msg_lower else
+        None
+    )
+
+
+def _extract_approved_json(message: str) -> str:
+    """
+    Pull the review JSON payload out of a [RESEARCHER REVIEW SUBMITTED] message.
+    Falls back to an empty approved_topics list if not parseable.
+    """
+    json_match = re.search(r"Review data:\s*(\{.*\})", message, re.DOTALL)
+    return json_match.group(1) if json_match else json.dumps({"approved_topics": []})
+
+
+# ─────────────────────────────────────────────────────────────────
+# PUBLIC API  (called by app.py)
+# ─────────────────────────────────────────────────────────────────
+
+def run_agent(user_message: str) -> dict:
+    """
+    Send user_message to the LangGraph agent and return:
+    {
+        "reply":        str,   # agent's response text
+        "phase":        int,   # current B&C phase (0-6)
+        "review_table": list | None  # rows for gr.Dataframe, or None
+    }
+    """
+    global _AGENT, _AGENT_STATE
+
+    # Lazy init
+    _build_agent() if _AGENT is None else None
+
+    # Detect run_type from message (update state)
+    detected_rt = _extract_run_type(user_message)
+    _AGENT_STATE["run_type"] = detected_rt or _AGENT_STATE["run_type"]
+    run_type = _AGENT_STATE["run_type"] or "abstract"
+
+    config = {"configurable": {"thread_id": _SESSION_ID}}
+
+    # Invoke LangGraph agent
+    result = _AGENT.invoke(
+        {"messages": [HumanMessage(content=user_message)]},
+        config=config,
+    )
+
+    # Extract last AI message
+    ai_messages = list(filter(
+        lambda m: m.__class__.__name__ in ("AIMessage", "AIMessageChunk"),
+        result.get("messages", []),
+    ))
+    reply = ai_messages[-1].content if ai_messages else "⚠️ Agent returned no response."
+
+    # Parse phase from reply
+    new_phase = _parse_phase(reply)
+    _AGENT_STATE["phase"] = new_phase
+
+    # Load review table if Phase 2 completed
+    review_table = (
+        _parse_review_table(run_type)
+        if "[REVIEW_TABLE_READY:" in reply or new_phase == 2
+        else _AGENT_STATE.get("review_table")
+    )
+    _AGENT_STATE["review_table"] = review_table
+
+    return {
+        "reply":        reply,
+        "phase":        new_phase,
+        "review_table": review_table,
+    }
+
+
+def get_agent_state() -> dict:
+    """Return the current agent state (phase, run_type, etc.)."""
+    return dict(_AGENT_STATE)
+
+
+def reset_agent() -> None:
+    """Reset agent, memory, and session state for a new analysis."""
+    global _AGENT, _MEMORY, _SESSION_ID, _AGENT_STATE
+    _AGENT       = None
+    _MEMORY      = None
+    _SESSION_ID  = str(uuid.uuid4())
+    _AGENT_STATE = {
+        "phase":        0,
+        "run_type":     None,
+        "review_table": None,
+        "history":      [],
+    }

app.py

@@ -0,0 +1,563 @@
+"""
+app.py — Gradio UI for BERTopic Thematic Analysis Agent
+Braun & Clarke (2006) Six-Phase Thematic Analysis
+Agentic AI Application with Mistral LLM + LangGraph
+"""
+
+import gradio as gr
+import pandas as pd
+import json
+import os
+from agent import run_agent, get_agent_state, reset_agent
+
+
+# ─────────────────────────────────────────────────────────────
+# PHASE PROGRESS BAR
+# ─────────────────────────────────────────────────────────────
+
+PHASES = [
+    "Phase 1: Familiarisation",
+    "Phase 2: Initial Codes",
+    "Phase 3: Themes",
+    "Phase 4: Saturation",
+    "Phase 5: Names",
+    "Phase 5.5: PAJAIS Mapping",
+    "Phase 6: Report",
+]
+
+PHASE_DESCRIPTIONS = {
+    0: "Upload your Scopus CSV to begin.",
+    1: "Familiarisation — Loading corpus, computing statistics.",
+    2: "Initial Codes — BERTopic clustering + LLM labelling. Review table below.",
+    3: "Themes — Consolidating approved topic groups into themes.",
+    4: "Saturation — Checking coverage and theme completeness.",
+    5: "Names — Finalising theme names and definitions.",
+    6: "PAJAIS Mapping — Mapping themes to taxonomy (MAPPED vs NOVEL).",
+    7: "Report — Generating comparison CSV + 500-word Section 7 narrative.",
+}
+
+
+def build_progress_bar(current_phase: int) -> str:
+    """Render an HTML phase progress bar."""
+    total = len(PHASES)
+    pct = int((current_phase / total) * 100)
+    steps_html = ""
+    for i, label in enumerate(PHASES, start=1):
+        if i < current_phase:
+            cls = "step done"
+            icon = "✔"
+        elif i == current_phase:
+            cls = "step active"
+            icon = str(i)
+        else:
+            cls = "step"
+            icon = str(i)
+        steps_html += f"""
+        <div class="{cls}">
+          <div class="circle">{icon}</div>
+          <div class="label">{label}</div>
+        </div>"""
+
+    description = PHASE_DESCRIPTIONS.get(current_phase, "")
+    return f"""
+    <style>
+      .progress-wrap {{
+        font-family: 'Segoe UI', sans-serif;
+        padding: 16px 8px 8px 8px;
+        background: #f8f9fa;
+        border-radius: 12px;
+        border: 1px solid #dee2e6;
+        margin-bottom: 8px;
+      }}
+      .steps-row {{
+        display: flex;
+        align-items: flex-start;
+        justify-content: space-between;
+        position: relative;
+      }}
+      .steps-row::before {{
+        content: '';
+        position: absolute;
+        top: 18px;
+        left: 5%;
+        right: 5%;
+        height: 3px;
+        background: #dee2e6;
+        z-index: 0;
+      }}
+      .step {{
+        display: flex;
+        flex-direction: column;
+        align-items: center;
+        width: {100 // total}%;
+        z-index: 1;
+      }}
+      .circle {{
+        width: 36px;
+        height: 36px;
+        border-radius: 50%;
+        background: #dee2e6;
+        color: #6c757d;
+        display: flex;
+        align-items: center;
+        justify-content: center;
+        font-weight: bold;
+        font-size: 13px;
+        border: 2px solid #adb5bd;
+      }}
+      .step.done .circle {{
+        background: #198754;
+        color: white;
+        border-color: #198754;
+      }}
+      .step.active .circle {{
+        background: #0d6efd;
+        color: white;
+        border-color: #0d6efd;
+        box-shadow: 0 0 0 4px rgba(13,110,253,0.2);
+      }}
+      .label {{
+        font-size: 10px;
+        color: #495057;
+        margin-top: 6px;
+        text-align: center;
+        line-height: 1.3;
+      }}
+      .step.active .label {{ color: #0d6efd; font-weight: 600; }}
+      .step.done .label {{ color: #198754; }}
+      .phase-desc {{
+        margin-top: 10px;
+        font-size: 13px;
+        color: #495057;
+        text-align: center;
+        padding: 6px 12px;
+        background: #e9f3ff;
+        border-radius: 8px;
+        border-left: 4px solid #0d6efd;
+      }}
+      .progress-bar-outer {{
+        height: 6px;
+        background: #dee2e6;
+        border-radius: 4px;
+        margin: 10px 0 4px 0;
+      }}
+      .progress-bar-inner {{
+        height: 6px;
+        width: {pct}%;
+        background: linear-gradient(90deg, #0d6efd, #198754);
+        border-radius: 4px;
+        transition: width 0.5s ease;
+      }}
+    </style>
+    <div class="progress-wrap">
+      <div class="steps-row">{steps_html}</div>
+      <div class="progress-bar-outer">
+        <div class="progress-bar-inner"></div>
+      </div>
+      <div class="phase-desc">📌 {description}</div>
+    </div>
+    """
+
+
+# ─────────────────────────────────────────────────────────────
+# EMPTY REVIEW TABLE
+# ────────��────────────────────────────────────────────────────
+
+REVIEW_COLUMNS = [
+    "#",
+    "Topic Label",
+    "Top Evidence",
+    "Sentences",
+    "Papers",
+    "Approve",
+    "Rename To",
+    "Reasoning",
+]
+
+
+def empty_review_table() -> pd.DataFrame:
+    return pd.DataFrame(columns=REVIEW_COLUMNS)
+
+
+def parse_review_table_to_json(df: pd.DataFrame) -> str:
+    """Convert review table edits into a JSON string for the agent."""
+    records = df.to_dict(orient="records")
+    approved = [r for r in records if str(r.get("Approve", "")).strip().lower() in ("yes", "y", "true", "1")]
+    return json.dumps({"approved_topics": approved}, indent=2)
+
+
+# ─────────────────────────────────────────────────────────────
+# CHART REGISTRY  (populated by agent via session state)
+# ─────────────────────────────────────────────────────────────
+
+CHART_OPTIONS = {
+    "Topic Size Distribution": "topic_size.html",
+    "Top 20 Topics Bar Chart": "top20_topics.html",
+    "Cluster Heatmap": "cluster_heatmap.html",
+    "Theme Coverage Treemap": "theme_treemap.html",
+}
+
+
+def load_chart_html(chart_name: str) -> str:
+    """Load a Plotly chart HTML file and wrap it in an iframe."""
+    fname = CHART_OPTIONS.get(chart_name, "")
+    if fname and os.path.exists(fname):
+        with open(fname, "r", encoding="utf-8") as f:
+            content = f.read()
+        # Wrap in srcdoc iframe so Gradio renders it safely
+        escaped = content.replace('"', "&quot;")
+        return f'<iframe srcdoc="{escaped}" width="100%" height="520px" style="border:none;border-radius:8px;"></iframe>'
+    return (
+        "<div style='padding:40px;text-align:center;color:#6c757d;font-size:14px;'>"
+        "📊 Chart will appear here after BERTopic analysis completes (Phase 2).</div>"
+    )
+
+
+# ─────────────────────────────────────────────────────────────
+# AGENT INTERACTION
+# ─────────────────────────────────────────────────────────────
+
+def handle_csv_upload(file_obj):
+    """Called when user uploads a CSV. Returns status message."""
+    if file_obj is None:
+        return "⚠️ No file uploaded.", build_progress_bar(0)
+    try:
+        df = pd.read_csv(file_obj.name)
+        rows, cols = df.shape
+        col_list = ", ".join(df.columns.tolist())
+        msg = (
+            f"✅ **CSV loaded successfully!**\n\n"
+            f"- **Rows:** {rows:,}\n"
+            f"- **Columns:** {cols}\n"
+            f"- **Fields:** {col_list}\n\n"
+            f"Type **`run abstract`** in the chat to begin Phase 1 (Familiarisation)."
+        )
+        # Store path in environment so agent can access it
+        os.environ["SCOPUS_CSV_PATH"] = file_obj.name
+        return msg, build_progress_bar(1)
+    except Exception as exc:
+        return f"❌ Error reading CSV: {exc}", build_progress_bar(0)
+
+
+def chat_with_agent(user_message: str, history: list, current_phase: int):
+    """
+    Send user message to the LangGraph agent and return:
+    - updated chat history
+    - updated review table DataFrame
+    - updated phase progress HTML
+    - updated current_phase int
+    """
+    if not user_message.strip():
+        return history, empty_review_table(), build_progress_bar(current_phase), current_phase
+
+    # Add user turn to history
+    history = history + [{"role": "user", "content": user_message}]
+
+    # Call LangGraph agent
+    response = run_agent(user_message)
+    agent_reply = response.get("reply", "⚠️ No response from agent.")
+    new_phase = response.get("phase", current_phase)
+    review_data = response.get("review_table", None)
+
+    history = history + [{"role": "assistant", "content": agent_reply}]
+
+    # Build review table if agent returned topic data
+    if review_data and isinstance(review_data, list) and len(review_data) > 0:
+        df = pd.DataFrame(review_data, columns=REVIEW_COLUMNS)
+    else:
+        df = empty_review_table()
+
+    return history, df, build_progress_bar(new_phase), new_phase
+
+
+def submit_review(review_df: pd.DataFrame, history: list, current_phase: int):
+    """
+    Called when researcher clicks 'Submit Review'.
+    Serialises the table edits and sends them to the agent as a structured message.
+    """
+    review_json = parse_review_table_to_json(review_df)
+    n_approved = len(json.loads(review_json).get("approved_topics", []))
+
+    system_message = (
+        f"[RESEARCHER REVIEW SUBMITTED]\n"
+        f"Approved topics: {n_approved}\n"
+        f"Review data:\n{review_json}\n\n"
+        f"Please proceed to the next phase."
+    )
+
+    response = run_agent(system_message)
+    agent_reply = response.get("reply", "⚠️ No response from agent.")
+    new_phase = response.get("phase", current_phase)
+    new_review_data = response.get("review_table", None)
+
+    history = history + [
+        {"role": "user", "content": f"✅ Review submitted — {n_approved} topics approved."},
+        {"role": "assistant", "content": agent_reply},
+    ]
+
+    if new_review_data and isinstance(new_review_data, list):
+        df = pd.DataFrame(new_review_data, columns=REVIEW_COLUMNS)
+    else:
+        df = review_df  # keep existing table if no new data
+
+    return history, df, build_progress_bar(new_phase), new_phase
+
+
+def refresh_downloads():
+    """Return list of output files for download."""
+    files = []
+    for fname in ["comparison.csv", "taxonomy_map.json", "narrative.txt", "summaries.json"]:
+        if os.path.exists(fname):
+            files.append(fname)
+    return files if files else None
+
+
+def reset_session():
+    """Reset everything for a new analysis run."""
+    reset_agent()
+    os.environ.pop("SCOPUS_CSV_PATH", None)
+    return (
+        [],               # clear chatbot
+        empty_review_table(),
+        build_progress_bar(0),
+        0,
+        "🔄 Session reset. Upload a new CSV to begin.",
+    )
+
+
+# ─────────────────────────────────────────────────────────────
+# GRADIO UI
+# ─────────────────────────────────────────────────────────────
+
+with gr.Blocks(
+    title="BERTopic Thematic Analysis Agent",
+    theme=gr.themes.Soft(
+        primary_hue="blue",
+        secondary_hue="green",
+        font=[gr.themes.GoogleFont("Inter"), "sans-serif"],
+    ),
+    css="""
+    #main-header { text-align: center; padding: 20px 0 8px 0; }
+    #main-header h1 { font-size: 1.8em; color: #1a1a2e; margin-bottom: 4px; }
+    #main-header p  { color: #6c757d; font-size: 14px; }
+    .section-title { font-weight: 700; font-size: 15px; color: #1a1a2e; margin-bottom: 4px; }
+    .submit-btn { background: #0d6efd !important; color: white !important; }
+    .reset-btn  { background: #dc3545 !important; color: white !important; }
+    .send-btn   { background: #198754 !important; color: white !important; }
+    """,
+    ssr_mode=False,
+) as demo:
+
+    # ── State ──────────────────────────────────────────────────
+    current_phase_state = gr.State(value=0)
+
+    # ── Header ─────────────────────────────────────────────────
+    with gr.Row(elem_id="main-header"):
+        gr.HTML("""
+        <div id="main-header">
+          <h1>🧠 BERTopic Thematic Analysis Agent</h1>
+          <p>Braun &amp; Clarke (2006) Six-Phase Thematic Analysis &nbsp;·&nbsp;
+             Mistral LLM &nbsp;·&nbsp; LangGraph &nbsp;·&nbsp; Sentence-BERT</p>
+        </div>
+        """)
+
+    # ── Phase Progress Bar ─────────────────────────────────────
+    phase_bar = gr.HTML(value=build_progress_bar(0), label="Phase Progress")
+
+    # ══════════════════════════════════════════════════════════
+    # SECTION 1 — DATA INPUT
+    # ══════════════════════════════════════════════════════════
+    with gr.Accordion("📂  Section 1 — Data Input", open=True):
+        gr.Markdown("""
+        **Upload your Scopus export CSV.**  
+        Required columns: `Authors`, `Title`, `Abstract`, `Author Keywords`, `Cited by`, `Source title`, `Year`
+        """)
+        with gr.Row():
+            csv_upload = gr.File(
+                label="📄 Upload Scopus CSV",
+                file_types=[".csv"],
+                scale=3,
+            )
+            upload_status = gr.Markdown(
+                value="⬆️ Upload your CSV file to begin.",
+                label="Upload Status",
+            )
+        reset_btn = gr.Button("🔄 Reset Session", variant="stop", elem_classes=["reset-btn"], scale=1)
+
+    # ══════════════════════════════════════════════════════════
+    # SECTION 2 — AGENT CONVERSATION
+    # ══════════════════════════════════════════════════════════
+    with gr.Accordion("💬  Section 2 — Agent Conversation", open=True):
+        gr.Markdown("""
+        **Chat with the BERTopic Agent.**  
+        Commands: `run abstract` · `run title` · `proceed` · `show themes` · `generate report`
+        """)
+        chatbot = gr.Chatbot(
+            label="🤖 Mistral Agent",
+            height=480,
+            show_copy_button=True,
+            bubble_full_width=False,
+            type="messages",
+            avatar_images=(None, "https://huggingface.co/front/assets/huggingface_logo-noborder.svg"),
+        )
+        with gr.Row():
+            user_input = gr.Textbox(
+                label="Your message",
+                placeholder='Type "run abstract" to start Phase 1...',
+                lines=2,
+                scale=5,
+                show_label=False,
+            )
+            send_btn = gr.Button("Send ➤", variant="primary", scale=1, elem_classes=["send-btn"])
+
+    # ══════════════════════════════════════════════════════════
+    # SECTION 3 — RESULTS
+    # ══════════════════════════════════════════════════════════
+    with gr.Accordion("📊  Section 3 — Results", open=True):
+        with gr.Tabs():
+
+            # ── TAB A: Review Table ────────────────────────────
+            with gr.Tab("🔍 Review Table"):
+                gr.Markdown("""
+                **Researcher Review Table** — Phase 2 Output.  
+                - Set **Approve** = `yes` to include a topic.  
+                - Fill **Rename To** if you want a better label.  
+                - Add **Reasoning** to document your decision.  
+                - Click **Submit Review** when done.
+                """)
+
+                review_table = gr.Dataframe(
+                    value=empty_review_table(),
+                    headers=REVIEW_COLUMNS,
+                    datatype=[
+                        "number",   # #
+                        "str",      # Topic Label
+                        "str",      # Top Evidence
+                        "number",   # Sentences
+                        "number",   # Papers
+                        "str",      # Approve  (yes/no)
+                        "str",      # Rename To
+                        "str",      # Reasoning
+                    ],
+                    col_count=(8, "fixed"),
+                    row_count=(10, "dynamic"),
+                    interactive=True,
+                    wrap=True,
+                    label="Topic Review Table (edit Approve / Rename To / Reasoning)",
+                    max_height=520,
+                )
+
+                with gr.Row():
+                    submit_review_btn = gr.Button(
+                        "✅ Submit Review to Agent",
+                        variant="primary",
+                        elem_classes=["submit-btn"],
+                    )
+                    gr.Markdown(
+                        "_After editing the table, click Submit Review to continue to Phase 3._",
+                        container=False,
+                    )
+
+            # ── TAB B: Charts ──────────────────────────────────
+            with gr.Tab("📈 Charts"):
+                gr.Markdown("**Interactive Plotly Charts** — generated after Phase 2 (BERTopic clustering).")
+                chart_dropdown = gr.Dropdown(
+                    choices=list(CHART_OPTIONS.keys()),
+                    value=list(CHART_OPTIONS.keys())[0],
+                    label="Select Chart",
+                    interactive=True,
+                )
+                chart_display = gr.HTML(
+                    value=load_chart_html(list(CHART_OPTIONS.keys())[0]),
+                    label="Chart",
+                )
+
+            # ── TAB C: Downloads ───────────────────────────────
+            with gr.Tab("⬇️ Downloads"):
+                gr.Markdown("""
+                **Download output files** — available after Phase 6 (Report generation).
+
+                | File | Description |
+                |------|-------------|
+                | `comparison.csv` | Abstract vs Title theme comparison |
+                | `taxonomy_map.json` | PAJAIS gap analysis (MAPPED vs NOVEL) |
+                | `narrative.txt` | 500-word Section 7 draft |
+                | `summaries.json` | BERTopic cluster summaries |
+                """)
+                refresh_btn = gr.Button("🔄 Refresh Downloads", variant="secondary")
+                download_files = gr.File(
+                    label="Output Files",
+                    file_count="multiple",
+                    interactive=False,
+                )
+
+    # ───────────────────────────────���──────────────────────────
+    # EVENT WIRING
+    # ──────────────────────────────────────────────────────────
+
+    # CSV Upload
+    csv_upload.change(
+        fn=handle_csv_upload,
+        inputs=[csv_upload],
+        outputs=[upload_status, phase_bar],
+    )
+
+    # Send message (button)
+    send_btn.click(
+        fn=chat_with_agent,
+        inputs=[user_input, chatbot, current_phase_state],
+        outputs=[chatbot, review_table, phase_bar, current_phase_state],
+    ).then(
+        fn=lambda: "",
+        outputs=[user_input],
+    )
+
+    # Send message (Enter key)
+    user_input.submit(
+        fn=chat_with_agent,
+        inputs=[user_input, chatbot, current_phase_state],
+        outputs=[chatbot, review_table, phase_bar, current_phase_state],
+    ).then(
+        fn=lambda: "",
+        outputs=[user_input],
+    )
+
+    # Submit Review
+    submit_review_btn.click(
+        fn=submit_review,
+        inputs=[review_table, chatbot, current_phase_state],
+        outputs=[chatbot, review_table, phase_bar, current_phase_state],
+    )
+
+    # Chart dropdown change
+    chart_dropdown.change(
+        fn=load_chart_html,
+        inputs=[chart_dropdown],
+        outputs=[chart_display],
+    )
+
+    # Refresh downloads
+    refresh_btn.click(
+        fn=refresh_downloads,
+        outputs=[download_files],
+    )
+
+    # Reset session
+    reset_btn.click(
+        fn=reset_session,
+        outputs=[chatbot, review_table, phase_bar, current_phase_state, upload_status],
+    )
+
+
+# ─────────────────────────────────────────────────────────────
+# LAUNCH
+# ─────────────────────────────────────────────────────────────
+
+if __name__ == "__main__":
+    demo.launch(
+        server_name="0.0.0.0",
+        server_port=7860,
+        share=False,
+        show_error=True,
+        ssr_mode=False,
+    )

requirements.txt

@@ -0,0 +1,17 @@
+# requirements.txt — BERTopic Thematic Analysis Agent
+# HuggingFace Spaces: Gradio SDK
+# All 13 packages required for full pipeline
+
+gradio==4.44.1
+langchain-core==0.3.15
+langchain-mistralai==0.2.4
+langchain==0.3.7
+langgraph==0.2.45
+sentence-transformers==3.3.1
+scikit-learn==1.5.2
+bertopic==0.16.4
+plotly==5.24.1
+numpy==1.26.4
+pandas==2.2.3
+huggingface-hub==0.26.2
+mistralai==1.2.5

tools.py

@@ -0,0 +1,783 @@
+"""
+tools.py — 7 @tool functions for BERTopic Thematic Analysis Agent
+Agentic Coding Rules (strictly followed):
+  ✗ ZERO if/elif/else statements  → decisions belong to the LLM
+  ✗ ZERO for/while loops          → use list(map(...)) / functional patterns
+  ✗ ZERO try/except blocks        → handle_tool_error=True lets LLM see errors
+  ✗ ZERO retry logic              → LLM adapts its own strategy on failure
+  ✓ ALL tools are stateless       → input in, output out, nothing more
+  ✓ Checkpoints saved as JSON/NPY → reproducibility guaranteed
+"""
+
+import os
+import re
+import json
+import numpy as np
+import pandas as pd
+import plotly.express as px
+import plotly.graph_objects as go
+from sklearn.cluster import AgglomerativeClustering
+from sklearn.metrics.pairwise import cosine_similarity as sk_cosine
+from sentence_transformers import SentenceTransformer
+from langchain_core.tools import tool
+from langchain_core.prompts import PromptTemplate
+from langchain_core.output_parsers import JsonOutputParser, StrOutputParser
+from langchain_mistralai import ChatMistralAI
+
+# ─────────────────────────────────────────────────────────────────
+# SHARED CONSTANTS
+# ─────────────────────────────────────────────────────────────────
+
+RUN_CONFIGS = {
+    "abstract": "Abstract",
+    "title":    "Title",
+}
+
+BOILERPLATE_RE = re.compile(
+    r"(©\s*\d{4}|all rights reserved|published by elsevier|"
+    r"springer nature|wiley online|emerald publishing|"
+    r"^\s*doi\s*:|this is an open.access|"
+    r"creative commons|permission of the copyright)",
+    re.IGNORECASE,
+)
+
+MIN_WORDS       = 5       # discard sentences shorter than this
+CLUSTER_THRESH  = 0.7     # AgglomerativeClustering cosine distance threshold
+TOP_N_EVIDENCE  = 5       # sentences nearest centroid per cluster
+TOP_N_TOPICS    = 100     # topics sent to LLM for labelling
+
+PAJAIS_25 = [
+    "AI in Accounting",
+    "AI in Finance",
+    "Machine Learning Applications",
+    "Natural Language Processing",
+    "Decision Support Systems",
+    "Blockchain and Distributed Ledger",
+    "Robotic Process Automation",
+    "Data Analytics and Big Data",
+    "Cybersecurity and Fraud Detection",
+    "Cloud Computing",
+    "Internet of Things",
+    "Digital Transformation",
+    "Ethics and Governance of AI",
+    "Explainable AI",
+    "Predictive Analytics",
+    "Neural Networks and Deep Learning",
+    "Text Mining and Sentiment Analysis",
+    "Knowledge Management Systems",
+    "Human-Computer Interaction",
+    "Enterprise Resource Planning",
+    "Audit and Assurance Technology",
+    "Tax Technology",
+    "Regulatory Technology (RegTech)",
+    "Financial Reporting Technology",
+    "Corporate Governance and Technology",
+]
+
+SENTENCE_MODEL = "all-MiniLM-L6-v2"   # 384-dim local model, ~200MB RAM
+MISTRAL_MODEL  = "mistral-small-latest"
+
+# ─────────────────────────────────────────────────────────────────
+# HELPER FUNCTIONS  (pure, no side-effects)
+# ─────────────────────────────────────────────────────────────────
+
+def _get_llm() -> ChatMistralAI:
+    return ChatMistralAI(
+        model=MISTRAL_MODEL,
+        api_key=os.environ.get("MISTRAL_API_KEY", ""),
+        temperature=0.1,
+    )
+
+
+def _split_sentences(text: str) -> list:
+    """Split text into sentences using regex (no loops)."""
+    raw = re.split(r"(?<=[.!?])\s+", str(text).strip())
+    return list(filter(
+        lambda s: len(s.split()) >= MIN_WORDS and not BOILERPLATE_RE.search(s),
+        raw,
+    ))
+
+
+def _file_path(run_type: str, suffix: str) -> str:
+    """Standardised checkpoint path: e.g. abstract_summaries.json"""
+    return f"{run_type}_{suffix}"
+
+
+def _save_json(data, path: str) -> None:
+    with open(path, "w", encoding="utf-8") as f:
+        json.dump(data, f, indent=2, ensure_ascii=False)
+
+
+def _load_json(path: str) -> dict:
+    with open(path, "r", encoding="utf-8") as f:
+        return json.load(f)
+
+
+def _save_chart(fig, path: str) -> None:
+    fig.write_html(path, include_plotlyjs="cdn", full_html=True)
+
+
+def _nearest_to_centroid(embeddings: np.ndarray, indices: np.ndarray, k: int) -> np.ndarray:
+    """Return indices of k sentences nearest to cluster centroid (no loops)."""
+    cluster_embs = embeddings[indices]
+    centroid     = cluster_embs.mean(axis=0, keepdims=True)
+    sims         = sk_cosine(centroid, cluster_embs)[0]
+    top_k        = np.argsort(sims)[-k:][::-1]
+    return indices[top_k]
+
+
+# ─────────────────────────────────────────────────────────────────
+# TOOL 1 — load_scopus_csv
+# ─────────────────────────────────────────────────────────────────
+
+@tool(handle_tool_error=True)
+def load_scopus_csv(run_type: str) -> str:
+    """
+    Load the Scopus CSV from SCOPUS_CSV_PATH environment variable.
+    run_type: 'abstract' or 'title'.
+    Returns corpus statistics including paper count, sentence count,
+    year range, and top journals.
+    Saves corpus sentences to {run_type}_sentences.json checkpoint.
+    """
+    csv_path  = os.environ.get("SCOPUS_CSV_PATH", "")
+    df        = pd.read_csv(csv_path)
+    col_name  = RUN_CONFIGS.get(run_type, "Abstract")
+
+    # Extract text column; fill missing with empty string
+    texts     = df[col_name].fillna("").astype(str).tolist()
+
+    # Split all texts into sentences (functional map)
+    all_sentences = list(map(_split_sentences, texts))
+
+    # Build flat list of (paper_idx, sentence) pairs
+    indexed_sentences = list(map(
+        lambda pair: {"paper_idx": pair[0], "sentence": sent},
+        [(i, sent) for i, sents in enumerate(all_sentences) for sent in sents],
+    ))
+    # Rebuild properly using enumerate + flatmap pattern
+    flat = [
+        {"paper_idx": i, "sentence": s}
+        for i, sents in enumerate(all_sentences)
+        for s in sents
+    ]
+
+    # Year statistics (pandas — no explicit loops)
+    year_col    = df["Year"].dropna().astype(int)
+    year_min    = int(year_col.min())
+    year_max    = int(year_col.max())
+    year_counts = year_col.value_counts().sort_index().to_dict()
+
+    # Top 5 journals
+    top_journals = (
+        df["Source title"].value_counts().head(5).to_dict()
+        if "Source title" in df.columns else {}
+    )
+
+    # Save checkpoint
+    checkpoint = {
+        "run_type":       run_type,
+        "column":         col_name,
+        "total_papers":   len(df),
+        "total_sentences": len(flat),
+        "year_min":       year_min,
+        "year_max":       year_max,
+        "year_counts":    year_counts,
+        "top_journals":   top_journals,
+        "sentences":      flat,
+    }
+    _save_json(checkpoint, _file_path(run_type, "sentences.json"))
+
+    stats = (
+        f"✅ Corpus loaded for run_type='{run_type}' (column: {col_name})\n"
+        f"  Papers     : {len(df):,}\n"
+        f"  Sentences  : {len(flat):,}  (after boilerplate filter)\n"
+        f"  Year range : {year_min} – {year_max}\n"
+        f"  Top journals: {list(top_journals.keys())[:3]}\n"
+        f"  Checkpoint : {_file_path(run_type, 'sentences.json')}\n\n"
+        f"Phase 1 (Familiarisation) complete. "
+        f"[PHASE:1] ⛔ STOP — Awaiting researcher acknowledgement before Phase 2."
+    )
+    return stats
+
+
+# ─────────────────────────────────────────────────────────────────
+# TOOL 2 — run_bertopic_discovery
+# ─────────────────────────────────────────────────────────────────
+
+@tool(handle_tool_error=True)
+def run_bertopic_discovery(run_type: str) -> str:
+    """
+    Core BERTopic pipeline:
+      1. Load sentences from checkpoint.
+      2. Embed with all-MiniLM-L6-v2 (384-dim, normalised).
+      3. Cluster with AgglomerativeClustering(metric='cosine', linkage='average', threshold=0.7).
+         NO UMAP — direct cosine clustering for transparency.
+      4. Find 5 evidence sentences nearest to each cluster centroid.
+      5. Generate 4 Plotly charts (size distribution, top-20 bar, heatmap, treemap).
+      6. Save summaries.json + emb.npy checkpoints.
+    run_type: 'abstract' or 'title'.
+    Returns cluster summary for LLM review.
+    """
+    data      = _load_json(_file_path(run_type, "sentences.json"))
+    sentences = list(map(lambda r: r["sentence"], data["sentences"]))
+    paper_ids = list(map(lambda r: r["paper_idx"], data["sentences"]))
+
+    # Step 1 — Embed
+    model      = SentenceTransformer(SENTENCE_MODEL)
+    embeddings = model.encode(
+        sentences,
+        normalize_embeddings=True,
+        show_progress_bar=False,
+        batch_size=64,
+    )
+    np.save(_file_path(run_type, "emb.npy"), embeddings)
+
+    # Step 2 — Cluster (cosine distance = 1 - cosine_similarity for normalised vectors)
+    clustering = AgglomerativeClustering(
+        n_clusters=None,
+        metric="cosine",
+        linkage="average",
+        distance_threshold=CLUSTER_THRESH,
+    )
+    labels = clustering.fit_predict(embeddings)
+
+    unique_labels = list(dict.fromkeys(labels.tolist()))  # preserve order, deduplicate
+    n_clusters    = len(unique_labels)
+
+    # Step 3 — Build cluster summaries (map over unique labels)
+    def build_cluster_summary(label_id):
+        mask      = np.where(labels == label_id)[0]
+        ev_idx    = _nearest_to_centroid(embeddings, mask, TOP_N_EVIDENCE)
+        evidence  = list(map(lambda i: sentences[i], ev_idx.tolist()))
+        pids      = list(set(map(lambda i: paper_ids[i], mask.tolist())))
+        return {
+            "cluster_id":    int(label_id),
+            "n_sentences":   int(len(mask)),
+            "n_papers":      int(len(pids)),
+            "evidence":      evidence,
+            "paper_indices": pids,
+        }
+
+    summaries = list(map(build_cluster_summary, unique_labels))
+
+    # Sort by sentence count descending
+    summaries.sort(key=lambda s: s["n_sentences"], reverse=True)
+
+    # Assign rank
+    ranked = list(map(
+        lambda pair: {**pair[1], "rank": pair[0] + 1},
+        enumerate(summaries),
+    ))
+
+    checkpoint = {
+        "run_type":   run_type,
+        "n_clusters": n_clusters,
+        "summaries":  ranked,
+    }
+    _save_json(checkpoint, _file_path(run_type, "summaries.json"))
+
+    # Step 4 — Generate 4 Plotly charts
+
+    # Chart 1: Topic Size Distribution (histogram)
+    sizes  = list(map(lambda s: s["n_sentences"], ranked))
+    fig1   = px.histogram(
+        x=sizes, nbins=40,
+        title=f"Topic Size Distribution ({run_type.title()})",
+        labels={"x": "Sentences per Topic", "y": "Number of Topics"},
+        color_discrete_sequence=["#0d6efd"],
+    )
+    fig1.update_layout(bargap=0.1, template="plotly_white")
+    _save_chart(fig1, f"{run_type}_topic_size.html")
+
+    # Chart 2: Top-20 Topics Bar Chart (placeholder labels = cluster IDs)
+    top20  = ranked[:20]
+    fig2   = px.bar(
+        x=list(map(lambda s: f"Topic {s['cluster_id']}", top20)),
+        y=list(map(lambda s: s["n_sentences"],           top20)),
+        title=f"Top 20 Topics by Sentence Count ({run_type.title()})",
+        labels={"x": "Topic", "y": "Sentence Count"},
+        color=list(map(lambda s: s["n_sentences"], top20)),
+        color_continuous_scale="Blues",
+    )
+    fig2.update_layout(showlegend=False, template="plotly_white", xaxis_tickangle=-45)
+    _save_chart(fig2, f"{run_type}_top20_topics.html")
+
+    # Chart 3: Papers per Topic (top 30 — scatter)
+    top30  = ranked[:30]
+    fig3   = px.scatter(
+        x=list(map(lambda s: s["n_sentences"], top30)),
+        y=list(map(lambda s: s["n_papers"],    top30)),
+        text=list(map(lambda s: f"T{s['cluster_id']}", top30)),
+        title=f"Sentences vs Papers per Topic ({run_type.title()})",
+        labels={"x": "Sentences", "y": "Papers"},
+        color=list(map(lambda s: s["n_papers"], top30)),
+        color_continuous_scale="Viridis",
+    )
+    fig3.update_traces(textposition="top center", marker_size=10)
+    fig3.update_layout(template="plotly_white")
+    _save_chart(fig3, f"{run_type}_cluster_heatmap.html")
+
+    # Chart 4: Coverage Treemap (top 40)
+    top40  = ranked[:40]
+    fig4   = px.treemap(
+        names=list(map(lambda s: f"Cluster {s['cluster_id']}", top40)),
+        values=list(map(lambda s: s["n_sentences"], top40)),
+        parents=["Topics"] * len(top40),
+        title=f"Topic Coverage Treemap ({run_type.title()})",
+        color=list(map(lambda s: s["n_papers"], top40)),
+        color_continuous_scale="RdYlGn",
+    )
+    fig4.update_layout(template="plotly_white")
+    _save_chart(fig4, f"{run_type}_theme_treemap.html")
+
+    result = (
+        f"✅ BERTopic discovery complete for run_type='{run_type}'\n"
+        f"  Sentences embedded : {len(sentences):,}  (384-dim, all-MiniLM-L6-v2)\n"
+        f"  Clusters found     : {n_clusters}\n"
+        f"  Threshold used     : cosine distance = {CLUSTER_THRESH}\n"
+        f"  Charts saved       : 4 Plotly HTML files\n"
+        f"  Checkpoints        : {run_type}_summaries.json, {run_type}_emb.npy\n\n"
+        f"Top 5 clusters by sentence count:\n" +
+        "\n".join(list(map(
+            lambda s: f"  #{s['rank']} Cluster {s['cluster_id']}: "
+                      f"{s['n_sentences']} sentences / {s['n_papers']} papers\n"
+                      f"     Evidence: {s['evidence'][0][:120]}…",
+            ranked[:5],
+        ))) +
+        f"\n\n[PHASE:2] ⛔ STOP — Run label_topics_with_llm next, "
+        f"then present the review table and await researcher Submit Review."
+    )
+    return result
+
+
+# ─────────────────────────────────────────────────────────────────
+# TOOL 3 — label_topics_with_llm
+# ─────────────────────────────────────────────────────────────────
+
+@tool(handle_tool_error=True)
+def label_topics_with_llm(run_type: str) -> str:
+    """
+    Send the top 100 clusters to the Mistral LLM for labelling.
+    Uses PromptTemplate + JsonOutputParser (LangChain LCEL chain).
+    Each cluster receives: research_area (name) and reasoning.
+    Saves labeled_topics.json checkpoint.
+    Returns the review table as a JSON string for UI rendering.
+    run_type: 'abstract' or 'title'.
+    """
+    data     = _load_json(_file_path(run_type, "summaries.json"))
+    top100   = data["summaries"][:TOP_N_TOPICS]
+
+    llm      = _get_llm()
+
+    label_prompt = PromptTemplate.from_template("""
+You are a computational thematic analysis expert applying Braun & Clarke (2006).
+Below are {n_topics} research topic clusters extracted from academic abstracts.
+Each cluster contains evidence sentences from related papers.
+
+For EACH cluster, provide:
+1. A concise research area label (3-7 words, title case)
+2. Brief reasoning (1 sentence) explaining what connects these sentences
+
+Respond ONLY with a valid JSON array. No preamble, no markdown fences.
+Format: [{{"cluster_id": 0, "label": "...", "reasoning": "..."}}]
+
+CLUSTERS:
+{clusters_json}
+""")
+
+    # Format clusters for the prompt
+    clusters_for_prompt = list(map(
+        lambda s: {
+            "cluster_id": s["cluster_id"],
+            "rank":       s["rank"],
+            "n_sentences": s["n_sentences"],
+            "n_papers":   s["n_papers"],
+            "evidence":   s["evidence"][:3],  # first 3 evidence sentences
+        },
+        top100,
+    ))
+
+    chain  = label_prompt | llm | JsonOutputParser()
+    result = chain.invoke({
+        "n_topics":     len(top100),
+        "clusters_json": json.dumps(clusters_for_prompt, indent=2),
+    })
+
+    # Merge labels back into summaries
+    label_map = {item["cluster_id"]: item for item in result}
+
+    labeled = list(map(
+        lambda s: {
+            **s,
+            "label":     label_map.get(s["cluster_id"], {}).get("label",     f"Topic {s['cluster_id']}"),
+            "reasoning": label_map.get(s["cluster_id"], {}).get("reasoning", ""),
+        },
+        top100,
+    ))
+
+    # Build review table rows for UI
+    review_rows = list(map(
+        lambda s: [
+            s["rank"],
+            s["label"],
+            s["evidence"][0][:200] if s["evidence"] else "",
+            s["n_sentences"],
+            s["n_papers"],
+            "yes",          # default approve = yes (researcher edits)
+            "",             # rename to
+            s["reasoning"],
+        ],
+        labeled,
+    ))
+
+    checkpoint = {
+        "run_type":      run_type,
+        "labeled_topics": labeled,
+        "review_rows":   review_rows,
+    }
+    _save_json(checkpoint, _file_path(run_type, "labeled_topics.json"))
+
+    summary = (
+        f"✅ {len(labeled)} topics labelled by Mistral LLM\n\n"
+        f"Sample labels:\n" +
+        "\n".join(list(map(
+            lambda s: f"  #{s['rank']:>3}. {s['label']}  ({s['n_sentences']} sentences)",
+            labeled[:10],
+        ))) +
+        f"\n  ...\n\n"
+        f"Review table populated with {len(review_rows)} rows.\n"
+        f"[REVIEW_TABLE_READY:{run_type}]\n\n"
+        f"[PHASE:2] ⛔ STOP — Review table is now visible. "
+        f"Researcher must edit Approve / Rename To / Reasoning columns, "
+        f"then click 'Submit Review' to proceed to Phase 3."
+    )
+    return summary
+
+
+# ─────────────────────────────────────────────────────────────────
+# TOOL 4 — consolidate_into_themes
+# ─────────────────────────────────────────────────────────────────
+
+@tool(handle_tool_error=True)
+def consolidate_into_themes(approved_json: str, run_type: str) -> str:
+    """
+    Merge approved and renamed topics into higher-level themes.
+    approved_json: JSON string with list of approved topic dicts
+                   (from researcher review table submission).
+    run_type: 'abstract' or 'title'.
+    Recomputes theme centroids from member cluster embeddings.
+    Saves themes.json checkpoint.
+    Returns theme summary.
+    """
+    approved_data = json.loads(approved_json)
+    approved      = approved_data.get("approved_topics", approved_data)
+
+    labeled_data  = _load_json(_file_path(run_type, "labeled_topics.json"))
+    label_index   = {t["cluster_id"]: t for t in labeled_data["labeled_topics"]}
+
+    # Apply renames: use 'Rename To' if non-empty, else keep original label
+    def apply_rename(row):
+        cid   = int(row.get("#", row.get("cluster_id", 0)))
+        orig  = label_index.get(cid, {})
+        rename = str(row.get("Rename To", "")).strip()
+        final_label = rename or row.get("Topic Label", orig.get("label", f"Theme {cid}"))
+        return {
+            "cluster_id":   cid,
+            "theme_label":  final_label,
+            "n_sentences":  orig.get("n_sentences", 0),
+            "n_papers":     orig.get("n_papers", 0),
+            "evidence":     orig.get("evidence", []),
+            "reasoning":    str(row.get("Reasoning", "")),
+        }
+
+    themes = list(map(apply_rename, approved))
+
+    # Sort themes by sentence count descending
+    themes.sort(key=lambda t: t["n_sentences"], reverse=True)
+    themes = list(map(
+        lambda pair: {**pair[1], "theme_rank": pair[0] + 1},
+        enumerate(themes),
+    ))
+
+    checkpoint = {
+        "run_type": run_type,
+        "n_themes": len(themes),
+        "themes":   themes,
+    }
+    _save_json(checkpoint, _file_path(run_type, "themes.json"))
+
+    coverage_sentences = sum(map(lambda t: t["n_sentences"], themes))
+    coverage_papers    = len(set(
+        pid
+        for t in themes
+        for pid in label_index.get(t["cluster_id"], {}).get("paper_indices", [])
+    ))
+
+    result = (
+        f"✅ Phase 3 — Themes consolidated for run_type='{run_type}'\n"
+        f"  Approved topics → {len(themes)} themes\n"
+        f"  Coverage         : {coverage_sentences:,} sentences\n"
+        f"  Unique papers    : {coverage_papers:,}\n\n"
+        f"Consolidated themes:\n" +
+        "\n".join(list(map(
+            lambda t: f"  #{t['theme_rank']:>2}. {t['theme_label']}  "
+                      f"({t['n_sentences']} sent / {t['n_papers']} papers)",
+            themes[:15],
+        ))) +
+        f"\n\n[PHASE:3] ⛔ STOP — Themes consolidated. "
+        f"Researcher must confirm coverage is satisfactory before Phase 4 (Saturation)."
+    )
+    return result
+
+
+# ─────────────────────────────────────────────────────────────────
+# TOOL 5 — compare_with_taxonomy
+# ─────────────────────────────────────────────────────────────────
+
+@tool(handle_tool_error=True)
+def compare_with_taxonomy(run_type: str) -> str:
+    """
+    Map consolidated themes to the PAJAIS 25-category taxonomy using Mistral LLM.
+    Each theme is classified as MAPPED (matches a PAJAIS category) or NOVEL (new contribution).
+    Saves taxonomy_map.json checkpoint.
+    run_type: 'abstract' or 'title'.
+    Returns mapping summary with MAPPED vs NOVEL breakdown.
+    """
+    themes_data = _load_json(_file_path(run_type, "themes.json"))
+    themes      = themes_data["themes"]
+    llm         = _get_llm()
+
+    taxonomy_prompt = PromptTemplate.from_template("""
+You are a PAJAIS taxonomy expert. The PAJAIS (Pacific-Asia Journal of the AIS) taxonomy
+has exactly 25 research categories listed below.
+
+PAJAIS 25 CATEGORIES:
+{pajais_categories}
+
+I will give you {n_themes} research themes extracted from academic literature.
+For each theme:
+- If it clearly matches a PAJAIS category, set status="MAPPED" and pajais_match=<exact category name>
+- If it represents research NOT covered by any PAJAIS category, set status="NOVEL" and pajais_match=null
+- Provide a 1-sentence justification
+
+Respond ONLY with a valid JSON array. No preamble, no markdown.
+Format: [{{"theme_label": "...", "status": "MAPPED"|"NOVEL", "pajais_match": "...|null", "justification": "..."}}]
+
+THEMES TO MAP:
+{themes_json}
+""")
+
+    chain  = taxonomy_prompt | llm | JsonOutputParser()
+    result = chain.invoke({
+        "pajais_categories": "\n".join(list(map(
+            lambda pair: f"{pair[0]+1:>2}. {pair[1]}",
+            enumerate(PAJAIS_25),
+        ))),
+        "n_themes":   len(themes),
+        "themes_json": json.dumps(list(map(
+            lambda t: {"theme_label": t["theme_label"], "evidence_sample": t["evidence"][0][:150]},
+            themes,
+        )), indent=2),
+    })
+
+    # Merge mapping back into themes
+    result_map = {r["theme_label"]: r for r in result}
+
+    mapped_themes = list(map(
+        lambda t: {
+            **t,
+            "status":        result_map.get(t["theme_label"], {}).get("status",        "UNKNOWN"),
+            "pajais_match":  result_map.get(t["theme_label"], {}).get("pajais_match",  None),
+            "justification": result_map.get(t["theme_label"], {}).get("justification", ""),
+        },
+        themes,
+    ))
+
+    checkpoint = {
+        "run_type":     run_type,
+        "taxonomy":     "PAJAIS-25",
+        "mapped_themes": mapped_themes,
+        "pajais_25":    PAJAIS_25,
+    }
+    _save_json(checkpoint, "taxonomy_map.json")
+
+    n_mapped = len(list(filter(lambda t: t["status"] == "MAPPED", mapped_themes)))
+    n_novel  = len(list(filter(lambda t: t["status"] == "NOVEL",  mapped_themes)))
+
+    novel_labels = list(map(
+        lambda t: f"  ★ {t['theme_label']}  — {t['justification'][:100]}",
+        filter(lambda t: t["status"] == "NOVEL", mapped_themes),
+    ))
+
+    result_str = (
+        f"✅ Phase 5.5 — PAJAIS Taxonomy Mapping complete for run_type='{run_type}'\n"
+        f"  Total themes : {len(mapped_themes)}\n"
+        f"  MAPPED       : {n_mapped}  (match existing PAJAIS categories)\n"
+        f"  NOVEL        : {n_novel}   (publishable research gaps)\n\n"
+        f"NOVEL themes (research gaps / publication opportunities):\n" +
+        "\n".join(novel_labels or ["  None — all themes are covered by PAJAIS taxonomy."]) +
+        f"\n\nCheckpoint saved: taxonomy_map.json\n\n"
+        f"[PHASE:5.5] ⛔ STOP — Taxonomy mapping complete. "
+        f"Researcher must review MAPPED vs NOVEL classification before Phase 6 (Report)."
+    )
+    return result_str
+
+
+# ─────────────────────────────────────────────────────────────────
+# TOOL 6 — generate_comparison_csv
+# ─────────────────────────────────────────────────────────────────
+
+@tool(handle_tool_error=True)
+def generate_comparison_csv(dummy: str = "run") -> str:
+    """
+    Generate a side-by-side comparison of abstract themes vs title themes.
+    Reads abstract_themes.json and title_themes.json checkpoints.
+    Identifies: convergent themes (appear in both), abstract-only, title-only.
+    Saves comparison.csv.
+    dummy: pass any string to invoke (no required arguments).
+    Returns summary of convergence/divergence findings.
+    """
+    abs_data   = _load_json("abstract_themes.json")
+    title_data = _load_json("title_themes.json")
+
+    abs_labels   = set(map(lambda t: t["theme_label"].lower(), abs_data["themes"]))
+    title_labels = set(map(lambda t: t["theme_label"].lower(), title_data["themes"]))
+
+    convergent    = abs_labels & title_labels
+    abs_only      = abs_labels - title_labels
+    title_only    = title_labels - abs_labels
+
+    # Build aligned rows
+    all_abs_themes   = {t["theme_label"].lower(): t for t in abs_data["themes"]}
+    all_title_themes = {t["theme_label"].lower(): t for t in title_data["themes"]}
+
+    all_labels = sorted(abs_labels | title_labels)
+
+    def build_row(label):
+        abs_t   = all_abs_themes.get(label, {})
+        title_t = all_title_themes.get(label, {})
+        presence = (
+            "CONVERGENT"  if label in convergent  else
+            "ABSTRACT ONLY" if label in abs_only   else
+            "TITLE ONLY"
+        )
+        return {
+            "Theme Label":            label.title(),
+            "Presence":               presence,
+            "Abstract Sentences":     abs_t.get("n_sentences", 0),
+            "Abstract Papers":        abs_t.get("n_papers",    0),
+            "Title Sentences":        title_t.get("n_sentences", 0),
+            "Title Papers":           title_t.get("n_papers",    0),
+            "Abstract Evidence":      abs_t.get("evidence",   [""])[0][:200],
+            "Title Evidence":         title_t.get("evidence", [""])[0][:200],
+        }
+
+    rows = list(map(build_row, all_labels))
+    df   = pd.DataFrame(rows)
+    df.to_csv("comparison.csv", index=False, encoding="utf-8")
+
+    result = (
+        f"✅ Comparison CSV generated: comparison.csv\n"
+        f"  Total unique themes : {len(all_labels)}\n"
+        f"  Convergent          : {len(convergent)}  (appear in both abstract & title runs)\n"
+        f"  Abstract-only       : {len(abs_only)}   (richer detail in full text)\n"
+        f"  Title-only          : {len(title_only)}  (surface-level signals)\n\n"
+        f"Convergent themes (high confidence):\n" +
+        "\n".join(list(map(lambda l: f"  ✓ {l.title()}", sorted(convergent)[:10]))) +
+        f"\n\ncomparison.csv ready for download."
+    )
+    return result
+
+
+# ─────────────────────────────────────────────────────────────────
+# TOOL 7 — export_narrative
+# ─────────────────────────────────────────────────────────────────
+
+@tool(handle_tool_error=True)
+def export_narrative(run_type: str) -> str:
+    """
+    Generate a 500-word Section 7 narrative report using Mistral LLM.
+    Synthesises themes, PAJAIS mapping, and convergence/divergence findings.
+    Saves narrative.txt.
+    run_type: 'abstract' or 'title' (use 'abstract' for the primary narrative).
+    Returns the generated narrative text.
+    """
+    themes_data   = _load_json(_file_path(run_type, "themes.json"))
+    taxonomy_data = _load_json("taxonomy_map.json")
+    themes        = themes_data["themes"]
+    mapped        = list(filter(lambda t: t["status"] == "MAPPED", taxonomy_data["mapped_themes"]))
+    novel         = list(filter(lambda t: t["status"] == "NOVEL",  taxonomy_data["mapped_themes"]))
+
+    llm = _get_llm()
+
+    narrative_prompt = PromptTemplate.from_template("""
+You are an academic writer producing Section 7 of a conference paper submitted to PAJAIS
+(Pacific-Asia Journal of the AIS). Write approximately 500 words.
+
+Section 7 should:
+1. Introduce the BERTopic thematic analysis methodology (Braun & Clarke 2006, six phases)
+2. Summarise the {n_themes} themes discovered from {run_type} analysis
+3. Discuss MAPPED themes (alignment with PAJAIS taxonomy)
+4. Highlight NOVEL themes as research gaps and publication opportunities
+5. Conclude with implications for IS research and future work
+
+Write in formal academic style. Do not use bullet points — use flowing paragraphs.
+
+THEMES DISCOVERED ({n_themes} total):
+{themes_list}
+
+MAPPED TO PAJAIS ({n_mapped} themes):
+{mapped_list}
+
+NOVEL / RESEARCH GAPS ({n_novel} themes):
+{novel_list}
+""")
+
+    chain     = narrative_prompt | llm | StrOutputParser()
+    narrative = chain.invoke({
+        "n_themes":   len(themes),
+        "run_type":   run_type,
+        "themes_list": "\n".join(list(map(
+            lambda t: f"  {t['theme_rank']}. {t['theme_label']} ({t['n_sentences']} sentences, {t['n_papers']} papers)",
+            themes[:20],
+        ))),
+        "n_mapped":   len(mapped),
+        "n_novel":    len(novel),
+        "mapped_list": "\n".join(list(map(
+            lambda t: f"  • {t['theme_label']} → {t['pajais_match']}",
+            mapped[:10],
+        ))),
+        "novel_list": "\n".join(list(map(
+            lambda t: f"  ★ {t['theme_label']}: {t['justification']}",
+            novel,
+        ))),
+    })
+
+    with open("narrative.txt", "w", encoding="utf-8") as f:
+        f.write(narrative)
+
+    result = (
+        f"✅ Section 7 narrative exported: narrative.txt\n"
+        f"  Word count : ~{len(narrative.split()):,}\n\n"
+        f"──── NARRATIVE PREVIEW (first 600 chars) ────\n"
+        f"{narrative[:600]}…\n"
+        f"──────────────────────────────────────────────\n\n"
+        f"[PHASE:6] ✅ All phases complete!\n"
+        f"Download: comparison.csv · taxonomy_map.json · narrative.txt · summaries.json"
+    )
+    return result
+
+
+# ─────────────────────────────────────────────────────────────────
+# TOOL REGISTRY  (imported by agent.py)
+# ─────────────────────────────────────────────────────────────────
+
+ALL_TOOLS = [
+    load_scopus_csv,
+    run_bertopic_discovery,
+    label_topics_with_llm,
+    consolidate_into_themes,
+    compare_with_taxonomy,
+    generate_comparison_csv,
+    export_narrative,
+]