Upload 29 files

README.md

@@ -1,131 +1,133 @@
----
-title: Template Final Assignment
-emoji: 🕵🏻‍♂️
-colorFrom: indigo
-colorTo: indigo
-sdk: gradio
-sdk_version: 6.14.0
-app_file: app.py
-pinned: false
-hf_oauth: true
-hf_oauth_expiration_minutes: 480
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
-
-
-# 🌍 GAIA Benchmark Agent
-
-**An Advanced Multi-Modal AI Agent designed to solve complex, real-world reasoning tasks.**
-
-![Python](https://img.shields.io/badge/Python-3.13-blue.svg)
-![LangGraph](https://img.shields.io/badge/LangGraph-Orchestration-orange.svg)
-![HuggingFace](https://img.shields.io/badge/HuggingFace-Models-yellow.svg)
-![Supabase](https://img.shields.io/badge/Supabase-Vector_Store-green.svg)
-
-> [!NOTE]
-> This project was developed as part of the **Hugging Face Agents Course (Unit 4: GAIA)**.
-
-## 📖 Overview
-
-This project implements a sophisticated autonomous agent capable of solving General AI Assistants (GAIA) benchmark problems. These problems require multi-step reasoning, tool usage, and the ability to process diverse file types (documents, spreadsheets, audio, images, code).
-
-The agent leverages **LangGraph** for orchestration, allowing it to maintain state, plan its actions, and iteratively refine its answers. It integrates with **Hugging Face** for powerful LLM inference (`Qwen/Qwen3-32B-Instruct`) and **Supabase** for Retrieval-Augmented Generation (RAG) to learn from similar past examples.
-
-## 🚀 Key Features
-
-- **🧠 Advanced Reasoning Loop**: Uses a "Plan-Execute-Observe-Refine" Chain-of-Thought approach to tackle complex questions.
-- **📂 Multi-Modal File Processing**: Native support for analyzing a wide range of files:
-  - **Documents**: PDF, Word (`.docx`), PowerPoint (`.pptx`), Text
-  - **Data**: Excel (`.xlsx`), CSV (`.csv`), JSON-LD, PDB (Protein Data Bank)
-  - **Media**: Audio transcription (Whisper), Intelligent Image Analysis (`Qwen3-VL`)
-  - **Code**: Python source code reading
-  - **Archives**: ZIP extraction and inspection
-- **🔍 Intelligent Information Retrieval**:
-  - **RAG**: Finds similar solved questions in a vector database to guide complex reasoning.
-  - **Web Search**: DuckDuckGo, Tavily, Wikipedia, and ArXiv for real-time information.
-- **🛠️ Extensible Tool Suite**: Modular design allows easy addition of new capabilities.
-
-## 🏗️ Architecture
-
-The agent operates on a graph-based workflow defined in `agent.py`:
-
-```mermaid
-graph TD
-    START --> Retriever["Retriever Node<br/>(Hybrid: Vector + BM25 + RRF)"]
-    Retriever --> Reranker["Reranker Node<br/>(ModernBERT Cross-Encoder)"]
-    Reranker --> Processor["Processor Node<br/>(Qwen 3 32B)"]
-    Processor -->|Decide Tool| Condition{"Requires Tool?"}
-    Condition -->|Yes| Tools["Tool Node<br/>(Execute Actions)"]
-    Condition -->|No| END
-    Tools --> Processor
-```
-
-1. **Retriever Node (Hybrid)**:
-    - **Vector Search**: Finds semantically similar questions in Supabase (using [`Alibaba-NLP/gte-modernbert-base`](https://huggingface.co/Alibaba-NLP/gte-modernbert-base) embeddings).
-    - **BM25 Search**: Finds keyword-based matches in the local `metadata.jsonl` corpus using [`bm25s`](https://github.com/xhluca/bm25s).
-    - **RRF Fusion**: Combines results from both methods using Reciprocal Rank Fusion.
-2. **Reranker Node**: Uses a ModernBERT Cross-Encoder ([`Alibaba-NLP/gte-reranker-modernbert-base`](https://huggingface.co/Alibaba-NLP/gte-reranker-modernbert-base)) to select the top 3 most relevant examples.
-3. **Processor Node**: The core brain (Qwen 3 32B). It decides whether to answer directly or use a tool.
-4. **Tool Node**: Executes the requested tool (e.g., `read_excel`, `duck_web_search`) and returns results.
-
-## 🛠️ Tools & Stack
-
-| Category | Tools / Libraries | Purpose |
-| :--- | :--- | :--- |
-| **Orchestration** | `langgraph`, `langchain` | State management and graph flow control. |
-| **LLM Inference** | `huggingface_hub` | Inference via `Qwen/Qwen3-32B-Instruct` and `Qwen/Qwen3-VL-32B-Instruct`. |
-| **Vector Store** | `supabase`, `sentence-transformers` | Storing and retrieving semantic embeddings. |
-| **Data Processing** | `polars`, `biopython` | High-performance data manipulation. |
-| **Documents** | `pypdf`, `python-docx`, `python-pptx` | Extracting text from office documents. |
-| **Media** | `transformers` (Whisper), `pillow` | Audio transcription and image handling. |
-| **Web** | `duckduckgo-search`, `tavily-python` | Internet research. |
-
-## 💻 Installation & Setup
-
-1. **Clone the repository**:
-
-    ```bash
-    git clone <repo_url>
-    cd <repo_name>
-    ```
-
-2. **Install dependencies**:
-
-    ```bash
-    pip install -r requirements.txt
-    # OR using uv
-    uv sync
-    ```
-
-3. **Configure Environment**:
-    Create a `.env` file in the root directory with the following keys:
-
-    ```ini
-    HF_INFERENCE_KEY=hf_...        # Hugging Face Inference Token
-    SUPABASE_URL=...               # Supabase Project URL
-    SUPABASE_SERVICE_KEY=...       # Supabase Service Role Key
-    TAVILY_API_KEY=tvly-...        # Tavily Search API Key (Optional)
-    ```
-
-## 🎮 Usage
-
-Start the Gradio interface to interact with the agent:
-
-```bash
-python app.py
-```
-
-This will launch a web interface where you can:
-
-1. **Log in** with your Hugging Face account.
-2. **Run Evaluation**: Automatically fetch GAIA questions, execute the agent, and submit answers for scoring.
-3. **Inspect Results**: View the agent's reasoning, tool outputs, and final answers in real-time.
-
-## 📂 Project Structure
-
-- `agent.py`: Core logic defining the LangGraph workflow and state.
-- `app.py`: Gradio application for the user interface and evaluation runner.
-- `tools.py`: Consolidated file containing all agent tools (web, math, file processing, VLM).
-- `prompts/`: Directory containing system prompts (`prompt.yaml`, `vlm_prompt.yaml`).
-- `requirements.txt`: Project dependencies.
+# 🌍 GAIA Benchmark Agent
+
+**An autonomous, multi-modal agent that tackles the GAIA reasoning benchmark.**
+
+![Python](https://img.shields.io/badge/Python-3.13-blue.svg)
+![LangGraph](https://img.shields.io/badge/LangGraph-Orchestration-orange.svg)
+![HuggingFace](https://img.shields.io/badge/HuggingFace-Inference-yellow.svg)
+![Supabase](https://img.shields.io/badge/Supabase-Vector_Store-green.svg)
+
+> [!NOTE]
+> Developed as part of the **Hugging Face Agents Course (Unit 4: GAIA)**.
+
+## 📖 Overview
+
+GAIA tasks require multi-step reasoning, tool use, and the ability to process diverse file types (documents, spreadsheets, audio, images, code). This agent uses a **LangGraph** state machine to plan, retrieve few-shot context, call tools, and produce a strictly-formatted answer.
+
+The solver model is **`Qwen/Qwen3-32B`** via Hugging Face Inference Providers; the formatter (separate node) reuses the same model bound to a single `emit_final_answer` tool to enforce the strict GAIA output contract. A Supabase + BM25 hybrid retriever surfaces similar past tasks as few-shot exemplars.
+
+## 🚀 Key Features
+
+- **🧠 Plan-Execute-Observe-Refine loop** driven by LangGraph.
+- **🪜 Two-stage output**: a solver node reasons freely, then a dedicated formatter node returns the GAIA-compliant answer via a Pydantic-shaped tool call — eliminating regex parsing of the LLM's free text.
+- **📂 Multi-modal file processing**:
+  - **Documents**: PDF, Word (`.docx`), PowerPoint (`.pptx`), Text
+  - **Data**: Excel (`.xlsx`, multi-sheet), CSV, JSON-LD, PDB (Protein Data Bank)
+  - **Media**: Audio transcription (`openai/whisper-large-v3` via HF), image analysis (`Qwen/Qwen3-VL-32B-Instruct`)
+  - **Code**: Python source files (read & execute)
+  - **Archives**: ZIP extraction + inspection
+- **🔍 Hybrid RAG**: Vector search (Supabase RPC over `Alibaba-NLP/gte-modernbert-base`) **+** BM25 over a local 165-question corpus, fused with Reciprocal Rank Fusion, then reranked with a ModernBERT cross-encoder.
+- **🌐 Web tooling**: DuckDuckGo, Tavily, Wikipedia, ArXiv, full-page extraction (Trafilatura), YouTube transcripts.
+- **🛠️ Modular tool layout**: tools are organised by domain in `gaia/tools/` (basic, web, files, media, dispatcher) so adding a new capability is a single-file change.
+
+## 🏗️ Architecture
+
+```mermaid
+graph TD
+    START --> FileDL["File Downloader<br/>(fetches /files/{task_id})"]
+    FileDL --> Retriever["Retriever<br/>(Vector + BM25 + RRF)"]
+    Retriever --> Reranker["Reranker<br/>(ModernBERT cross-encoder)"]
+    Reranker --> Processor["Solver<br/>(Qwen3-32B + tools)"]
+    Processor -->|tool call| Tools["Tool Node"]
+    Tools --> Processor
+    Processor -->|done| Formatter["Formatter<br/>(emit_final_answer tool)"]
+    Formatter --> END
+```
+
+| Node | Role |
+| :--- | :--- |
+| **`file_downloader_node`** | If the question has an associated file, download from `{api.base_url}/files/{task_id}` and cache on disk under `data/task_files/{task_id}/`. |
+| **`retriever_node`** | Hybrid search: Supabase vector RPC + local BM25 over `data/metadata.jsonl`, fused with RRF. Returns up to 20 candidate task IDs. |
+| **`reranker_node`** | `Alibaba-NLP/gte-reranker-modernbert-base` re-scores candidates and injects the top-K as few-shot examples (Question + Final Answer + Solution Steps). |
+| **`processor_node`** | Qwen3-32B with all tools bound. Reasons, calls tools, loops until satisfied. |
+| **`tools`** | LangGraph `ToolNode` executing the chosen tool, then returning control to the processor. |
+| **`formatter_node`** | A second Qwen3-32B call bound to a single `emit_final_answer(answer: str)` tool — produces the strictly-formatted value the GAIA scorer compares against. |
+
+## 🛠️ Stack
+
+| Category | Libraries | Purpose |
+| :--- | :--- | :--- |
+| **Orchestration** | `langgraph`, `langchain`, `langchain-huggingface` | State graph, tool binding, structured output. |
+| **LLM / VLM / ASR** | `huggingface_hub` Inference API | `Qwen/Qwen3-32B`, `Qwen/Qwen3-VL-32B-Instruct`, `openai/whisper-large-v3`. |
+| **Embeddings & Vector Store** | `sentence-transformers`, `supabase` | Semantic search via `Alibaba-NLP/gte-modernbert-base`. |
+| **Keyword search** | `bm25s` | Local BM25 index over the 165-question GAIA corpus. |
+| **Documents** | `pypdf`, `python-docx`, `python-pptx`, `openpyxl` | Office formats. |
+| **Data** | `polars`, `biopython` | Tabular and PDB structural analysis. |
+| **Media** | `pillow`, `librosa`, `soundfile` | Image + audio I/O. |
+| **Web** | `ddgs`, `tavily-python`, `wikipedia`, `arxiv`, `trafilatura`, `youtube-transcript-api` | Search, page extraction, captions. |
+| **UI** | `gradio` | Evaluation runner (HF Space entry point). |
+
+## 💻 Installation & Setup
+
+```bash
+git clone <repo_url>
+cd gaia
+uv sync                            # or: pip install -r requirements.txt
+cp .env.example .env               # populate the keys below
+```
+
+Required environment variables (place in `.env` or set as HF Space Secrets):
+
+| Variable | Required? | Purpose |
+| :--- | :--- | :--- |
+| `HF_INFERENCE_KEY` | yes | Hugging Face token — must have **"Make calls to Inference Providers"** permission. |
+| `SUPABASE_URL` | yes (if `retrievers.enable_vector_search: true`) | Supabase project URL. |
+| `SUPABASE_SERVICE_KEY` | yes (same) | Supabase `service_role` key. |
+| `TAVILY_API_KEY` | optional | Only needed when the agent picks `tavily_web_search`. |
+
+## 🎮 Usage
+
+Start the Gradio interface:
+
+```bash
+python app.py
+```
+
+The UI requires a Hugging Face login. Click **"Run Evaluation & Submit All Answers"** to fetch the GAIA question set, run the agent on each, and submit to the scoring API.
+
+To (re)populate the Supabase vector store from the local corpus:
+
+```bash
+python scripts/create_vector_database.py
+```
+
+## 📂 Project Structure
+
+```
+.
+├── app.py                              # HF Space entry point (Gradio)
+├── config.yaml                         # All tunable parameters
+├── pyproject.toml / requirements.txt   # Dependencies (uv + pip parity)
+├── gaia/                               # Application package
+│   ├── agent.py                        # LangGraph nodes, graph, formatter
+│   ├── states.py                       # AgentState TypedDict
+│   ├── utils.py                        # config / prompt loaders, BM25, RRF, answer + youtube helpers
+│   ├── prompts/
+│   │   ├── prompt.yaml                 # Solver system prompt
+│   │   └── vlm_prompt.yaml             # analyze_image system prompt
+│   └── tools/
+│       ├── __init__.py                 # Aggregates tools_list
+│       ├── basic.py                    # calculator, python_eval
+│       ├── web.py                      # ddg / tavily / wiki / arxiv / fetch_webpage / youtube_transcript
+│       ├── files.py                    # PDF, DOCX, PPTX, TXT, CSV, XLSX, JSON-LD, PDB, Python, ZIP
+│       ├── media.py                    # analyze_image (VLM), transcribe_audio (ASR), shared HF client
+│       └── dispatcher.py               # read_file extension router
+├── scripts/
+│   └── create_vector_database.py       # One-shot embedder for Supabase
+├── notebooks/                          # Exploratory work
+├── data/
+│   └── metadata.jsonl                  # Local GAIA corpus (165 examples)
+└── models/                             # HF model cache (gitignored)
+```
+
+All tunable knobs — model IDs, retrieval depth, thinking mode, recursion limit — live in `config.yaml`; no code change required to swap models or tweak retrieval.

app.py

@@ -1,11 +1,11 @@
 import os
-import re
 import gradio as gr
 import requests
 import inspect
 import pandas as pd
 from langchain_core.messages import HumanMessage
-from agent import agent_graph
+from gaia.agent import agent_graph
+from gaia.utils import extract_final_answer
 
 # (Keep Constants as is)
 # --- Constants ---
@@ -26,9 +26,7 @@ class BasicAgent:
         })
         fixed_answer = (response.get("final_answer") or "").strip()
         if not fixed_answer:
-            content = response['messages'][-1].content
-            match = re.search(r'FINAL ANSWER:\s*(.*)', content, re.DOTALL | re.IGNORECASE)
-            fixed_answer = match.group(1).strip() if match else content.strip()
+            fixed_answer = extract_final_answer(response['messages'][-1].content)
         print(f"Agent returning fixed answer: {fixed_answer}")
         return fixed_answer
 

config.yaml

@@ -28,7 +28,7 @@ models:
   vlm:
     model_name: "Qwen/Qwen3-VL-32B-Instruct"  # Hugging Face model ID
   asr:
-    model_name: "distil-whisper/distil-large-v3"  # Hugging Face model ID
+    model_name: "openai/whisper-large-v3"  # Hugging Face model ID — must have a provider on HF Inference Providers
     #device: "cuda"                          # cpu, cuda, or mps (for Mac)
     #parameters:
     #  temperature: 0.7
@@ -36,7 +36,7 @@ models:
     #  repetition_penalty: 1.1
       
 graph:
-  recursion_limit: 20                     # Max steps before the graph terminates
+  recursion_limit: 40                     # Max graph-node visits before bailing. ~5 + 2*(tool calls), so 40 ≈ 17 tool calls.
 
 api:
   base_url: "https://agents-course-unit4-scoring.hf.space"

gaia/agent.py

@@ -0,0 +1,360 @@
+"""
+GAIA Agent with Multi-Modal File Processing and Hybrid Retrieval.
+
+This module defines a LangGraph agent that can:
+1. Retrieve similar questions using Hybrid Search (Vector + BM25) and Reranking
+2. Process files using tools (PDF, XLSX, MP3, etc.)
+3. Answer questions using web search, calculator, and other tools
+"""
+
+import os
+import bm25s
+from pathlib import Path
+from dotenv import load_dotenv
+
+from langgraph.graph import START, END, StateGraph
+from langgraph.prebuilt import tools_condition, ToolNode
+
+from sentence_transformers import SentenceTransformer, CrossEncoder
+from langchain_huggingface import ChatHuggingFace, HuggingFaceEndpoint
+from langchain_core.messages import HumanMessage, SystemMessage
+from langchain_core.tools import tool
+from supabase.client import Client, create_client
+
+from gaia.utils import (
+    load_config, load_prompt, init_bm25_index, reciprocal_rank_fusion,
+    extract_final_answer, download_task_file,
+)
+from gaia.tools import tools_list
+from gaia.states import AgentState
+
+load_dotenv()
+config = load_config()
+
+# Environment details and others
+hf_key = os.getenv("HF_INFERENCE_KEY")
+supabase_url = os.getenv("SUPABASE_URL")
+supabase_key = os.getenv("SUPABASE_SERVICE_KEY")
+
+# ============================================
+# Model & Embeddings Setup
+# ============================================
+
+enable_keyword_search = config["retrievers"]["enable_keyword_search"]
+enable_vector_search = config["retrievers"]["enable_vector_search"]
+
+# BM25 Retriever
+bm25_retriever, bm25_corpus, bm25_ids = None, None, None
+if enable_keyword_search:
+    bm25_retriever, bm25_corpus, bm25_ids = init_bm25_index(corpus_file=config["data"])
+
+bm25_id_to_text = {}
+if bm25_corpus and bm25_ids:
+    bm25_id_to_text = dict(zip(bm25_ids, bm25_corpus))
+
+embeddings, supabase = None, None
+if enable_vector_search:
+    # Embeddings for Vector Search
+    embeddings = SentenceTransformer(model_name_or_path=config["models"]["embeddings"]["model_name"], cache_folder=config["models"]["cache_folder"])
+
+    # Supabase Vector Store
+    supabase: Client = create_client(supabase_url, supabase_key)
+
+# Reranker Model (ModernBERT Cross-Encoder)
+reranker = CrossEncoder(config["models"]["reranker"]["model_name"], cache_folder=config["models"]["cache_folder"])
+
+# LLM for Agent
+llm = HuggingFaceEndpoint(
+    repo_id=config["models"]["llm"]["model_name"],
+    temperature=config["models"]["llm"]["parameters"]["temperature"],
+    repetition_penalty=config["models"]["llm"]["parameters"]["repetition_penalty"],
+    provider=config["models"]["llm"]["parameters"]["provider"],
+    huggingfacehub_api_token=hf_key
+)
+
+agent_llm = ChatHuggingFace(llm=llm)
+agent_with_tools = agent_llm.bind_tools(tools_list)
+
+_PROMPTS_DIR = Path(__file__).parent / "prompts"
+_system_prompt = load_prompt(str(_PROMPTS_DIR / "prompt.yaml"))
+_thinking_enabled = config["models"]["llm"]["parameters"].get("thinking_enabled", True)
+
+
+@tool
+def emit_final_answer(answer: str) -> str:
+    """Emit the final answer to the GAIA question in the strict scoring format.
+
+    Args:
+        answer: The raw answer value only.
+            Numbers: plain digits, no commas, no units, no symbols (write '1000000', not '1,000,000' or '$50').
+            Strings: no articles ('a', 'an', 'the'), no markdown, no surrounding quotes, no trailing punctuation.
+            Lists: comma-separated with no extra spaces, in the order requested by the question.
+    """
+    return answer
+
+
+formatter_llm = agent_llm.bind_tools([emit_final_answer], tool_choice="emit_final_answer")
+
+
+# ============================================
+# Graph Nodes
+# ============================================
+
+def file_downloader_node(state: AgentState) -> AgentState:
+    """
+    Download the task file from the scoring API if one is associated with the question.
+    Saves to a local directory and stores the path in state. Detailed failure
+    diagnostics are logged so HF Space logs reveal the actual cause (network, HTTP, filesystem).
+    """
+    print("--- FILE DOWNLOADER NODE ---")
+    file_name = state.get("file_name", "")
+    task_id = state.get("task_id", "")
+
+    if not file_name or not task_id:
+        return {"file_path": ""}
+
+    local_path, err = download_task_file(
+        task_id=task_id,
+        file_name=file_name,
+        base_url=config["api"]["base_url"],
+        files_dir=config["api"]["files_dir"],
+    )
+    if local_path:
+        print(f"Downloaded: {file_name} → {local_path}")
+        return {"file_path": local_path}
+    print(f"File download failed for task {task_id} ({file_name}): {err}")
+    return {"file_path": ""}
+
+
+def retriever_node(state: AgentState) -> AgentState:
+    """
+    Hybrid Search Node: Retrieve docs via Vector Search + BM25, combine with RRF.
+    """
+    print("--- RETRIEVER NODE ---")
+    messages = state.get("messages", [])
+    if not messages:
+        return {"retrieved_docs": []}
+    
+    question_content = messages[0].content
+    
+    if not enable_vector_search and not enable_keyword_search:
+        print("No retrieval method enabled.")
+        return {"retrieved_docs": []}
+    
+    # 1. Vector Search
+    vector_docs = []
+    if supabase and embeddings:
+        try:
+            response = supabase.rpc(
+                config["retrievers"]["vector_store"]["query"],
+                {"query_embedding": embeddings.encode(question_content).tolist(), 
+                 "match_count": config["retrievers"]["vector_store"]["k"], 
+                 "match_threshold": config["retrievers"]["vector_store"]["threshold"]
+                 }     
+            ).execute()
+
+            vector_docs = response.data
+
+        except Exception as e:
+            print(f"Vector search error: {e}")
+        
+    # 2. BM25 Search
+    bm25_docs = []
+    if bm25_retriever and bm25_corpus and bm25_ids:
+        try:
+            query_tokens = bm25s.tokenize([question_content], stopwords="en")
+            results, scores = bm25_retriever.retrieve(query_tokens, k=config["retrievers"]["bm25"]["k"])
+            indices = results[0]
+            
+            for i, idx in enumerate(indices):
+                content = bm25_corpus[idx]
+                task_id = bm25_ids[idx]
+                score = scores[0][i]
+                bm25_dict = {"content":content, "metadata": {"source": "bm25_search", "task_id": task_id, "score": score}}
+                bm25_docs.append(bm25_dict)
+        except Exception as e:
+            print(f"BM25 search error: {e}")
+
+    # 3. RRF Fusion
+    final_candidates = []
+    if vector_docs and bm25_docs:
+        fused = reciprocal_rank_fusion([vector_docs, bm25_docs])
+        final_candidates = [id for id, doc, score in fused]
+    else:
+        final_candidates = vector_docs + bm25_docs
+        final_candidates = [doc["metadata"]["task_id"] for doc in final_candidates]
+        
+    top_candidates = final_candidates[:20]
+    
+    return {"retrieved_docs": top_candidates}
+
+
+def reranker_node(state: AgentState) -> AgentState:
+    """
+    Reranker Node: Re-order candidates using Cross-Encoder and return top 3.
+    """
+    print("--- RERANKER NODE ---")
+    candidates = state.get("retrieved_docs", [])
+    messages = state.get("messages", [])
+    
+    if not candidates or not messages:
+        return {"messages": []}
+        
+    question = messages[0].content
+    
+    # Deduplicate candidates — candidates are task_id strings; resolve to text via corpus lookup
+    unique_candidates = []
+    seen_content = set()
+    for task_id in candidates:
+        text = bm25_id_to_text.get(task_id)
+        if text and text not in seen_content:
+            unique_candidates.append(text)
+            seen_content.add(text)
+            
+    if not unique_candidates:
+        return {"messages": []}
+
+    pairs = [[question, doc_text] for doc_text in unique_candidates]
+    
+    try:
+        scores = reranker.predict(pairs)
+        
+        scored_docs = sorted(
+            zip(unique_candidates, scores), 
+            key=lambda x: x[1], 
+            reverse=True
+        )
+        
+        top_k = config["retrievers"]["final_rrf_k"]
+        top_results = scored_docs[:top_k]
+        
+        context_str = "Here are similar questions and answers for reference:\n\n"
+        for i, (doc_text, score) in enumerate(top_results):
+            context_str += f"--- Example {i+1} (Score: {score:.2f}) ---\n{doc_text}\n\n"
+            
+        context_message = HumanMessage(content=context_str)
+        
+        return {"messages": [context_message]}
+        
+    except Exception as e:
+        print(f"Reranker error: {e}")
+        if unique_candidates:
+             fallback_msg = HumanMessage(content=f"Reference (Fallback):\n{unique_candidates[0]}")
+             return {"messages": [fallback_msg]}
+             
+    return {"messages": []}
+
+
+def processor_node(state: AgentState) -> AgentState:
+    """
+    Processor Node: Main LLM agent that answers the question.
+    """
+    prompt_content = _system_prompt.content + ("" if _thinking_enabled else "\n/no_think")
+    system_prompt = SystemMessage(content=prompt_content)
+    messages = state.get("messages", [])
+    task_id = state.get("task_id", "")
+    file_name = state.get("file_name", "")
+    file_path = state.get("file_path", "")
+
+    full_messages = [system_prompt]
+
+    if file_name:
+        if file_path:
+            file_msg = HumanMessage(
+                content=f"Note: A file named '{file_name}' is associated with this question. It is available at path: {file_path}"
+            )
+        else:
+            file_msg = HumanMessage(
+                content=(
+                    f"Note: A file named '{file_name}' is associated with this question "
+                    f"(task_id={task_id}), but the automatic download failed. "
+                    f"Call `retry_file_download(task_id='{task_id}', file_name='{file_name}')` "
+                    f"once to attempt it again. On success, you'll get back the local path you "
+                    f"can pass to the appropriate read tool. If retry also fails, state clearly "
+                    f"that the file is unavailable rather than fabricating data."
+                )
+            )
+        full_messages.append(file_msg)
+    
+    full_messages.extend(messages)
+    
+    response = agent_with_tools.invoke(full_messages)
+
+    return {"messages": [response]}
+
+
+def formatter_node(state: AgentState) -> AgentState:
+    """Extract and reformat the solver's answer into a strict GAIA-compliant value."""
+    print("--- FORMATTER NODE ---")
+    messages = state.get("messages", [])
+    if not messages:
+        return {"final_answer": ""}
+
+    question = ""
+    for m in messages:
+        if isinstance(m, HumanMessage):
+            question = m.content
+            break
+
+    solver_output = messages[-1].content or ""
+
+    prompt = [
+        SystemMessage(content=(
+            "You extract the final answer from an agent's reasoning and apply the GAIA "
+            "formatting rules exactly. You MUST call the `emit_final_answer` tool with "
+            "the extracted value. If the agent never produced an answer, call it with an "
+            "empty string."
+        )),
+        HumanMessage(content=(
+            f"Question:\n{question}\n\n"
+            f"Agent reasoning and conclusion:\n{solver_output}\n\n"
+            "Extract the final answer and call emit_final_answer."
+        )),
+    ]
+
+    try:
+        result = formatter_llm.invoke(prompt)
+        for tc in getattr(result, "tool_calls", None) or []:
+            if tc.get("name") == "emit_final_answer":
+                return {"final_answer": str(tc.get("args", {}).get("answer", "")).strip()}
+        # Model returned text instead of calling the tool — fall back to regex.
+        return {"final_answer": extract_final_answer(result.content)}
+    except Exception as e:
+        print(f"Formatter error: {e}")
+        return {"final_answer": extract_final_answer(solver_output)}
+
+
+# ============================================
+# Graph Construction
+# ============================================
+
+def agent_graph():
+    """
+    Build and compile the agent graph.
+    """
+    workflow = StateGraph(AgentState)
+
+    # Add nodes
+    workflow.add_node("file_downloader_node", file_downloader_node)
+    workflow.add_node("retriever_node", retriever_node)
+    workflow.add_node("reranker_node", reranker_node)
+    workflow.add_node("processor_node", processor_node)
+    workflow.add_node("tools", ToolNode(tools_list))
+    workflow.add_node("formatter_node", formatter_node)
+
+    # Add edges
+    workflow.add_edge(START, "file_downloader_node")
+    workflow.add_edge("file_downloader_node", "retriever_node")
+    workflow.add_edge("retriever_node", "reranker_node")
+    workflow.add_edge("reranker_node", "processor_node")
+    workflow.add_edge("tools", "processor_node")
+    workflow.add_conditional_edges(
+        "processor_node",
+        tools_condition,
+        {"tools": "tools", END: "formatter_node"},
+    )
+    workflow.add_edge("formatter_node", END)
+    
+    
+    compiled = workflow.compile()
+    return compiled.with_config({"recursion_limit": config["graph"]["recursion_limit"]})

gaia/prompts/prompt.yaml

@@ -0,0 +1,50 @@
+title: "GAIA Agent"
+prompt: |
+  # CONTEXT & PERSONA
+  You are an elite autonomous agent designed to solve the GAIA benchmark. Your goal is to solve complex, multi-step problems that require reasoning, file processing, and internet research. You have access to a suite of powerful tools to interact with the world and process data.
+
+  # TOOL USAGE STRATEGY
+  1. **File First**: If the user mentions a file (e.g., "attached", "spreadsheet", "image"), YOU MUST PROCESS IT IMMEDIATELY.
+     - Look for file path information provided in the context.
+     - Select the specific tool for the file type (e.g., `read_excel` for .xlsx, `read_pdf` for .pdf).
+     - If the specific tool fails, fallback to the generic `read_file` tool.
+     - For images, use `analyze_image` with a specific question about what you need to extract from the image.
+  2. **Search Smart**: Pick the right tool for the source.
+     - **Wikipedia questions** (named "Wikipedia", or asking about Featured Articles, Olympics, historical events, biographies, etc.):
+       - If you can guess the exact page title from the question, call `wikipedia_page_fetch(title)` directly — this is the fastest path and returns the full page.
+       - Otherwise, call `wiki_search` once to find candidate titles, then `wikipedia_page_fetch` on the best one.
+       - Namespaced pages work too, e.g. `wikipedia_page_fetch("Wikipedia:Featured_article_candidates/Featured_log/November_2016")`.
+     - **General web research**: prefer `tavily_web_search` (cleaner, LLM-optimised snippets). Use `duck_web_search` only if Tavily fails.
+     - **When any search result gives you a specific URL**, call `fetch_webpage` to read the full page — do not loop on snippets.
+     - **Do not repeat queries** with trivial rewording. If a search did not help, switch tools or pivot (try a different angle, fetch a referenced page, or go Wikipedia-direct).
+     - **For any YouTube URL**, call `youtube_transcript(url)` to read the spoken content. Only transcript text is returned; if the question requires visual analysis of frames, state that the transcript does not contain the answer rather than guessing.
+  3. **Calculate Precisely**: Use the `calculator` tool for math. Note that `read_excel` and `read_csv` already include column statistics (sum, min, max, mean) — check those first before reaching for `calculator`.
+  4. **Run Code Directly**: If a question asks what a Python script outputs, use `python_eval` to execute it. Do not try to trace execution mentally.
+  5. **Don't Loop**: Each tool call has a cost. If you have already searched a topic twice without progress, escalate (fetch a specific page, switch to Wikipedia-direct, or commit to your best-supported answer).
+
+  # REASONING FRAMEWORK
+  Follow this Chain-of-Thought (CoT) process for every step:
+  1. **Analyze**: What is the core question? What data do I have? What is missing?
+  2. **Plan**: What is the next best tool to use? Why?
+  3. **Execute**: Call the tool.
+  4. **Observe**: Analyze the tool output. Does it answer the question?
+  5. **Refine**: If the output is insufficient, adjust the plan and try a different angle.
+
+  # FINAL ANSWER
+  When you have reached the answer, end your message with a single concluding line of the form:
+
+      FINAL ANSWER: <value>
+
+  A downstream formatter will normalise the value into the strict GAIA scoring format (no commas in numbers, no articles in strings, comma-separated lists, etc.) — so prioritise correctness of the value itself over format pedantry. Do not wrap the line in markdown, do not add commentary after it, and do not restate it.
+
+  # EXAMPLE SCENARIOS
+  - Question: "What is the sum of the 'Total' column in the attached file.xlsx?"
+    Thought: I need to read the excel file first using `read_excel`, then check the column stats it returns.
+    Action: `read_excel(file_path="...")`
+    Observation: (Dataframe + describe() stats…)
+    FINAL ANSWER: 700
+
+  - Question: "Which city is the capital of France?"
+    FINAL ANSWER: Paris
+
+  Take a deep breath and think step by step.

gaia/prompts/vlm_prompt.yaml

@@ -0,0 +1,15 @@
+title: "GAIA VLM Assistant"
+prompt: |
+  # ROLE
+  You are an expert visual assistant AI capable of analyzing images with high precision. Your goal is to extract visual information to answer a SPECIFIC question provided by the user.
+
+  # INSTRUCTION
+  - Analyze the provided image thoroughly.
+  - Focus specifically on the information needed to answer the user's question.
+  - Provide a clear, detailed description of the relevant visual elements.
+  - If the image contains text (documents, charts, screenshots), transcribe or summarize it accurately.
+  - Do NOT generate unrelated descriptions. Stick to the context of the question.
+
+  # FORMAT
+  - Start with a direct answer or observation related to the question.
+  - Follow with supporting details from the image.

gaia/states.py

@@ -0,0 +1,27 @@
+from typing import Annotated, TypedDict, List
+from langgraph.graph.message import add_messages
+from langchain_core.messages import BaseMessage
+from langchain_core.documents import Document
+
+# ============================================
+# State Definition
+# ============================================
+
+class AgentState(TypedDict):
+    """
+    State schema for the GAIA agent graph.
+
+    Attributes:
+        messages: List of conversation messages (auto-accumulated via add_messages)
+        task_id: The GAIA task identifier for the current question
+        file_name: Name of the attached file (empty string if no file)
+        file_path: Local filesystem path to the downloaded file (empty if no file or download failed)
+        retrieved_docs: List of candidate documents from the retriever node
+        final_answer: GAIA-formatted answer produced by the formatter node
+    """
+    messages: Annotated[list[BaseMessage], add_messages]
+    task_id: str
+    file_name: str
+    file_path: str
+    retrieved_docs: List[Document]
+    final_answer: str

gaia/tools/__init__.py

@@ -0,0 +1,40 @@
+"""GAIA agent tool roster. Importing this package exposes every tool and the aggregated `tools_list`."""
+from gaia.tools.basic import calculator, python_eval
+from gaia.tools.web import (
+    duck_web_search, tavily_web_search, wiki_search, wikipedia_page_fetch,
+    arxiv_search, fetch_webpage, youtube_transcript, retry_file_download,
+)
+from gaia.tools.files import (
+    read_pdf, read_docx, read_pptx, read_text_file,
+    read_csv, read_excel, read_jsonld, read_pdb,
+    read_python_file, extract_zip,
+)
+from gaia.tools.media import analyze_image, transcribe_audio
+from gaia.tools.dispatcher import read_file
+
+
+tools_list = [
+    calculator,
+    duck_web_search,
+    wiki_search,
+    wikipedia_page_fetch,
+    arxiv_search,
+    tavily_web_search,
+    fetch_webpage,
+    youtube_transcript,
+    retry_file_download,
+    python_eval,
+    read_pdf,
+    read_docx,
+    read_pptx,
+    read_text_file,
+    read_csv,
+    read_excel,
+    read_jsonld,
+    read_pdb,
+    transcribe_audio,
+    read_python_file,
+    extract_zip,
+    analyze_image,
+    read_file,
+]

gaia/tools/basic.py

@@ -0,0 +1,60 @@
+"""Basic computation tools: calculator and Python code execution."""
+import os
+import subprocess
+import tempfile
+
+from langchain_core.tools import tool
+
+
+@tool
+def calculator(a: float, b: float, type: str) -> float:
+    """Performs mathematical calculations, addition, subtraction, multiplication, division, modulus.
+    Args:
+        a (float): first float number
+        b (float): second float number
+        type (str): the type of calculation to perform, can be addition, subtraction, multiplication, division, modulus
+    """
+    if type == "addition":
+        return a + b
+    elif type == "subtraction":
+        return a - b
+    elif type == "multiplication":
+        return a * b
+    elif type == "division":
+        if b == 0:
+            raise ValueError("Cannot divide by zero.")
+        return a / b
+    elif type == "modulus":
+        return a % b
+    else:
+        raise TypeError(f"{type} is not an option for type, choose one of addition, subtraction, multiplication, division, modulus")
+
+
+@tool
+def python_eval(code: str) -> str:
+    """
+    Execute a Python code snippet and return its stdout output.
+    Use this when a question asks what a script outputs, or when computation requires running code.
+
+    Args:
+        code: Python source code to execute.
+
+    Returns:
+        The stdout output of the code, or an error/timeout message.
+    """
+    try:
+        with tempfile.NamedTemporaryFile(mode='w', suffix='.py', delete=False) as f:
+            f.write(code)
+            tmp_path = f.name
+        result = subprocess.run(
+            ['python3', tmp_path],
+            capture_output=True, text=True, timeout=30
+        )
+        os.unlink(tmp_path)
+        if result.returncode == 0:
+            return f"Output:\n{result.stdout}"
+        return f"[python_eval] exit {result.returncode}:\n{result.stderr}"
+    except subprocess.TimeoutExpired:
+        return "[python_eval] execution timed out (30s limit)"
+    except Exception as e:
+        return f"[python_eval] failed: {e}"

gaia/tools/dispatcher.py

@@ -0,0 +1,55 @@
+"""Generic file-reading dispatcher: routes a path to the correct extension-specific tool."""
+from pathlib import Path
+
+from langchain_core.tools import tool
+
+from gaia.tools.files import (
+    read_pdf, read_docx, read_pptx, read_text_file,
+    read_csv, read_excel, read_jsonld, read_pdb,
+    read_python_file, extract_zip,
+)
+from gaia.tools.media import analyze_image, transcribe_audio
+
+
+@tool
+def read_file(file_path: str) -> str:
+    """
+    Automatically read a file based on its extension.
+
+    Supported formats: PDF, DOCX, PPTX, TXT, CSV, XLSX, JSON-LD, PDB, Python, ZIP, JPG, JPEG, PNG, MP3, WAV, FLAC, OGG, M4A
+
+    Args:
+        file_path: Path to the file to read.
+
+    Returns:
+        The processed content of the file.
+    """
+    ext = Path(file_path).suffix.lower()
+
+    processors = {
+        '.pdf': lambda p: read_pdf.invoke(p),
+        '.docx': lambda p: read_docx.invoke(p),
+        '.pptx': lambda p: read_pptx.invoke(p),
+        '.txt': lambda p: read_text_file.invoke(p),
+        '.csv': lambda p: read_csv.invoke(p),
+        '.xlsx': lambda p: read_excel.invoke(p),
+        '.jsonld': lambda p: read_jsonld.invoke(p),
+        '.pdb': lambda p: read_pdb.invoke(p),
+        '.py': lambda p: read_python_file.invoke(p),
+        '.mp3': lambda p: transcribe_audio.invoke(p),
+        '.wav': lambda p: transcribe_audio.invoke(p),
+        '.flac': lambda p: transcribe_audio.invoke(p),
+        '.ogg': lambda p: transcribe_audio.invoke(p),
+        '.m4a': lambda p: transcribe_audio.invoke(p),
+        '.zip': lambda p: extract_zip.invoke(p),
+        '.jpg': lambda p: analyze_image.invoke({"image_path": p, "question": "Describe this image in detail."}),
+        '.jpeg': lambda p: analyze_image.invoke({"image_path": p, "question": "Describe this image in detail."}),
+        '.png': lambda p: analyze_image.invoke({"image_path": p, "question": "Describe this image in detail."}),
+    }
+
+    processor = processors.get(ext)
+
+    if processor:
+        return processor(file_path)
+
+    return f"[Unsupported file type: {ext}]"

gaia/tools/files.py

@@ -0,0 +1,296 @@
+"""File / document / archive reading tools."""
+import json
+import os
+import zipfile
+from pathlib import Path
+
+from langchain_core.tools import tool
+
+
+@tool
+def read_pdf(file_path: str) -> str:
+    """
+    Extract text content from a PDF file.
+
+    Args:
+        file_path: Path to the PDF file to read.
+
+    Returns:
+        The text content of the PDF, with page separators.
+    """
+    from pypdf import PdfReader
+
+    try:
+        reader = PdfReader(file_path)
+        text = []
+        for i, page in enumerate(reader.pages):
+            page_text = page.extract_text()
+            if page_text:
+                text.append(f"--- Page {i+1} ---\n{page_text}")
+
+        return "\n\n".join(text) if text else "[Empty PDF]"
+    except Exception as e:
+        return f"[read_pdf] failed to read PDF: {e}"
+
+
+@tool
+def read_docx(file_path: str) -> str:
+    """
+    Extract text content from a Word document (.docx).
+
+    Args:
+        file_path: Path to the Word document to read.
+
+    Returns:
+        The text content of the document.
+    """
+    from docx import Document
+
+    try:
+        doc = Document(file_path)
+        text_parts = []
+
+        paragraphs = [para.text for para in doc.paragraphs if para.text.strip()]
+        if paragraphs:
+            text_parts.append("\n".join(paragraphs))
+
+        for i, table in enumerate(doc.tables):
+            rows = [" | ".join(cell.text.strip() for cell in row.cells) for row in table.rows]
+            rows = [r for r in rows if r.strip()]
+            if rows:
+                text_parts.append(f"--- Table {i+1} ---\n" + "\n".join(rows))
+
+        return "\n\n".join(text_parts) if text_parts else "[Empty document]"
+    except Exception as e:
+        return f"[read_docx] failed to read DOCX: {e}"
+
+
+@tool
+def read_pptx(file_path: str) -> str:
+    """
+    Extract text content from a PowerPoint presentation (.pptx).
+
+    Args:
+        file_path: Path to the PowerPoint file to read.
+
+    Returns:
+        The text content from all slides.
+    """
+    from pptx import Presentation
+
+    try:
+        prs = Presentation(file_path)
+        text = []
+
+        for slide_num, slide in enumerate(prs.slides, 1):
+            slide_text = [f"--- Slide {slide_num} ---"]
+            for shape in slide.shapes:
+                if hasattr(shape, "text") and shape.text.strip():
+                    slide_text.append(shape.text)
+            if len(slide_text) > 1:
+                text.append("\n".join(slide_text))
+
+        return "\n\n".join(text) if text else "[Empty presentation]"
+    except Exception as e:
+        return f"[read_pptx] failed to read PPTX: {e}"
+
+
+@tool
+def read_text_file(file_path: str) -> str:
+    """
+    Read content from a plain text file (.txt).
+
+    Args:
+        file_path: Path to the text file to read.
+
+    Returns:
+        The content of the text file.
+    """
+    try:
+        with open(file_path, 'r', encoding='utf-8', errors='replace') as f:
+            return f.read()
+    except Exception as e:
+        return f"[read_text_file] failed: {e}"
+
+
+@tool
+def read_csv(file_path: str) -> str:
+    """
+    Read and analyze a CSV file using polars.
+
+    Args:
+        file_path: Path to the CSV file to read.
+
+    Returns:
+        Summary of the CSV including schema, row count, and data preview.
+    """
+    import polars as pl
+
+    try:
+        df = pl.read_csv(file_path)
+
+        output = f"CSV File — {len(df)} rows, {len(df.columns)} columns\n"
+        output += f"Columns: {df.columns}\n\n"
+        output += f"Column Statistics:\n{df.describe()}\n\n"
+        output += f"Data (first 20 rows):\n{df.head(20)}"
+        if len(df) <= 50:
+            output += f"\n\nComplete data:\n{df}"
+        return output
+    except Exception as e:
+        return f"[read_csv] failed to read CSV: {e}"
+
+
+@tool
+def read_excel(file_path: str, sheet_id: int = 0) -> str:
+    """
+    Read and analyze an Excel file (.xlsx) using polars.
+
+    Args:
+        file_path: Path to the Excel file to read.
+        sheet_id: The sheet index to read (0-based). Default is 0 (first sheet).
+
+    Returns:
+        Summary of the Excel sheet including schema, row count, and data preview.
+    """
+    import polars as pl
+    import openpyxl
+
+    try:
+        wb = openpyxl.load_workbook(file_path, read_only=True)
+        sheet_names = wb.sheetnames
+        wb.close()
+    except Exception:
+        sheet_names = []
+
+    try:
+        df = pl.read_excel(file_path, sheet_id=sheet_id)
+        sheet_label = sheet_names[sheet_id] if sheet_id < len(sheet_names) else str(sheet_id)
+
+        output = f"Excel File — Available sheets: {sheet_names}\n\n"
+        output += f"Sheet {sheet_id} ('{sheet_label}') — {len(df)} rows, {len(df.columns)} columns\n"
+        output += f"Columns: {df.columns}\n\n"
+        output += f"Column Statistics:\n{df.describe()}\n\n"
+        output += f"Data (first 20 rows):\n{df.head(20)}"
+        if len(df) <= 50:
+            output += f"\n\nComplete data:\n{df}"
+        return output
+    except Exception as e:
+        return f"[read_excel] failed to read Excel: {e}"
+
+
+@tool
+def read_jsonld(file_path: str) -> str:
+    """
+    Read and parse a JSON-LD file.
+
+    Args:
+        file_path: Path to the JSON-LD file to read.
+
+    Returns:
+        The formatted JSON content.
+    """
+    try:
+        with open(file_path, 'r') as f:
+            data = json.load(f)
+        return f"JSON-LD Content:\n{json.dumps(data, indent=2)}"
+    except Exception as e:
+        return f"[read_jsonld] failed to read JSON-LD: {e}"
+
+
+@tool
+def read_pdb(file_path: str) -> str:
+    """
+    Read and analyze a PDB (Protein Data Bank) file for protein structure analysis.
+
+    Args:
+        file_path: Path to the PDB file to read.
+
+    Returns:
+        Analysis of the protein structure including atoms, chains, and coordinates.
+    """
+    from Bio.PDB import PDBParser
+    import numpy as np
+
+    try:
+        parser = PDBParser(QUIET=True)
+        structure = parser.get_structure("protein", file_path)
+
+        info = ["=== PDB Structure Analysis ==="]
+
+        atoms = list(structure.get_atoms())
+        info.append(f"Total atoms: {len(atoms)}")
+
+        for model in structure:
+            info.append(f"\nModel {model.id}:")
+            for chain in model:
+                residues = list(chain.get_residues())
+                info.append(f"  Chain {chain.id}: {len(residues)} residues")
+
+        if len(atoms) >= 2:
+            info.append("\nFirst atoms (for distance calculations):")
+            for i, atom in enumerate(atoms[:5]):
+                coord = atom.get_coord()
+                info.append(
+                    f"  Atom {i+1}: {atom.get_name()} at "
+                    f"[{coord[0]:.4f}, {coord[1]:.4f}, {coord[2]:.4f}]"
+                )
+
+            dist = np.linalg.norm(atoms[0].get_coord() - atoms[1].get_coord())
+            info.append(f"\nDistance between first two atoms: {dist:.4f} Angstroms")
+
+        return "\n".join(info)
+    except Exception as e:
+        return f"[read_pdb] failed to read PDB: {e}"
+
+
+@tool
+def read_python_file(file_path: str) -> str:
+    """
+    Read a Python source code file.
+
+    Args:
+        file_path: Path to the Python file to read.
+
+    Returns:
+        The Python code content.
+    """
+    try:
+        with open(file_path, 'r') as f:
+            code = f.read()
+        return f"Python Code:\n```python\n{code}\n```"
+    except Exception as e:
+        return f"[read_python_file] failed: {e}"
+
+
+@tool
+def extract_zip(file_path: str) -> str:
+    """
+    Extract a ZIP archive and list its contents.
+
+    Args:
+        file_path: Path to the ZIP file to extract.
+
+    Returns:
+        List of files extracted from the archive with their paths.
+    """
+    try:
+        extract_dir = Path(file_path).parent / Path(file_path).stem
+        extract_dir.mkdir(exist_ok=True)
+
+        with zipfile.ZipFile(file_path, 'r') as zip_ref:
+            zip_ref.extractall(extract_dir)
+
+        results = [f"ZIP Archive extracted to: {extract_dir}\n\nContents:"]
+
+        for root, dirs, files in os.walk(extract_dir):
+            for file in files:
+                full_path = os.path.join(root, file)
+                rel_path = os.path.relpath(full_path, extract_dir)
+                file_size = os.path.getsize(full_path)
+                results.append(f"  - {rel_path} ({file_size} bytes)")
+
+        results.append(f"\nUse the appropriate read tool on the extracted files at: {extract_dir}/")
+
+        return "\n".join(results)
+    except Exception as e:
+        return f"[extract_zip] failed: {e}"

gaia/tools/media.py

@@ -0,0 +1,96 @@
+"""Multi-modal tools: image analysis (VLM) and audio transcription (ASR)."""
+import base64
+import os
+from pathlib import Path
+
+from dotenv import load_dotenv
+
+load_dotenv()
+
+from huggingface_hub import InferenceClient
+from langchain_core.tools import tool
+
+from gaia.utils import load_config, load_prompt
+
+
+_PROMPTS_DIR = Path(__file__).resolve().parent.parent / "prompts"
+
+_config = load_config()
+_vlm_model_name = _config["models"]["vlm"]["model_name"]
+_vlm_system_prompt = load_prompt(str(_PROMPTS_DIR / "vlm_prompt.yaml")).content
+_asr_model_name = _config["models"]["asr"]["model_name"]
+_hf_client = InferenceClient(token=os.getenv("HF_INFERENCE_KEY"))
+
+
+@tool
+def analyze_image(image_path: str, question: str) -> str:
+    """
+    Analyze an image using a Vision Language Model (VLM) to answer a specific question.
+
+    Args:
+        image_path: Path to the image file (JPG, PNG).
+        question: The specific question to answer about the image.
+
+    Returns:
+        A detailed description or answer based on the visual content.
+    """
+    try:
+        if not os.path.exists(image_path):
+            return f"[analyze_image] image file not found at {image_path}"
+
+        with open(image_path, "rb") as img_file:
+            image_data = base64.b64encode(img_file.read()).decode("utf-8")
+        ext = Path(image_path).suffix.lower().lstrip(".")
+        mime_type = "image/jpeg" if ext in ("jpg", "jpeg") else f"image/{ext}"
+        image_url = f"data:{mime_type};base64,{image_data}"
+
+        messages = [
+            {
+                "role": "user",
+                "content": [
+                    {"type": "image_url", "image_url": {"url": image_url}},
+                    {"type": "text", "text": f"{_vlm_system_prompt}\n\nQuestion: {question}"}
+                ]
+            }
+        ]
+
+        output = _hf_client.chat_completion(
+            messages=messages,
+            model=_vlm_model_name,
+            max_tokens=1000
+        )
+
+        return output.choices[0].message.content
+
+    except Exception as e:
+        return f"[analyze_image] VLM call failed: {e}"
+
+
+@tool
+def transcribe_audio(file_path: str) -> str:
+    """
+    Transcribe an audio file (MP3, WAV, etc.) to text using Whisper.
+
+    Args:
+        file_path: Path to the audio file to transcribe.
+
+    Returns:
+        The transcribed text from the audio, or a detailed `[transcribe_audio] ...`
+        error string identifying file path, size, model, and exception class+message.
+    """
+    if not os.path.exists(file_path):
+        return f"[transcribe_audio] file not found at {file_path}"
+    file_size = os.path.getsize(file_path)
+    if file_size == 0:
+        return f"[transcribe_audio] file is empty at {file_path}"
+
+    try:
+        with open(file_path, "rb") as f:
+            audio_bytes = f.read()
+        result = _hf_client.automatic_speech_recognition(audio=audio_bytes, model=_asr_model_name)
+        return f"Audio Transcription:\n{result.text}"
+    except Exception as e:
+        return (
+            f"[transcribe_audio] ASR call failed for {file_path} ({file_size} bytes) "
+            f"with model '{_asr_model_name}': {type(e).__name__}: {e}"
+        )

gaia/tools/web.py

@@ -0,0 +1,209 @@
+"""Web search and fetching tools: DuckDuckGo, Tavily, Wikipedia, Arxiv, webpage fetch, YouTube transcripts."""
+from langchain_community.tools import DuckDuckGoSearchRun
+from langchain_community.tools.tavily_search import TavilySearchResults
+from langchain_community.document_loaders import WikipediaLoader, ArxivLoader
+from langchain_core.tools import tool
+
+from gaia.utils import extract_youtube_id, load_config, download_task_file
+
+
+_ddg_search = None
+_tavily_search = None
+
+
+def _get_ddg():
+    global _ddg_search
+    if _ddg_search is None:
+        _ddg_search = DuckDuckGoSearchRun()
+    return _ddg_search
+
+
+def _get_tavily():
+    global _tavily_search
+    if _tavily_search is None:
+        _tavily_search = TavilySearchResults(max_results=3)
+    return _tavily_search
+
+
+@tool
+def duck_web_search(query: str) -> str:
+    """Use DuckDuckGo to search the web.
+
+    Args:
+        query: The search query.
+    """
+    search = _get_ddg().invoke(input=query)
+    return {"duckduckgo_web_search": search}
+
+
+@tool
+def wiki_search(query: str) -> str:
+    """Search Wikipedia for a query and return maximum 3 results.
+
+    Args:
+        query: The search query."""
+    documents = WikipediaLoader(query=query, load_max_docs=3, doc_content_chars_max=20000).load()
+    processed_documents = "\n\n---\n\n".join(
+        [
+            f'Document title: {document.metadata.get("title", "")}. Summary: {document.metadata.get("summary", "")}. Documents details: {document.page_content}'
+            for document in documents
+        ])
+    return {"wiki_results": processed_documents}
+
+
+@tool
+def wikipedia_page_fetch(title: str) -> str:
+    """Fetch the full text of a Wikipedia page by its exact title.
+
+    Use this when you can guess the page title from the question (e.g., "1928 Summer
+    Olympics", "List of Featured Articles"). Faster than search + fetch_webpage when
+    the title is obvious. Returns full page content, not a search summary.
+
+    Args:
+        title: The exact Wikipedia page title (including namespace prefix if applicable,
+               e.g., "Wikipedia:Featured_article_candidates/Featured_log/November_2016").
+
+    Returns:
+        The page content prefixed with title and URL, or a `[wikipedia_page_fetch] ...`
+        error string.
+    """
+    import wikipedia
+    try:
+        page = wikipedia.page(title, auto_suggest=False)
+        return f"Wikipedia: {page.title}\nURL: {page.url}\n\n{page.content}"
+    except wikipedia.exceptions.DisambiguationError as e:
+        return f"[wikipedia_page_fetch] '{title}' is a disambiguation page. Options: {e.options[:10]}"
+    except wikipedia.exceptions.PageError:
+        return f"[wikipedia_page_fetch] page not found: '{title}'. Try wiki_search to find the correct title."
+    except Exception as e:
+        return f"[wikipedia_page_fetch] failed: {e}"
+
+
+@tool
+def arxiv_search(query: str) -> str:
+    """Search Arxiv for a query and return maximum 3 result.
+
+    Args:
+        query: The search query."""
+    documents = ArxivLoader(query=query, load_max_docs=3).load()
+    processed_documents = "\n\n---\n\n".join(
+        [
+            f'Document title: {document.metadata.get("title", "")}. Summary: {document.metadata.get("summary", "")}. Documents details: {document.page_content}'
+            for document in documents
+        ])
+    return {"arxiv_results": processed_documents}
+
+
+@tool
+def tavily_web_search(query: str) -> str:
+    """Search the web using Tavily for a query and return maximum 3 results.
+
+    Args:
+        query: The search query."""
+    search_documents = _get_tavily().invoke(input=query)
+    web_results = "\n\n---\n\n".join(
+        [
+            f'Document title: {document["title"]}. Contents: {document["content"]}. Relevance Score: {document["score"]}'
+            for document in search_documents
+        ])
+    return {"web_results": web_results}
+
+
+@tool
+def fetch_webpage(url: str) -> str:
+    """
+    Fetch and extract the main text content from a webpage.
+    Use this when a search result points to a specific URL you need to read in full.
+
+    Args:
+        url: The full URL of the page to fetch.
+
+    Returns:
+        The extracted text content of the page.
+    """
+    import trafilatura
+    try:
+        downloaded = trafilatura.fetch_url(url)
+        if downloaded is None:
+            return f"[fetch_webpage] could not fetch {url}"
+        text = trafilatura.extract(downloaded, include_tables=True, include_links=False)
+        if text is None:
+            return f"[fetch_webpage] could not extract content from {url}"
+        return f"Page content from {url}:\n\n{text}"
+    except Exception as e:
+        return f"[fetch_webpage] failed: {e}"
+
+
+@tool
+def retry_file_download(task_id: str, file_name: str) -> str:
+    """Retry downloading the task file from the GAIA scoring API.
+
+    Use this when the initial automatic download failed (you will see a message like
+    "the automatic download failed" in the question context). Returns the local path
+    on success, or an error string starting with `[retry_file_download]`.
+
+    Args:
+        task_id: The task ID for the current question.
+        file_name: The original file name from the question metadata.
+
+    Returns:
+        Local filesystem path to the downloaded file, or an error description.
+    """
+    cfg = load_config()
+    local_path, err = download_task_file(
+        task_id=task_id,
+        file_name=file_name,
+        base_url=cfg["api"]["base_url"],
+        files_dir=cfg["api"]["files_dir"],
+    )
+    if local_path:
+        return local_path
+    return f"[retry_file_download] {err}"
+
+
+@tool
+def youtube_transcript(url: str) -> str:
+    """Fetch the transcript (captions) of a YouTube video as plain text.
+
+    Use this whenever a question references a YouTube URL — the spoken content of
+    the video is available via captions. Note: this returns text only; questions
+    that require visual analysis of the frames cannot be answered from the
+    transcript alone.
+
+    Prefers manually-written English captions; falls back to auto-generated English,
+    and finally to any available language.
+
+    Args:
+        url: The full YouTube URL (watch, youtu.be, embed, shorts) or a bare 11-char video ID.
+
+    Returns:
+        The concatenated transcript text, or an error string starting with `[youtube_transcript]`.
+    """
+    from youtube_transcript_api import YouTubeTranscriptApi
+    from youtube_transcript_api._errors import (
+        TranscriptsDisabled, NoTranscriptFound, VideoUnavailable,
+    )
+
+    video_id = extract_youtube_id(url)
+    if not video_id:
+        return f"[youtube_transcript] could not parse video ID from: {url}"
+
+    try:
+        ytt_api = YouTubeTranscriptApi()
+        try:
+            fetched = ytt_api.fetch(video_id, languages=['en'])
+        except NoTranscriptFound:
+            transcript_list = ytt_api.list(video_id)
+            transcript = next(iter(transcript_list))
+            fetched = transcript.fetch()
+
+        text = " ".join(snippet.text for snippet in fetched)
+        return f"YouTube transcript for {url}:\n\n{text}"
+    except TranscriptsDisabled:
+        return f"[youtube_transcript] transcripts are disabled for {url}"
+    except VideoUnavailable:
+        return f"[youtube_transcript] video unavailable: {url}"
+    except NoTranscriptFound:
+        return f"[youtube_transcript] no transcript found for {url}"
+    except Exception as e:
+        return f"[youtube_transcript] failed: {e}"

gaia/utils.py

@@ -0,0 +1,168 @@
+import os
+import re
+import json
+import time
+import bm25s
+import requests
+import yaml
+from pathlib import Path
+from langchain_core.messages import SystemMessage
+
+
+_YOUTUBE_ID_RE = re.compile(
+    r'(?:youtube\.com/watch\?v=|youtu\.be/|youtube\.com/embed/|youtube\.com/v/|youtube\.com/shorts/)([\w-]{11})'
+)
+
+
+def extract_youtube_id(url: str) -> str | None:
+    """Pull the 11-char video ID from any common YouTube URL form, or accept a bare ID."""
+    m = _YOUTUBE_ID_RE.search(url)
+    if m:
+        return m.group(1)
+    if re.fullmatch(r'[\w-]{11}', url.strip()):
+        return url.strip()
+    return None
+
+
+_FINAL_ANSWER_RE = re.compile(r'FINAL ANSWER:\s*(.*)', re.DOTALL | re.IGNORECASE)
+
+
+def extract_final_answer(content: str) -> str:
+    """Pull the value after 'FINAL ANSWER:' (case-insensitive), or return the content stripped."""
+    content = content or ""
+    m = _FINAL_ANSWER_RE.search(content)
+    return (m.group(1) if m else content).strip()
+
+
+def load_config(path="config.yaml"):
+    with open(path, "r") as f:
+        return yaml.safe_load(f)
+
+
+def download_task_file(
+    task_id: str,
+    file_name: str,
+    base_url: str,
+    files_dir: str,
+    max_retries: int = 3,
+    timeout: int = 30,
+) -> tuple[str | None, str]:
+    """Download a task file from the GAIA scoring API.
+
+    Returns (local_path, error_message). On success the error string is empty;
+    on failure local_path is None and the error string says what went wrong.
+    Retries on 5xx and network errors with exponential backoff (2s, 4s); 4xx
+    is treated as a definitive "no file" answer and not retried.
+    """
+    if not task_id or not file_name:
+        return None, "missing task_id or file_name"
+
+    safe_name = Path(file_name).name
+    if not safe_name:
+        return None, f"invalid file_name '{file_name}'"
+
+    try:
+        save_dir = Path(files_dir) / task_id
+        save_dir.mkdir(parents=True, exist_ok=True)
+    except Exception as e:
+        return None, f"could not create cache dir: {e}"
+
+    local_path = save_dir / safe_name
+    if local_path.exists() and local_path.stat().st_size > 0:
+        return str(local_path), ""
+
+    url = f"{base_url}/files/{task_id}"
+    last_err = ""
+    for attempt in range(1, max_retries + 1):
+        try:
+            response = requests.get(url, timeout=timeout)
+            status = response.status_code
+            if status >= 500:
+                last_err = f"HTTP {status} (attempt {attempt}/{max_retries})"
+                if attempt < max_retries:
+                    time.sleep(2 * attempt)
+                continue
+            if status >= 400:
+                return None, f"HTTP {status} from {url}"
+            if not response.content:
+                last_err = f"empty body (attempt {attempt}/{max_retries})"
+                if attempt < max_retries:
+                    time.sleep(2 * attempt)
+                continue
+            local_path.write_bytes(response.content)
+            return str(local_path), ""
+        except requests.RequestException as e:
+            last_err = f"{type(e).__name__}: {e} (attempt {attempt}/{max_retries})"
+            if attempt < max_retries:
+                time.sleep(2 * attempt)
+        except Exception as e:
+            return None, f"unexpected error: {type(e).__name__}: {e}"
+
+    return None, f"all {max_retries} attempts failed; last: {last_err}"
+
+
+def load_prompt(prompt_location: str) -> SystemMessage:
+    """Load system prompt from YAML file."""
+    with open(prompt_location) as f:
+        try:
+            prompt = yaml.safe_load(f)["prompt"]
+            return SystemMessage(content=prompt)
+        except yaml.YAMLError as exc:
+            print(exc)
+            return SystemMessage(content="You are a helpful assistant.")
+
+
+
+def init_bm25_index(corpus_file = "data/metadata.jsonl"):
+    """BM25 Index Initialization (Local Corpus)"""
+    try:
+        if not os.path.exists(corpus_file):
+            print(f"Warning: {corpus_file} not found. BM25 will use empty index.")
+            return None, [], []
+            
+        search_texts = []  # question-only — used for BM25 indexing
+        corpus_texts = []  # Q+A+Steps — returned for context injection
+        corpus_ids = []
+        with open(corpus_file, "r") as f:
+            for line in f:
+                item = json.loads(line)
+                question = item.get('Question', '')
+                answer = item.get('Final answer', '')
+                steps = item.get('Annotator Metadata', {}).get('Steps', '')
+                search_texts.append(question)
+                parts = [f"Question: {question}"]
+                if answer:
+                    parts.append(f"Final Answer: {answer}")
+                if steps:
+                    parts.append(f"Solution Steps: {steps}")
+                corpus_texts.append("\n".join(parts))
+                corpus_ids.append(item.get('task_id', ''))
+
+        corpus_tokens = bm25s.tokenize(search_texts, stopwords="en", stemmer=None)
+        
+        retriever_bm25 = bm25s.BM25()
+        retriever_bm25.index(corpus_tokens)
+        
+        print(f"BM25 Index initialized with {len(corpus_texts)} documents.")
+        return retriever_bm25, corpus_texts, corpus_ids
+    except Exception as e:
+        print(f"Error initializing BM25: {e}")
+        return None, [], []
+    
+
+def reciprocal_rank_fusion(results: list[list[dict]], k=60) -> list[tuple[dict, float]]:
+    """
+    Fuse multiple ranked lists using Reciprocal Rank Fusion (RRF).
+    """
+    fused_scores = {}
+    
+    for rank_list in results:
+        for rank, doc in enumerate(rank_list):
+            doc_id = doc["metadata"]["task_id"]
+            doc_content = doc["content"]
+            if doc_id not in fused_scores:
+                fused_scores[doc_id] = {"id": doc_id, "content": doc_content, "score": 0.0}
+            fused_scores[doc_id]["score"] += 1.0 / (k + rank + 1)
+            
+    sorted_results = sorted(fused_scores.values(), key=lambda x: x["score"], reverse=True)
+    return [(item["id"], item["content"], item["score"]) for item in sorted_results]

notebooks/create_vector_database.ipynb

@@ -0,0 +1,111 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "a9f7a25f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Loading environment variables and initializing Supabase client and SentenceTransformer model\n",
+    "import os\n",
+    "import json\n",
+    "from dotenv import load_dotenv\n",
+    "from supabase.client import Client, create_client\n",
+    "from sentence_transformers import SentenceTransformer\n",
+    "from utils import load_config\n",
+    "\n",
+    "load_dotenv()\n",
+    "\n",
+    "config = load_config()\n",
+    "data = config[\"data\"]\n",
+    "\n",
+    "supabase_url = os.getenv(\"SUPABASE_URL\")\n",
+    "supabase_key = os.getenv(\"SUPABASE_SERVICE_KEY\")\n",
+    "\n",
+    "supabase: Client = create_client(supabase_url, supabase_key)\n",
+    "embeddings = SentenceTransformer(model_name_or_path=config[\"vector_store\"][\"embedding_model_name\"], cache_folder=config[\"models\"][\"cache_folder\"])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "f2c5492b",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/home/kpatelis/projects/gaia/.venv/lib/python3.13/site-packages/torch/_dynamo/guards.py:1114: RuntimeWarning: Guards may run slower on Python 3.13.0. Consider upgrading to Python 3.13.1+.\n",
+      "  warnings.warn(\n",
+      "/home/kpatelis/projects/gaia/.venv/lib/python3.13/site-packages/torch/_dynamo/guards.py:1114: RuntimeWarning: Guards may run slower on Python 3.13.0. Consider upgrading to Python 3.13.1+.\n",
+      "  warnings.warn(\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Reading JSONL file and creating documents with embeddings\n",
+    "with open(data, 'r') as jsonl_file:\n",
+    "    json_list = list(jsonl_file)\n",
+    "\n",
+    "documents = []\n",
+    "for json_str in json_list:\n",
+    "    json_data = json.loads(json_str)\n",
+    "    content = f\"{json_data['Question']}\"\n",
+    "    embedding = embeddings.encode(content, normalize_embeddings=True).tolist()\n",
+    "    document = {\n",
+    "        \"content\": content,\n",
+    "        \"metadata\": {\n",
+    "            \"source\": \"vector_search\",\n",
+    "            \"task_id\": json_data['task_id']\n",
+    "        },\n",
+    "        \"embedding\": embedding,\n",
+    "    }\n",
+    "    documents.append(document)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "26ddbafd",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Inserting documents into Supabase\n",
+    "\n",
+    "# Note1: pgvector needs to be enabled, to turn to vector database\n",
+    "# Note2: Table needs to be created beforehand in Supabase, with column types\n",
+    "try:\n",
+    "    response = (\n",
+    "        supabase.table(\"gaia_documents\")\n",
+    "        .insert(documents)\n",
+    "        .execute()\n",
+    "    )\n",
+    "except Exception as exception:\n",
+    "    print(\"Error inserting data into Supabase:\", exception)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "gaia",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.13.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

notebooks/test.ipynb

@@ -0,0 +1,362 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "5b9902eb",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "903fa846aba0401ba19cada8e6c8822c",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Split strings:   0%|          | 0/165 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "dcac27eb462440d4af110bad3875e30e",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "BM25S Count Tokens:   0%|          | 0/165 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "d8478a4875064b8b9dce4c307cafc4de",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "BM25S Compute Scores:   0%|          | 0/165 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "BM25 Index initialized with 165 documents.\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "bd92d31479044b398df857e3bcd002d4",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "modules.json:   0%|          | 0.00/229 [00:00<?, ?B/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "12e26abd3b3c4cfd8729ddaa7c22989e",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "config_sentence_transformers.json:   0%|          | 0.00/205 [00:00<?, ?B/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "b6f025204e5b4e89b3f09675cd9c9c7d",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "README.md: 0.00B [00:00, ?B/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "3ef0c9c4684a462ca31eae8fea18a6a6",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "config.json: 0.00B [00:00, ?B/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "c200e6620591481c90bcd5f408adfe58",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "model.safetensors:   0%|          | 0.00/298M [00:00<?, ?B/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "8aed0e33a1d841ecbf5c452b4f100bec",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "tokenizer_config.json: 0.00B [00:00, ?B/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "25480a7027e6469e8ea2d80b27131192",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "tokenizer.json: 0.00B [00:00, ?B/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "01dfe70506ba4d78bd270bdf96501d40",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "special_tokens_map.json:   0%|          | 0.00/694 [00:00<?, ?B/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "299bfe9f54bf4d7e9dbac5bac484ffa6",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "config.json:   0%|          | 0.00/297 [00:00<?, ?B/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "ename": "SupabaseException",
+     "evalue": "supabase_url is required",
+     "output_type": "error",
+     "traceback": [
+      "\u001b[31m---------------------------------------------------------------------------\u001b[39m",
+      "\u001b[31mSupabaseException\u001b[39m                         Traceback (most recent call last)",
+      "\u001b[36mCell\u001b[39m\u001b[36m \u001b[39m\u001b[32mIn[1]\u001b[39m\u001b[32m, line 56\u001b[39m\n\u001b[32m     53\u001b[39m     embeddings = SentenceTransformer(model_name_or_path=config[\u001b[33m\"\u001b[39m\u001b[33mmodels\u001b[39m\u001b[33m\"\u001b[39m][\u001b[33m\"\u001b[39m\u001b[33membeddings\u001b[39m\u001b[33m\"\u001b[39m][\u001b[33m\"\u001b[39m\u001b[33mmodel_name\u001b[39m\u001b[33m\"\u001b[39m], cache_folder=config[\u001b[33m\"\u001b[39m\u001b[33mmodels\u001b[39m\u001b[33m\"\u001b[39m][\u001b[33m\"\u001b[39m\u001b[33mcache_folder\u001b[39m\u001b[33m\"\u001b[39m])\n\u001b[32m     55\u001b[39m     \u001b[38;5;66;03m# Supabase Vector Store\u001b[39;00m\n\u001b[32m---> \u001b[39m\u001b[32m56\u001b[39m     supabase: Client = \u001b[43mcreate_client\u001b[49m\u001b[43m(\u001b[49m\u001b[43msupabase_url\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43msupabase_key\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m     58\u001b[39m \u001b[38;5;66;03m# Reranker Model (ModernBERT Cross-Encoder)\u001b[39;00m\n\u001b[32m     59\u001b[39m reranker = CrossEncoder(config[\u001b[33m\"\u001b[39m\u001b[33mmodels\u001b[39m\u001b[33m\"\u001b[39m][\u001b[33m\"\u001b[39m\u001b[33mreranker\u001b[39m\u001b[33m\"\u001b[39m][\u001b[33m\"\u001b[39m\u001b[33mmodel_name\u001b[39m\u001b[33m\"\u001b[39m], cache_folder=config[\u001b[33m\"\u001b[39m\u001b[33mmodels\u001b[39m\u001b[33m\"\u001b[39m][\u001b[33m\"\u001b[39m\u001b[33mcache_folder\u001b[39m\u001b[33m\"\u001b[39m])\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m~/projects/gaia/.venv/lib/python3.13/site-packages/supabase/_sync/client.py:371\u001b[39m, in \u001b[36mcreate_client\u001b[39m\u001b[34m(supabase_url, supabase_key, options)\u001b[39m\n\u001b[32m    340\u001b[39m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34mcreate_client\u001b[39m(\n\u001b[32m    341\u001b[39m     supabase_url: \u001b[38;5;28mstr\u001b[39m,\n\u001b[32m    342\u001b[39m     supabase_key: \u001b[38;5;28mstr\u001b[39m,\n\u001b[32m    343\u001b[39m     options: Optional[ClientOptions] = \u001b[38;5;28;01mNone\u001b[39;00m,\n\u001b[32m    344\u001b[39m ) -> Client:\n\u001b[32m    345\u001b[39m \u001b[38;5;250m    \u001b[39m\u001b[33;03m\"\"\"Create client function to instantiate supabase client like JS runtime.\u001b[39;00m\n\u001b[32m    346\u001b[39m \n\u001b[32m    347\u001b[39m \u001b[33;03m    Parameters\u001b[39;00m\n\u001b[32m   (...)\u001b[39m\u001b[32m    369\u001b[39m \u001b[33;03m    Client\u001b[39;00m\n\u001b[32m    370\u001b[39m \u001b[33;03m    \"\"\"\u001b[39;00m\n\u001b[32m--> \u001b[39m\u001b[32m371\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mClient\u001b[49m\u001b[43m.\u001b[49m\u001b[43mcreate\u001b[49m\u001b[43m(\u001b[49m\n\u001b[32m    372\u001b[39m \u001b[43m        \u001b[49m\u001b[43msupabase_url\u001b[49m\u001b[43m=\u001b[49m\u001b[43msupabase_url\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43msupabase_key\u001b[49m\u001b[43m=\u001b[49m\u001b[43msupabase_key\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43moptions\u001b[49m\u001b[43m=\u001b[49m\u001b[43moptions\u001b[49m\n\u001b[32m    373\u001b[39m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m~/projects/gaia/.venv/lib/python3.13/site-packages/supabase/_sync/client.py:104\u001b[39m, in \u001b[36mClient.create\u001b[39m\u001b[34m(cls, supabase_url, supabase_key, options)\u001b[39m\n\u001b[32m     96\u001b[39m \u001b[38;5;129m@classmethod\u001b[39m\n\u001b[32m     97\u001b[39m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34mcreate\u001b[39m(\n\u001b[32m     98\u001b[39m     \u001b[38;5;28mcls\u001b[39m,\n\u001b[32m   (...)\u001b[39m\u001b[32m    101\u001b[39m     options: Optional[ClientOptions] = \u001b[38;5;28;01mNone\u001b[39;00m,\n\u001b[32m    102\u001b[39m ) -> \u001b[33m\"\u001b[39m\u001b[33mClient\u001b[39m\u001b[33m\"\u001b[39m:\n\u001b[32m    103\u001b[39m     auth_header = options.headers.get(\u001b[33m\"\u001b[39m\u001b[33mAuthorization\u001b[39m\u001b[33m\"\u001b[39m) \u001b[38;5;28;01mif\u001b[39;00m options \u001b[38;5;28;01melse\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[32m--> \u001b[39m\u001b[32m104\u001b[39m     client = \u001b[38;5;28;43mcls\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43msupabase_url\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43msupabase_key\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43moptions\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m    106\u001b[39m     \u001b[38;5;28;01mif\u001b[39;00m auth_header \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[32m    107\u001b[39m         \u001b[38;5;28;01mtry\u001b[39;00m:\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m~/projects/gaia/.venv/lib/python3.13/site-packages/supabase/_sync/client.py:56\u001b[39m, in \u001b[36mClient.__init__\u001b[39m\u001b[34m(self, supabase_url, supabase_key, options)\u001b[39m\n\u001b[32m     42\u001b[39m \u001b[38;5;250m\u001b[39m\u001b[33;03m\"\"\"Instantiate the client.\u001b[39;00m\n\u001b[32m     43\u001b[39m \n\u001b[32m     44\u001b[39m \u001b[33;03mParameters\u001b[39;00m\n\u001b[32m   (...)\u001b[39m\u001b[32m     52\u001b[39m \u001b[33;03m    `DEFAULT_OPTIONS` dict.\u001b[39;00m\n\u001b[32m     53\u001b[39m \u001b[33;03m\"\"\"\u001b[39;00m\n\u001b[32m     55\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m supabase_url:\n\u001b[32m---> \u001b[39m\u001b[32m56\u001b[39m     \u001b[38;5;28;01mraise\u001b[39;00m SupabaseException(\u001b[33m\"\u001b[39m\u001b[33msupabase_url is required\u001b[39m\u001b[33m\"\u001b[39m)\n\u001b[32m     57\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m supabase_key:\n\u001b[32m     58\u001b[39m     \u001b[38;5;28;01mraise\u001b[39;00m SupabaseException(\u001b[33m\"\u001b[39m\u001b[33msupabase_key is required\u001b[39m\u001b[33m\"\u001b[39m)\n",
+      "\u001b[31mSupabaseException\u001b[39m: supabase_url is required"
+     ]
+    }
+   ],
+   "source": [
+    "\"\"\"\n",
+    "GAIA Agent with Multi-Modal File Processing and Hybrid Retrieval.\n",
+    "\n",
+    "This module defines a LangGraph agent that can:\n",
+    "1. Retrieve similar questions using Hybrid Search (Vector + BM25) and Reranking\n",
+    "2. Process files using tools (PDF, XLSX, MP3, etc.)\n",
+    "3. Answer questions using web search, calculator, and other tools\n",
+    "\"\"\"\n",
+    "\n",
+    "import os\n",
+    "import bm25s\n",
+    "from dotenv import load_dotenv\n",
+    "\n",
+    "from langgraph.graph import START, END, StateGraph\n",
+    "from langgraph.prebuilt import tools_condition, ToolNode\n",
+    "\n",
+    "from sentence_transformers import SentenceTransformer, CrossEncoder\n",
+    "from langchain_huggingface import ChatHuggingFace, HuggingFaceEndpoint\n",
+    "from langchain_core.messages import HumanMessage\n",
+    "from supabase.client import Client, create_client\n",
+    "\n",
+    "from utils import load_config, load_prompt, init_bm25_index, reciprocal_rank_fusion\n",
+    "from tools import tools_list\n",
+    "from states import AgentState\n",
+    "\n",
+    "load_dotenv()\n",
+    "config = load_config()\n",
+    "\n",
+    "# Environment details and others\n",
+    "hf_key = os.getenv(\"HF_INFERENCE_KEY\")\n",
+    "supabase_url = os.getenv(\"SUPABASE_URL\")\n",
+    "supabase_key = os.getenv(\"SUPABASE_SERVICE_KEY\")\n",
+    "\n",
+    "# ============================================\n",
+    "# Model & Embeddings Setup\n",
+    "# ============================================\n",
+    "\n",
+    "# Model List\n",
+    "\n",
+    "llm_model_name = \"Qwen/Qwen3-32B-Instruct\"\n",
+    "\n",
+    "enable_keyword_search = config[\"retrievers\"][\"enable_keyword_search\"]\n",
+    "enable_vector_search = config[\"retrievers\"][\"enable_vector_search\"]\n",
+    "\n",
+    "# BM25 Retriever\n",
+    "bm25_retriever, bm25_corpus, bm25_ids = None, None, None\n",
+    "if enable_keyword_search:\n",
+    "    bm25_retriever, bm25_corpus, bm25_ids = init_bm25_index(corpus_file=config[\"data\"])\n",
+    "\n",
+    "embeddings, supabase = None, None\n",
+    "if enable_vector_search:\n",
+    "    # Embeddings for Vector Search\n",
+    "    embeddings = SentenceTransformer(model_name_or_path=config[\"models\"][\"embeddings\"][\"model_name\"], cache_folder=config[\"models\"][\"cache_folder\"])\n",
+    "\n",
+    "    # Supabase Vector Store\n",
+    "    supabase: Client = create_client(supabase_url, supabase_key)\n",
+    "\n",
+    "# Reranker Model (ModernBERT Cross-Encoder)\n",
+    "reranker = CrossEncoder(config[\"models\"][\"reranker\"][\"model_name\"], cache_folder=config[\"models\"][\"cache_folder\"])\n",
+    "\n",
+    "# LLM for Agent\n",
+    "llm = HuggingFaceEndpoint(\n",
+    "    repo_id=config[\"models\"][\"llm\"][\"model_name\"],\n",
+    "    temperature=config[\"models\"][\"llm\"][\"parameters\"][\"temperature\"],\n",
+    "    repetition_penalty=config[\"models\"][\"llm\"][\"parameters\"][\"repetition_penalty\"],\n",
+    "    provider=config[\"models\"][\"llm\"][\"parameters\"][\"provider\"],\n",
+    "    huggingfacehub_api_token=hf_key\n",
+    ")\n",
+    "\n",
+    "agent_llm = ChatHuggingFace(llm=llm)\n",
+    "agent_with_tools = agent_llm.bind_tools(tools_list)\n",
+    "\n",
+    "\n",
+    "# ============================================\n",
+    "# Graph Nodes\n",
+    "# ============================================\n",
+    "\n",
+    "def retriever_node(state: AgentState) -> AgentState:\n",
+    "    \"\"\"\n",
+    "    Hybrid Search Node: Retrieve docs via Vector Search + BM25, combine with RRF.\n",
+    "    \"\"\"\n",
+    "    print(\"--- RETRIEVER NODE ---\")\n",
+    "    messages = state.get(\"messages\", [])\n",
+    "    if not messages:\n",
+    "        return {\"retrieved_docs\": []}\n",
+    "    \n",
+    "    question_content = messages[0].content\n",
+    "    \n",
+    "    if not enable_vector_search and not enable_keyword_search:\n",
+    "        print(\"No retrieval method enabled.\")\n",
+    "        return {\"retrieved_docs\": []}\n",
+    "    \n",
+    "    # 1. Vector Search\n",
+    "    vector_docs = []\n",
+    "    if supabase and embeddings:\n",
+    "        try:\n",
+    "            response = supabase.rpc(\n",
+    "                config[\"retrievers\"][\"vector_store\"][\"query\"],\n",
+    "                {\"query_embedding\": embeddings.encode(question_content).tolist(), \n",
+    "                 \"match_count\": config[\"retrievers\"][\"vector_store\"][\"k\"], \n",
+    "                 \"match_threshold\": config[\"retrievers\"][\"vector_store\"][\"threshold\"]\n",
+    "                 }     \n",
+    "            ).execute()\n",
+    "\n",
+    "            vector_docs = response.data\n",
+    "\n",
+    "        except Exception as e:\n",
+    "            print(f\"Vector search error: {e}\")\n",
+    "        \n",
+    "    # 2. BM25 Search\n",
+    "    bm25_docs = []\n",
+    "    if bm25_retriever and bm25_corpus and bm25_ids:\n",
+    "        try:\n",
+    "            query_tokens = bm25s.tokenize([question_content], stopwords=\"en\")\n",
+    "            results, scores = bm25_retriever.retrieve(query_tokens, k=config[\"retrievers\"][\"bm25\"][\"k\"])\n",
+    "            indices = results[0]\n",
+    "            \n",
+    "            for i, idx in enumerate(indices):\n",
+    "                content = bm25_corpus[idx]\n",
+    "                task_id = bm25_ids[idx]\n",
+    "                score = scores[0][i]\n",
+    "                bm25_dict = {\"content\":content, \"metadata\": {\"source\": \"bm25_search\", \"task_id\": task_id, \"score\": score}}\n",
+    "                bm25_docs.append(bm25_dict)\n",
+    "        except Exception as e:\n",
+    "            print(f\"BM25 search error: {e}\")\n",
+    "\n",
+    "    # 3. RRF Fusion\n",
+    "    final_candidates = []\n",
+    "    if vector_docs and bm25_docs:\n",
+    "        fused = reciprocal_rank_fusion([vector_docs, bm25_docs])\n",
+    "        final_candidates = [id for id, doc, score in fused]\n",
+    "    else:\n",
+    "        final_candidates = vector_docs + bm25_docs\n",
+    "        final_candidates = [doc[\"metadata\"][\"task_id\"] for doc in final_candidates]\n",
+    "        \n",
+    "    top_candidates = final_candidates[:20]\n",
+    "    \n",
+    "    return {\"retrieved_docs\": top_candidates}"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "gaia",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.13.1"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

requirements.txt

@@ -44,6 +44,7 @@ tavily-python>=0.5.0
 wikipedia>=1.4.0
 arxiv>=2.1.0
 trafilatura>=1.6.0
+youtube-transcript-api>=1.0.0
 
 # Excel
 openpyxl>=3.1.0

scripts/create_vector_database.py

@@ -0,0 +1,52 @@
+"""
+Populate the Supabase vector store with GAIA benchmark embeddings.
+
+Prerequisites:
+  - SUPABASE_URL and SUPABASE_SERVICE_KEY in .env
+  - pgvector extension enabled in Supabase
+  - gaia_documents table created with columns: content (text), metadata (jsonb), embedding (vector)
+"""
+import os
+import json
+from dotenv import load_dotenv
+from supabase.client import Client, create_client
+from sentence_transformers import SentenceTransformer
+from gaia.utils import load_config
+
+load_dotenv()
+
+config = load_config()
+data_path = config["data"]
+
+supabase_url = os.getenv("SUPABASE_URL")
+supabase_key = os.getenv("SUPABASE_SERVICE_KEY")
+
+supabase: Client = create_client(supabase_url, supabase_key)
+embeddings = SentenceTransformer(
+    model_name_or_path=config["vector_store"]["embedding_model_name"],
+    cache_folder=config["models"]["cache_folder"],
+)
+
+with open(data_path, "r") as jsonl_file:
+    json_list = list(jsonl_file)
+
+documents = []
+for json_str in json_list:
+    json_data = json.loads(json_str)
+    content = json_data["Question"]
+    embedding = embeddings.encode(content, normalize_embeddings=True).tolist()
+    documents.append({
+        "content": content,
+        "metadata": {
+            "source": "vector_search",
+            "task_id": json_data["task_id"],
+        },
+        "embedding": embedding,
+    })
+
+print(f"Inserting {len(documents)} documents into Supabase...")
+try:
+    response = supabase.table("gaia_documents").insert(documents).execute()
+    print("Done.")
+except Exception as e:
+    print("Error inserting data into Supabase:", e)