Project page: https://huggingface.co/spaces/Anirudh1950/TITANS-Research-Agent

🧠 TITANS Agent

AI-Powered Multi-Source Research Agent with Long-Term Memory

An intelligent research agent that automatically collects, filters, and synthesizes information from multiple sources using semantic search, BERT-based quality filtering, and persistent memory via ChromaDB.

Python License


✨ Features

  • πŸ” Multi-Source Scraping - Parallel data collection from Reddit, Twitter, Medium, Google, and Perplexity
  • 🧠 Long-Term Memory - ChromaDB vector store for persistent knowledge retention
  • πŸ“Š BERT Quality Filter - Semantic re-ranking to filter low-quality content
  • πŸ’¬ Sentiment Analysis - Automatic sentiment scoring for collected data
  • πŸ“ Master Synthesis - AI-powered report generation with debate-style analysis
  • ⚑ Turbo Mode - Concurrent scraping for faster data collection

πŸ—οΈ Architecture

[ USER INPUT: "Topic" ]
          |
          v
+-----------------------------+
|  1. LONG-TERM MEMORY CHECK  |  <--- (ChromaDB Vector Store)
+-----------------------------+
          |
    (Found/Not Found)
          |
          v
+-----------------------------+
|    2. KNOWLEDGE PRIMING     |  <--- (Foundational KB Fetch)
+-----------------------------+
          |
          v
+-----------------------------+
|   3. SEMANTIC QUERY GEN     |  <--- (Perplexity LLM)
|  (Expand: "Topic" -> Q1-Q3) |
+-----------------------------+
          |
          v
+------------------------------------------------+
|           4. PARALLEL SCRAPING                 |
|  [Reddit]  [Twitter]  [Medium]  [Google] [Perplexity]
+------------------------------------------------+
          |
          v
+------------------------------+
|    5. BERT QUALITY FILTER    |  <--- (Sentence-Transformers)
|  (Re-rank & Discard Noise)   |
+------------------------------+
          |
          v
+------------------------------+
|   6. MEMORIZATION (TITANS)   |  ---> [ Store in ChromaDB ]
+------------------------------+
          |
          v
+------------------------------+
|     7. REPORT GENERATION     |
|      (Markdown Output)       |
+------------------------------+
          |
          V
     [ OUTPUT FILES ]

πŸ“¦ Installation

Prerequisites

  • Python 3.8+
  • API keys for data sources (see Configuration)

Setup

  1. Clone the repository

    git clone https://github.com/yourusername/scr-agent.git
    cd scr-agent
    
  2. Install dependencies

    pip install -r requirements.txt
    
  3. Configure environment variables

    Create a .env file in the root directory:

    PERPLEXITY_API_KEY=your_perplexity_api_key
    REDDIT_CLIENT_ID=your_reddit_client_id
    REDDIT_CLIENT_SECRET=your_reddit_client_secret
    TWITTER_BEARER_TOKEN=your_twitter_bearer_token
    

πŸš€ Usage

Run the agent with a topic:

python -m agent.main "Your research topic here"

Example

python -m agent.main "Comparison of traditional browsers vs AI browsers"

Output

Reports are saved to the output/ directory in Markdown format:

  • {id}_topic_perplexity.md - Initial Perplexity summary
  • {id}_topic_reddit.md - Reddit findings
  • {id}_topic_medium.md - Medium articles
  • {id}_topic_googlesearch.md - Google search results
  • {id}_topic_MASTER_REPORT.md - Synthesized master report

πŸ“ Project Structure

scr-agent/
β”œβ”€β”€ agent/
β”‚   β”œβ”€β”€ main.py              # Main entry point & pipeline orchestrator
β”‚   β”œβ”€β”€ scrapers/            # Data source scrapers
β”‚   β”‚   β”œβ”€β”€ reddit.py        # Reddit scraper
β”‚   β”‚   β”œβ”€β”€ twitter.py       # Twitter/X scraper
β”‚   β”‚   β”œβ”€β”€ medium.py        # Medium articles scraper
β”‚   β”‚   β”œβ”€β”€ google_search.py # Google search scraper
β”‚   β”‚   └── perplexity.py    # Perplexity AI scraper
β”‚   └── utils/               # Utility modules
β”‚       β”œβ”€β”€ memory.py        # ChromaDB memory manager
β”‚       β”œβ”€β”€ ranker.py        # BERT content ranker
β”‚       β”œβ”€β”€ query_generator.py  # Semantic query expansion
β”‚       β”œβ”€β”€ sentiment.py     # Sentiment analysis
β”‚       β”œβ”€β”€ synthesizer.py   # Report synthesis
β”‚       β”œβ”€β”€ extractor.py     # Deep content extraction
β”‚       β”œβ”€β”€ formatter.py     # Output formatting
β”‚       └── knowledge_integrator.py # Knowledge base integration
β”œβ”€β”€ chroma_db/               # Persistent vector storage
β”œβ”€β”€ output/                  # Generated reports
β”œβ”€β”€ requirements.txt         # Python dependencies
β”œβ”€β”€ .env                     # Environment variables (create this)
└── README.md

πŸ”§ Configuration

Environment Variables

Variable Description Required
PERPLEXITY_API_KEY API key for Perplexity AI Yes
REDDIT_CLIENT_ID Reddit API client ID No
REDDIT_CLIENT_SECRET Reddit API client secret No
TWITTER_BEARER_TOKEN Twitter/X API bearer token No

Note: The agent will gracefully handle missing API keys and skip unavailable sources.


πŸ§ͺ How It Works

  1. Memory Check - First checks ChromaDB for existing knowledge about the topic
  2. Knowledge Priming - If no memory exists, fetches foundational knowledge
  3. Query Expansion - Uses LLM to generate 3 semantically related queries
  4. Parallel Scraping - Concurrently scrapes all configured sources
  5. Quality Filtering - BERT-based re-ranking filters out low-quality content (threshold: 0.75)
  6. Sentiment Analysis - Scores content sentiment for each finding
  7. Memorization - Stores high-quality findings in ChromaDB for future use
  8. Master Synthesis - Generates a comprehensive report using "debate mode" analysis

πŸ“Š Dependencies

Package Purpose
requests HTTP requests for API calls
python-dotenv Environment variable management
beautifulsoup4 HTML parsing for web scraping
sentence-transformers BERT-based semantic similarity
chromadb Vector database for long-term memory
newspaper3k Article content extraction
lxml_html_clean HTML cleaning utilities

🀝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

  1. Fork the repository
  2. Create your feature branch (git checkout -b feature/AmazingFeature)
  3. Commit your changes (git commit -m 'Add some AmazingFeature')
  4. Push to the branch (git push origin feature/AmazingFeature)
  5. Open a Pull Request

πŸ“„ License

This project is licensed under the MIT License - see the LICENSE file for details.


πŸ™ Acknowledgments

  • Inspired by the TITANS memory architecture
  • Uses Sentence-Transformers for semantic understanding
  • ChromaDB for efficient vector storage

Made with ❀️ for intelligent research automation


Source code: https://github.com/rudzz1950/TITANS-Research-Agent-

Run locally

pip install -r requirements.txt
python app.py

app.py is a Gradio app; the original project scripts are unchanged alongside it.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Space using Anirudh1950/TITANS-Research-Agent 1