Project page: https://huggingface.co/spaces/Anirudh1950/TITANS-Research-Agent
π§ TITANS Agent
AI-Powered Multi-Source Research Agent with Long-Term Memory
An intelligent research agent that automatically collects, filters, and synthesizes information from multiple sources using semantic search, BERT-based quality filtering, and persistent memory via ChromaDB.
β¨ Features
- π Multi-Source Scraping - Parallel data collection from Reddit, Twitter, Medium, Google, and Perplexity
- π§ Long-Term Memory - ChromaDB vector store for persistent knowledge retention
- π BERT Quality Filter - Semantic re-ranking to filter low-quality content
- π¬ Sentiment Analysis - Automatic sentiment scoring for collected data
- π Master Synthesis - AI-powered report generation with debate-style analysis
- β‘ Turbo Mode - Concurrent scraping for faster data collection
ποΈ Architecture
[ USER INPUT: "Topic" ]
|
v
+-----------------------------+
| 1. LONG-TERM MEMORY CHECK | <--- (ChromaDB Vector Store)
+-----------------------------+
|
(Found/Not Found)
|
v
+-----------------------------+
| 2. KNOWLEDGE PRIMING | <--- (Foundational KB Fetch)
+-----------------------------+
|
v
+-----------------------------+
| 3. SEMANTIC QUERY GEN | <--- (Perplexity LLM)
| (Expand: "Topic" -> Q1-Q3) |
+-----------------------------+
|
v
+------------------------------------------------+
| 4. PARALLEL SCRAPING |
| [Reddit] [Twitter] [Medium] [Google] [Perplexity]
+------------------------------------------------+
|
v
+------------------------------+
| 5. BERT QUALITY FILTER | <--- (Sentence-Transformers)
| (Re-rank & Discard Noise) |
+------------------------------+
|
v
+------------------------------+
| 6. MEMORIZATION (TITANS) | ---> [ Store in ChromaDB ]
+------------------------------+
|
v
+------------------------------+
| 7. REPORT GENERATION |
| (Markdown Output) |
+------------------------------+
|
V
[ OUTPUT FILES ]
π¦ Installation
Prerequisites
- Python 3.8+
- API keys for data sources (see Configuration)
Setup
Clone the repository
git clone https://github.com/yourusername/scr-agent.git cd scr-agentInstall dependencies
pip install -r requirements.txtConfigure environment variables
Create a
.envfile in the root directory:PERPLEXITY_API_KEY=your_perplexity_api_key REDDIT_CLIENT_ID=your_reddit_client_id REDDIT_CLIENT_SECRET=your_reddit_client_secret TWITTER_BEARER_TOKEN=your_twitter_bearer_token
π Usage
Run the agent with a topic:
python -m agent.main "Your research topic here"
Example
python -m agent.main "Comparison of traditional browsers vs AI browsers"
Output
Reports are saved to the output/ directory in Markdown format:
{id}_topic_perplexity.md- Initial Perplexity summary{id}_topic_reddit.md- Reddit findings{id}_topic_medium.md- Medium articles{id}_topic_googlesearch.md- Google search results{id}_topic_MASTER_REPORT.md- Synthesized master report
π Project Structure
scr-agent/
βββ agent/
β βββ main.py # Main entry point & pipeline orchestrator
β βββ scrapers/ # Data source scrapers
β β βββ reddit.py # Reddit scraper
β β βββ twitter.py # Twitter/X scraper
β β βββ medium.py # Medium articles scraper
β β βββ google_search.py # Google search scraper
β β βββ perplexity.py # Perplexity AI scraper
β βββ utils/ # Utility modules
β βββ memory.py # ChromaDB memory manager
β βββ ranker.py # BERT content ranker
β βββ query_generator.py # Semantic query expansion
β βββ sentiment.py # Sentiment analysis
β βββ synthesizer.py # Report synthesis
β βββ extractor.py # Deep content extraction
β βββ formatter.py # Output formatting
β βββ knowledge_integrator.py # Knowledge base integration
βββ chroma_db/ # Persistent vector storage
βββ output/ # Generated reports
βββ requirements.txt # Python dependencies
βββ .env # Environment variables (create this)
βββ README.md
π§ Configuration
Environment Variables
| Variable | Description | Required |
|---|---|---|
PERPLEXITY_API_KEY |
API key for Perplexity AI | Yes |
REDDIT_CLIENT_ID |
Reddit API client ID | No |
REDDIT_CLIENT_SECRET |
Reddit API client secret | No |
TWITTER_BEARER_TOKEN |
Twitter/X API bearer token | No |
Note: The agent will gracefully handle missing API keys and skip unavailable sources.
π§ͺ How It Works
- Memory Check - First checks ChromaDB for existing knowledge about the topic
- Knowledge Priming - If no memory exists, fetches foundational knowledge
- Query Expansion - Uses LLM to generate 3 semantically related queries
- Parallel Scraping - Concurrently scrapes all configured sources
- Quality Filtering - BERT-based re-ranking filters out low-quality content (threshold: 0.75)
- Sentiment Analysis - Scores content sentiment for each finding
- Memorization - Stores high-quality findings in ChromaDB for future use
- Master Synthesis - Generates a comprehensive report using "debate mode" analysis
π Dependencies
| Package | Purpose |
|---|---|
requests |
HTTP requests for API calls |
python-dotenv |
Environment variable management |
beautifulsoup4 |
HTML parsing for web scraping |
sentence-transformers |
BERT-based semantic similarity |
chromadb |
Vector database for long-term memory |
newspaper3k |
Article content extraction |
lxml_html_clean |
HTML cleaning utilities |
π€ Contributing
Contributions are welcome! Please feel free to submit a Pull Request.
- Fork the repository
- Create your feature branch (
git checkout -b feature/AmazingFeature) - Commit your changes (
git commit -m 'Add some AmazingFeature') - Push to the branch (
git push origin feature/AmazingFeature) - Open a Pull Request
π License
This project is licensed under the MIT License - see the LICENSE file for details.
π Acknowledgments
- Inspired by the TITANS memory architecture
- Uses Sentence-Transformers for semantic understanding
- ChromaDB for efficient vector storage
Made with β€οΈ for intelligent research automation
Source code: https://github.com/rudzz1950/TITANS-Research-Agent-
Run locally
pip install -r requirements.txt
python app.py
app.py is a Gradio app; the original project scripts are unchanged alongside it.