- AI Agent Flow: Smart Content Automation System
- Overview
- User Review Required
- Architecture Components
- 1. Data Ingestion & Embedding Pipeline
- Hybrid Vector Database Architecture
- 1a. News Ingestion (RSS + Web)
- 1b. PDF/Document Reader (PyPDF2)
- 1c. Medium Paywall Bypass (Optional)
- 1d. On-Demand Mode (Paste Link โ Generate Post)
- 2. RAG Pipeline with Semantic Search
- 3. Intelligence Features
- 4. Multi-Agent Architecture (LangGraph)
- 5. MCP (Model Context Protocol) Integration
- 6. LLM Configuration & Response Handling
- 6. LLM Configuration & Response Handling
- 8. Observability with LangSmith
- 9. Memory Management: Forgetting & Aging Strategies
- 10. Summarization & Compression
- 11. Notion Integration (Post Storage)
- 12. Prompt Engineering Best Practices
- Proposed File Structure
- 12. Main Execution Loop
- 14. Testing with Pytest Fixtures
- Technology Stack
- RSS Sources (12 Feeds)
- Verification Plan
- Run locally
- Overview
Project page: https://huggingface.co/spaces/Anirudh1950/AI-Content-Generation-for-LinkedIn
AI Agent Flow: Smart Content Automation System
Overview
A sophisticated multi-agent system that automates personal brand content creation using cutting-edge AI technologies.
graph TB
subgraph "Data Ingestion Layer"
RSS[RSS Feeds] --> Fetcher[News Fetcher]
PDF[PDF/Image Upload] --> DocReader[Document Reader]
DocReader --> Fetcher
Fetcher --> Embedder[Sentence Transformer]
Embedder --> VectorDB[(ChromaDB Local)]
end
subgraph "Intelligence Layer"
VectorDB --> SemanticSearch[Semantic Search]
SemanticSearch --> RAG[RAG Pipeline]
RAG --> Classifier[Topic Classifier]
Classifier --> Trends[Trend Detector]
Trends --> Recommender[Recommendation Engine]
end
subgraph "Agent Layer (LangChain)"
Recommender --> Orchestrator[Orchestrator Agent]
Orchestrator --> ResearchAgent[Research Agent<br/>Perplexity]
Orchestrator --> WriterAgent[Writer Agent<br/>Gemini]
Orchestrator --> CriticAgent[Critic Agent<br/>Groq Llama]
Orchestrator --> ImageAgent[Image Agent<br/>Gemini Imagen]
WriterAgent --> CriticAgent
CriticAgent --> WriterAgent
end
subgraph "Output Layer"
CriticAgent --> Post[LinkedIn Post]
ImageAgent --> Image[Generated Image]
Post --> Email[Email Delivery]
Post --> Notion[Notion Database]
Image --> Email
Image --> Notion
end
subgraph "MCP Tools"
MCP[Model Context Protocol]
MCP --> WebSearch[Web Search]
MCP --> FileSystem[File System]
MCP --> Calendar[Scheduler]
end
Orchestrator <--> MCP
User Review Required
API Keys Required:
- Gemini API Key (Google GenAI) - for writing and image generation
- Perplexity API Key - for deep research
- Groq API Key - for Critic Agent (free tier)
- LangSmith API Key - for tracing and observability [!NOTE] Simplified Architecture: Using ChromaDB (Local) for all vector storage. No cloud DB sync required for this version.
Architecture Components
1. Data Ingestion & Embedding Pipeline
| Component | Technology | Purpose |
|---|---|---|
| RSS Fetcher | feedparser |
Collect articles from 12+ sources |
| Text Processor | Custom | Clean HTML, extract text |
| Chunking | langchain |
Split into semantic chunks |
| Embeddings | sentence-transformers |
Convert chunks to vectors |
| Hot Storage | ChromaDB (local) |
Fast RAG for recent articles (0-7 days) |
| Storage | ChromaDB (local) |
Fast RAG for recent articles (0-7 days) |
Hybrid Vector Database Architecture
subgraph "Single Vector Store (ChromaDB)"
New[New Articles] --> Chroma[(ChromaDB<br/>Local Persistence)]
Chroma --> RAG[Daily RAG Queries]
Chroma --> Trends[Trend Analysis]
end
| Layer | Technology | Data Age | Purpose | Storage Limit |
|---|---|---|---|---|
| Hot | ChromaDB | 0-90 days | Fast RAG & Trends | Local disk |
Why This Split Works:
- ChromaDB handles ~50-100 recent articles (fast, no latency)
- MongoDB stores ~500-1000 older articles (within 512MB free tier)
- Daily RAG only queries ChromaDB (fastest path)
- Historical/trend queries use MongoDB (when needed)
1a. News Ingestion (RSS + Web)
Purpose: Fetch latest AI news from RSS feeds and scrape article content for full context.
# pipelines/ingestion.py
import feedparser
import requests
from bs4 import BeautifulSoup
from datetime import datetime
import time
class NewsFetcher:
def __init__(self):
self.headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'}
# See RSS Sources section for feed URLs
self.feeds = [...]
def fetch_feeds(self) -> list:
"""Fetch and parse all RSS feeds."""
articles = []
for feed_url in self.feeds:
try:
feed = feedparser.parse(feed_url)
for entry in feed.entries[:5]: # Top 5 per feed
articles.append({
"title": entry.title,
"link": entry.link,
"published": entry.get("published", datetime.now().isoformat()),
"summary": entry.get("summary", ""),
"source": feed.feed.get("title", "Unknown")
})
except Exception as e:
print(f"Error fetching {feed_url}: {e}")
return articles
class WebScraper:
def __init__(self):
self.headers = {'User-Agent': 'Mozilla/5.0'}
def scrape(self, url: str) -> dict:
"""Extract main content from a webpage."""
try:
response = requests.get(url, headers=self.headers, timeout=10)
soup = BeautifulSoup(response.text, 'html.parser')
# Simple heuristic for main content
# 1. Look for <article> tag
article = soup.find('article')
if article:
text = article.get_text(separator="\n")
else:
# 2. Fallback to extracting all paragraphs
text = "\n".join([p.get_text() for p in soup.find_all('p')])
title = soup.find('h1').get_text() if soup.find('h1') else "Untitled"
return {
"title": title,
"content": text.strip(),
"source": "Web Scraper",
"url": url,
"date": datetime.now().isoformat()
}
except Exception as e:
print(f"Scrape error {url}: {e}")
return {"title": "Error", "content": "", "source": "Error", "url": url}
1b. PDF/Document Reader (PyPDF2)
Purpose: Process research papers, whitepapers, and text-based PDFs.
Use Cases:
- arXiv AI research papers
- Company whitepapers (OpenAI, Google, Anthropic)
- Standard text PDFs
graph LR
subgraph "Input Sources"
RSS[RSS Feeds] --> Pipeline
PDF[PDF Upload] --> Extractor[PyPDF2]
Extractor --> Pipeline[Ingestion Pipeline]
end
Implementation
# pipelines/document_reader.py
import PyPDF2
import io
import os
class DocumentReader:
def __init__(self):
pass
def read_pdf_local(self, pdf_path: str) -> str:
"""Extract text from text-based PDF (no OCR needed)."""
with open(pdf_path, 'rb') as file:
reader = PyPDF2.PdfReader(file)
text = ""
for page in reader.pages:
text += page.extract_text() + "\n"
return text
When to Use Each Method
| Method | When to Use | Cost |
|---|---|---|
read_pdf_local() |
Text-based PDFs (most articles) | Free |
read_pdf_with_ocr() |
Scanned PDFs, images in PDF | ~$1.50/1000 pages |
read_image() |
Screenshots, image articles | ~$1.50/1000 images |
Integration with Pipeline
# In main.py or ingestion.py
from pipelines.document_reader import DocumentReader
doc_reader = DocumentReader()
# Process uploaded PDF
if user_uploaded_pdf:
text = doc_reader.read_pdf_local(pdf_path)
if not text.strip(): # Scanned PDF, needs OCR
text = doc_reader.read_pdf_with_ocr(pdf_path)
article = {
"title": extract_title(text),
"content": text,
"source": "PDF Upload",
"date": datetime.now().isoformat()
}
# Add to ingestion pipeline
chunks = chunker.chunk_article(article)
# ... rest of pipeline
1c. Medium Paywall Bypass (Optional)
This feature bypasses Medium's paywall via Freedium mirror. Use for personal research only. Disabled by default.
# pipelines/medium_reader.py
import requests
from bs4 import BeautifulSoup
import os
class MediumReader:
def __init__(self):
self.enabled = os.getenv("ENABLE_MEDIUM_BYPASS", "false").lower() == "true"
self.mirror = "https://freedium-mirror.cfd"
def read_article(self, medium_url: str) -> dict:
"""Read Medium article via Freedium bypass."""
if not self.enabled:
"content": content,
"source": "Medium",
"url": medium_url
}
1d. On-Demand Mode (Paste Link โ Generate Post)
Purpose: Generate a LinkedIn post from any article URL you paste, with related content from your sources.
graph TB
subgraph "On-Demand Flow"
URL[Paste Article URL] --> Detect{Detect Source}
Detect -->|Medium| Medium[Medium Reader]
Detect -->|PDF| PDF[PDF Reader]
Detect -->|Web| Web[Web Scraper]
Medium --> Extract[Extract Content]
PDF --> Extract
Web --> Extract
Extract --> Topics[Classify Topics]
Topics --> Related[Find Related Articles]
Related --> Research[Deep Research]
Research --> Draft[Draft + Critic + Image]
Draft --> Output[Email + Notion]
end
Implementation
# pipelines/on_demand.py
from pipelines.document_reader import DocumentReader
from pipelines.medium_reader import MediumReader
from pipelines.ingestion import WebScraper
from intelligence.classifier import TopicClassifier
from storage.chroma_store import ChromaStore
import re
class OnDemandPipeline:
def __init__(self, orchestrator, vector_store):
self.doc_reader = DocumentReader()
self.medium_reader = MediumReader()
self.web_scraper = WebScraper()
self.classifier = TopicClassifier()
self.vector_store = vector_store
self.orchestrator = orchestrator
def process_url(self, url: str) -> dict:
"""Process any URL and generate a LinkedIn post."""
# Step 1: Detect source and read content
article = self._read_article(url)
print(f"Read article: {article['title']}")
# Step 2: Classify topic
article["topic"] = self.classifier.classify(article["content"])
print(f"Topic: {article['topic']}")
# Step 3: Find related articles from VectorDB
related = self.vector_store.similarity_search(
article["content"][:500], # Use first 500 chars as query
k=5
)
print(f"Found {len(related)} related articles")
# Step 4-7: Run through orchestrator (Research โ Draft โ Critic โ Image)
result = self.orchestrator.run(
articles=[article],
related_articles=related,
mode="on_demand"
)
return result
def _read_article(self, url: str) -> dict:
"""Route to appropriate reader based on URL."""
# Medium detection
if "medium.com" in url or self._is_medium_domain(url):
return self.medium_reader.read_article(url)
# PDF detection
if url.endswith(".pdf"):
return self._read_pdf_url(url)
# Generic web scraping
return self.web_scraper.scrape(url)
def _is_medium_domain(self, url: str) -> bool:
"""Check if URL is a Medium custom domain."""
# Could check for Medium meta tags
return False
def _read_pdf_url(self, url: str) -> dict:
"""Download and read PDF from URL."""
import tempfile
import requests
response = requests.get(url)
with tempfile.NamedTemporaryFile(suffix=".pdf", delete=False) as f:
f.write(response.content)
text = self.doc_reader.read_pdf_local(f.name)
return {
"title": url.split("/")[-1].replace(".pdf", ""),
"content": text,
"source": "PDF URL",
"url": url
}
CLI Usage
# main.py - Add on-demand mode
import argparse
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--url", help="Process single article URL")
parser.add_argument("--daily", action="store_true", help="Run daily pipeline")
args = parser.parse_args()
pipeline = DailyContentPipeline()
if args.url:
# On-Demand Mode
on_demand = OnDemandPipeline(pipeline.orchestrator, pipeline.vector_store)
result = on_demand.process_url(args.url)
print(f"Post generated! Notion: {result.get('notion_url')}")
elif args.daily or not args.url:
# Daily Mode (scheduled or immediate)
pipeline.run_daily_pipeline()
Example Commands
# On-demand: Generate post from specific article
python main.py --url "https://techcrunch.com/gpt-5-announcement"
# On-demand: Medium article (requires ENABLE_MEDIUM_BYPASS=true)
python main.py --url "https://medium.com/@author/paywalled-article"
# Daily mode: Run scheduled pipeline
python main.py --daily
Chunking Strategies:
| Strategy | Chunk Size | Overlap | Use Case |
|---|---|---|---|
| Fixed Size | 500 tokens | 50 tokens | Simple, fast |
| Semantic | Variable | Context-aware | Better coherence |
| Sentence | 3-5 sentences | 1 sentence | Preserves meaning |
| Recursive | 500 tokens | 100 tokens | Respects structure |
# pipelines/chunking.py
from langchain.text_splitter import RecursiveCharacterTextSplitter
class ArticleChunker:
def __init__(self):
self.splitter = RecursiveCharacterTextSplitter(
chunk_size=500,
chunk_overlap=100,
separators=["\n\n", "\n", ". ", " ", ""]
)
def chunk_article(self, article: dict) -> list:
"""Split article into indexed chunks."""
chunks = self.splitter.split_text(article["content"])
return [
{
"text": chunk,
"metadata": {
"source": article["source"],
"title": article["title"],
"date": article["date"],
"chunk_index": i,
"total_chunks": len(chunks)
}
}
for i, chunk in enumerate(chunks)
]
Flow:
RSS โ Fetch โ Clean โ Chunk โ Embed โ Index โ Store in ChromaDB
2. RAG Pipeline with Semantic Search
Purpose: When writing a post, retrieve the most relevant recent articles to provide context.
| Component | Technology | Purpose |
|---|---|---|
| Query Encoder | sentence-transformers |
Encode user query |
| Semantic Search | chromadb similarity |
Find top-k relevant docs |
| Context Builder | Custom | Assemble RAG context |
| LLM Generation | Gemini/Perplexity | Generate with context |
Embedding Model: all-MiniLM-L6-v2 (fast, good quality, 384 dims)
3. Intelligence Features
Topic Classification
- Model: Fine-tuned BERT or zero-shot with
facebook/bart-large-mnli - Categories: Research, Product Launch, Funding, Policy, Tutorial, Opinion
- Purpose: Tag articles for filtering and recommendation
Anomaly Detection
- Method: Statistical deviation in topic frequency + semantic outliers
- Purpose: Identify "breaking" or unusually trending topics
- Implementation: Track rolling 7-day topic distribution, flag spikes
Recommendation Engine
- Input: User preferences (topics, sources, past engagement)
- Output: Ranked list of articles to feature
- Algorithm: Hybrid (content-based + popularity weighting)
4. Multi-Agent Architecture (LangGraph)
stateDiagram-v2
[*] --> Research
Research --> Write
Write --> Critique
Critique --> CheckScore
state CheckScore <<choice>>
CheckScore --> Revise: Score < 8
CheckScore --> Approved: Score >= 8
Revise --> Critique: Loop (Max 3x)
Approved --> ImageGen
ImageGen --> Publish
Publish --> [*]
Agent Definitions
| Agent | LLM | Tools | Role |
|---|---|---|---|
| Orchestrator | Gemini 2.0 Flash | MCP Tools, DB Access | Coordinates workflow |
| Research Agent | Perplexity (sonar-pro) | Web Search | Deep dive on topics |
| Writer Agent | Gemini 2.0 Flash | RAG Context | Draft posts in user tone |
| Critic Agent | Groq Llama 3.3 70B | - | Review, rate, improve drafts |
| Image Agent | Gemini (Imagen) | - | Generate visuals |
Critic Agent (Council) Implementation
Why Groq?
- โ Free tier (no credit card required)
- โ Llama 3.3 70B (high quality for critique)
- โ Ultra-fast inference (~10x faster than others)
- โ Rate limits: 30 req/min, 6000 tokens/min (plenty for reviews)
# agents/critic_agent.py
from groq import Groq
import os
class CriticAgent:
def __init__(self):
self.client = Groq(api_key=os.getenv("GROQ_API_KEY"))
self.model = "llama-3.3-70b-versatile"
def review_post(self, draft: str, context: dict) -> dict:
"""Review a LinkedIn post and provide feedback."""
prompt = f'''You are a LinkedIn content critic for a tech founder.
**Draft Post**:
{draft}
**Topic Context**:
{context.get("topic", "AI News")}
**Review Criteria**:
1. Hook strength (first line grabs attention?)
2. Value delivery (clear insight for readers?)
3. Tone (professional but conversational?)
4. Length (150-200 words ideal)
5. Call-to-action (ends with engagement prompt?)
6. Hashtag relevance
**Output Format** (JSON):
{{
"score": 1-10,
"strengths": ["..."],
"improvements": ["..."],
"revised_post": "Your improved version here"
}}
'''
response = self.client.chat.completions.create(
model=self.model,
messages=[{"role": "user", "content": prompt}],
temperature=0.3,
max_tokens=1500
)
return self._parse_response(response.choices[0].message.content)
def _parse_response(self, content: str) -> dict:
import json
try:
return json.loads(content)
except:
return {"score": 7, "revised_post": content}
Review Loop in Orchestrator
# In orchestrator.py
def generate_post_with_review(self, context):
# Step 1: Generate draft
draft = self.writer.generate(context)
# Step 2: Critic review
review = self.critic.review_post(draft, context)
# Step 3: Accept or revise
if review["score"] >= 8:
return review.get("revised_post", draft)
else:
# Use critic's improved version
return review["revised_post"]
5. MCP (Model Context Protocol) Integration
MCP allows agents to use external tools in a standardized way.
Tools to Expose:
search_web: Query Perplexity or Googlefetch_rss: Get latest from a specific feedquery_vectordb: Semantic search over stored articlessend_email: Deliver the final contentschedule_task: Set future execution times
6. LLM Configuration & Response Handling
Review Loop in Orchestrator
# In orchestrator.py
def generate_post_with_review(self, context):
# Step 1: Generate draft
draft = self.writer.generate(context)
# Step 2: Critic review
review = self.critic.review_post(draft, context)
# Step 3: Accept or revise
if review["score"] >= 8:
return review.get("revised_post", draft)
else:
# Use critic's improved version
return review["revised_post"]
Response Modes
| Mode | Use Case | Implementation |
|---|---|---|
| Streamed | Real-time UI, long outputs | stream=True, yield chunks |
| Unstreamed | Batch processing, API calls | stream=False, await full response |
Generation Parameters
# config.py - LLM Parameter Presets
LLM_CONFIGS = {
"creative_writing": {
"temperature": 0.9, # Higher for creativity
"top_p": 0.95, # Nucleus sampling
"top_k": 50, # Limit token pool
"frequency_penalty": 0.3, # Reduce repetition
"max_tokens": 1500, # LinkedIn post length
},
"research": {
"temperature": 0.3, # Lower for accuracy
"top_p": 0.8,
"top_k": 40,
"frequency_penalty": 0.1,
"max_tokens": 2000,
},
"structured_output": {
"temperature": 0.1, # Very precise
"top_p": 0.5,
"top_k": 20,
"frequency_penalty": 0.0,
"max_tokens": 500,
}
}
| Parameter | Range | Effect |
|---|---|---|
temperature |
0.0 - 2.0 | Randomness in output (higher = more creative) |
top_p |
0.0 - 1.0 | Nucleus sampling threshold |
top_k |
1 - 100 | Limit vocabulary to top K tokens |
frequency_penalty |
-2.0 - 2.0 | Penalize repeated tokens |
max_tokens |
1 - 8192 | Maximum output length |
6. LLM Configuration & Response Handling
Orchestrator Implementation (LangGraph)
# agents/orchestrator.py
from langgraph.graph import StateGraph, END
from typing import TypedDict, List
class AgentState(TypedDict):
topic: str
context: dict
draft: str
review: dict
revision_count: int
class OrchestratorAgent:
def __init__(self, writer, critic, researcher):
self.writer = writer
self.critic = critic
self.researcher = researcher
self.workflow = self._build_graph()
def _build_graph(self):
workflow = StateGraph(AgentState)
# Define Nodes
workflow.add_node("research", self.research_node)
workflow.add_node("draft", self.draft_node)
workflow.add_node("critique", self.critique_node)
workflow.add_node("revise", self.revise_node)
# Define Edges
workflow.set_entry_point("research")
workflow.add_edge("research", "draft")
workflow.add_edge("draft", "critique")
# Conditional Edge: Critique -> (Revise OR End)
workflow.add_conditional_edges(
"critique",
self.check_score,
{
"revise": "revise",
"end": END
}
)
workflow.add_edge("revise", "critique") # Loop back
return workflow.compile()
def check_score(self, state: AgentState):
if state["review"]["score"] < 8 and state["revision_count"] < 3:
return "revise"
return "end"
Response Modes
graph LR
subgraph "ReAct Pattern"
R1[Reason] --> A1[Act/Tool]
A1 --> O1[Observe]
O1 --> R1
end
subgraph "Chain of Thought"
C1[Problem] --> C2[Step 1]
C2 --> C3[Step 2]
C3 --> C4[...]
C4 --> C5[Answer]
end
subgraph "Tree of Thoughts"
T1[Problem] --> T2A[Path A]
T1 --> T2B[Path B]
T1 --> T2C[Path C]
T2A --> T3[Evaluate]
T2B --> T3
T2C --> T3
T3 --> T4[Best Path]
end
ReAct (Reason + Act) - Primary Pattern
Used by all agents for tool-based reasoning:
Thought: I need to find the latest AI news about GPT-5 rumors.
Action: search_web("GPT-5 release date rumors January 2026")
Observation: Found 3 articles mentioning potential Q2 2026 release...
Thought: I should verify this with a second source.
Action: query_vectordb("GPT-5 OpenAI announcements")
Observation: Previous articles mentioned Sam Altman's hints about...
Thought: I have enough context to write about this.
Final Answer: [Synthesized response]
Chain of Thought (CoT) - For Complex Reasoning
Used by Writer Agent for structured post creation:
Let me think step by step:
1. First, identify the main insight from today's top article
2. Then, consider why this matters to my LinkedIn audience
3. Next, craft a hook that grabs attention
4. Finally, add a call-to-action
Tree of Thoughts (ToT) - For Creative Exploration
Used by Writer Agent when multiple angles are possible:
# Explore multiple post angles
thoughts = [
{"angle": "technical_deep_dive", "score": 0.7},
{"angle": "business_impact", "score": 0.9}, # Winner
{"angle": "personal_story", "score": 0.6},
]
# Select highest scoring path
Observation & Reflection Loop
sequenceDiagram
participant Agent
participant Tool
participant Memory
Agent->>Agent: Plan action
Agent->>Tool: Execute action
Tool-->>Agent: Return result
Agent->>Agent: Observe outcome
Agent->>Memory: Store observation
Agent->>Agent: Reflect on success/failure
Agent->>Agent: Adjust strategy if needed
8. Observability with LangSmith
Purpose: Trace, debug, and monitor all LLM calls and agent actions.
# config.py
import os
os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_API_KEY"] = "your-langsmith-api-key"
os.environ["LANGCHAIN_PROJECT"] = "weekly-article-flow"
What Gets Traced
| Component | Traced Data |
|---|---|
| LLM Calls | Input, output, tokens, latency, cost |
| Agent Steps | Reasoning, tool calls, observations |
| RAG Pipeline | Query, retrieved docs, relevance scores |
| Embeddings | Input text, vector dimensions |
Dashboard Features
- Run History: See every execution with full traces
- Latency Tracking: Identify slow steps
- Cost Analysis: Monitor API spend per agent
- Error Debugging: Inspect failed runs with full context
- Feedback Loop: Rate outputs for future fine-tuning
9. Memory Management: Forgetting & Aging Strategies
Problem: Vector databases grow indefinitely. Old news becomes irrelevant. We need smart memory management.
graph TB
subgraph "Memory Lifecycle"
New[New Article] --> Active["Active Memory (0-7 days)"]
Active --> Recent["Recent Memory (7-30 days)"]
Recent --> Archive["Archive (30-90 days)"]
Archive --> Forget["Forgotten / Deleted"]
end
Aging Strategies
| Strategy | Formula | Use Case |
|---|---|---|
| Linear Decay | relevance = 1 - (age_days / max_age) |
Gradual, predictable decay |
| Exponential Decay | relevance = e^(-ฮป * age_days) |
Fast initial decay, long tail |
| Step Function | if age > threshold: relevance = 0 |
Hard cutoff (news is binary) |
| Importance-Weighted | relevance = base_score * decay_factor |
Keep high-impact articles longer |
Implementation
# intelligence/memory_manager.py
from datetime import datetime, timedelta
import math
class MemoryManager:
def __init__(self, vector_store):
self.store = vector_store
self.config = {
"hot_window": 7, # Days in active memory
"warm_window": 30, # Days in recent memory
"cold_window": 90, # Days before deletion
"decay_lambda": 0.1, # Exponential decay rate
}
def calculate_relevance(self, article_date: datetime) -> float:
"""Exponential decay based on article age."""
age_days = (datetime.now() - article_date).days
return math.exp(-self.config["decay_lambda"] * age_days)
def apply_forgetting(self):
"""Remove articles past the cold window."""
cutoff = datetime.now() - timedelta(days=self.config["cold_window"])
self.store.delete(where={"date": {"$lt": cutoff.isoformat()}})
def weighted_search(self, query: str, k: int = 10):
"""Search with time-decayed relevance scoring."""
results = self.store.similarity_search(query, k=k*2)
for doc in results:
age_score = self.calculate_relevance(doc.metadata["date"])
doc.score = doc.similarity * 0.6 + age_score * 0.4
return sorted(results, key=lambda x: x.score, reverse=True)[:k]
Forgetting Policies
| Policy | Trigger | Action |
|---|---|---|
| Time-Based | Article > 90 days old | Delete from vector DB |
| Capacity-Based | DB size > 10,000 docs | Remove lowest-scored 20% |
| Relevance-Based | Score < 0.1 threshold | Archive to cold storage |
| Duplicate Detection | >0.95 similarity | Keep only most recent |
10. Summarization & Compression
Problem: Long articles waste tokens. Context windows are limited. We need efficient compression.
graph LR
Full[Full Article<br/>2000 tokens] --> Extract[Extractive<br/>Summary]
Extract --> Abstract[Abstractive<br/>Summary]
Abstract --> Compressed[Compressed<br/>200 tokens]
subgraph "Compression Ratio"
CR["10:1 compression"]
end
Summarization Levels
| Level | Tokens | Use Case | Method |
|---|---|---|---|
| Full | 2000+ | Deep research, archival | None |
| Detailed | 500-800 | RAG context | Extractive |
| Concise | 150-300 | Multiple article comparison | Abstractive |
| Headline | 20-50 | Quick scan, recommendations | Key phrase extraction |
Implementation
# intelligence/summarizer.py
from transformers import pipeline
from langchain_google_genai import ChatGoogleGenerativeAI
class ArticleSummarizer:
def __init__(self):
# Fast extractive summarization (local)
self.extractive = pipeline(
"summarization",
model="facebook/bart-large-cnn",
max_length=300
)
# High-quality abstractive (API)
self.abstractive = ChatGoogleGenerativeAI(
model="gemini-2.0-flash",
temperature=0.3
)
def extract_summary(self, text: str, max_tokens: int = 300) -> str:
"""Fast local extractive summarization."""
return self.extractive(text, max_length=max_tokens)[0]["summary_text"]
def abstract_summary(self, text: str, style: str = "concise") -> str:
"""LLM-powered abstractive summarization."""
prompts = {
"concise": "Summarize in 2-3 sentences, focusing on key facts:",
"bullet": "Extract 3-5 key bullet points:",
"headline": "Write a single compelling headline:",
}
return self.abstractive.invoke(f"{prompts[style]}\n\n{text}").content
def compress_for_rag(self, articles: list) -> str:
"""Compress multiple articles into RAG context."""
summaries = []
for article in articles:
summary = self.extract_summary(article["content"], max_tokens=150)
summaries.append(f"[{article['source']}] {summary}")
return "\n\n".join(summaries)
Compression Strategies
| Strategy | When to Use | Savings |
|---|---|---|
| Extractive | Preserving exact quotes/facts | 60-70% |
| Abstractive | Capturing essence, paraphrasing | 80-90% |
| Hierarchical | Very long docs (chunk โ summarize โ merge) | 90%+ |
| Semantic Chunking | Split on meaning, not tokens | Variable |
Context Window Optimization
# When building RAG context:
def build_optimized_context(articles: list, max_tokens: int = 4000):
"""Fit maximum info in limited context window."""
# Sort by relevance
articles = sorted(articles, key=lambda x: x["score"], reverse=True)
context_parts = []
token_count = 0
for article in articles:
# Progressively compress as we fill the window
if token_count < max_tokens * 0.5:
summary = summarizer.extract_summary(article, max_tokens=300)
elif token_count < max_tokens * 0.8:
summary = summarizer.abstract_summary(article, style="concise")
else:
summary = summarizer.abstract_summary(article, style="headline")
context_parts.append(summary)
token_count += len(summary.split()) * 1.3 # Rough token estimate
if token_count >= max_tokens:
break
return "\n\n".join(context_parts)
11. Notion Integration (Post Storage)
Purpose: Automatically save generated LinkedIn posts to your Notion database for archival and review.
Your Notion Database: LinkedIn-Posts-2ecb15cc6736806e9dd7f174b4a1dbe0
graph LR
Post[Generated Post] --> Notion[Notion API]
Notion --> Page[New Page in Database]
Page --> Props[Properties: Date, Topic, Status]
Page --> Content[Post Content + Image]
Setup Steps
- Create Notion Integration: Go to notion.so/my-integrations
- Get API Token: Copy the "Internal Integration Secret"
- Share Database: Open your LinkedIn Posts database โ Share โ Add your integration
Implementation
# utils/notion_client.py
from notion_client import Client
from datetime import datetime
import os
class NotionPostSaver:
def __init__(self):
self.client = Client(auth=os.getenv("NOTION_API_TOKEN"))
self.database_id = os.getenv("NOTION_DATABASE_ID")
def save_post(
self,
post_content: str,
topic: str,
sources: list[str],
image_url: str = None
) -> str:
"""Create a new page in the LinkedIn Posts database."""
# Build page properties
properties = {
"Title": {
"title": [{"text": {"content": f"AI Digest - {datetime.now().strftime('%B %d, %Y')}"}}]
},
"Date": {
"date": {"start": datetime.now().isoformat()}
},
"Topic": {
"select": {"name": topic}
},
"Status": {
"select": {"name": "Draft"}
},
"Sources": {
"multi_select": [{"name": src[:100]} for src in sources[:5]]
}
}
# Build page content (blocks)
children = [
{
"object": "block",
"type": "heading_2",
"heading_2": {"rich_text": [{"text": {"content": "LinkedIn Post"}}]}
},
{
"object": "block",
"type": "paragraph",
"paragraph": {"rich_text": [{"text": {"content": post_content}}]}
}
]
# Add image if available
if image_url:
children.append({
"object": "block",
"type": "image",
"image": {"type": "external", "external": {"url": image_url}}
})
# Create the page
response = self.client.pages.create(
parent={"database_id": self.database_id},
properties=properties,
children=children
)
return response["url"]
Database Schema (Recommended)
| Property | Type | Purpose |
|---|---|---|
| Title | Title | Post headline/date |
| Date | Date | When generated |
| Topic | Select | Main AI topic covered |
| Status | Select | Draft / Published / Archived |
| Sources | Multi-select | News sources used |
| Engagement | Number | Track LinkedIn metrics (manual) |
12. Prompt Engineering Best Practices
Purpose: Craft high-quality prompts that produce consistent, accurate outputs.
graph TB
subgraph "Prompt Engineering Workflow"
A[Be Specific] --> B[Add Context]
B --> C[Use Technical Terms]
C --> D[Include Examples]
D --> E[Iterate & Test]
E --> F[Specify Format]
end
Core Principles
| Principle | Description | Example |
|---|---|---|
| Be Specific | Clearly state what you want | "Write a LinkedIn post" โ "Write a 150-word LinkedIn post about GPT-5 for tech founders" |
| Add Context | Provide background info | Include article summaries, target audience, tone |
| Technical Terms | Use domain vocabulary | "AI", "LLM", "inference", "fine-tuning" |
| Use Examples | Show desired output format | Few-shot prompting with sample posts |
| Iterate & Test | Refine based on results | A/B test different prompt versions |
| Specify Format | Define structure, length | "Use bullet points, max 200 words, include emoji" |
Prompt Templates
# agents/prompts.py
PROMPT_TEMPLATES = {
"linkedin_post": '''
You are a LinkedIn content creator for a tech founder.
**Context**:
{rag_context}
**Task**: Write a LinkedIn post about the most interesting AI development from today's news.
**Requirements**:
- Hook in the first line (question or bold statement)
- 150-200 words maximum
- Professional but conversational tone
- Include 1-2 relevant hashtags
- End with a call-to-action or question
**Tone Reference**:
{user_tone_examples}
**Output Format**:
[Hook]
[Main insight - 2-3 paragraphs]
[Call-to-action]
#Hashtags
''',
"research_query": '''
You are a research assistant finding the latest information.
**Topic**: {topic}
**Time Range**: Last 24 hours
**Focus Areas**: {focus_areas}
**Task**: Provide a comprehensive summary including:
1. Key developments
2. Major players involved
3. Potential implications
4. Credible sources
Be factual and cite specific details.
''',
"summarize_article": '''
Summarize this article for a busy tech executive.
**Article**:
{article_content}
**Output**:
- 2-3 sentence summary
- Key takeaway
- Relevance to AI industry (1-10 score)
'''
}
Dynamic Prompt Construction
# agents/prompt_builder.py
class PromptBuilder:
def __init__(self):
self.templates = PROMPT_TEMPLATES
def build_linkedin_prompt(
self,
rag_context: str,
tone_examples: list,
constraints: dict = None
) -> str:
"""Build a context-aware LinkedIn post prompt."""
# Format tone examples
tone_str = "\n---\n".join(tone_examples[:3])
# Apply constraints
constraints = constraints or {}
length = constraints.get("max_words", 200)
style = constraints.get("style", "professional")
prompt = self.templates["linkedin_post"].format(
rag_context=rag_context,
user_tone_examples=tone_str
)
# Add dynamic constraints
prompt += f"\n**Additional Constraints**: Max {length} words, {style} style."
return prompt
def few_shot_examples(self, examples: list) -> str:
"""Format examples for few-shot prompting."""
formatted = []
for i, ex in enumerate(examples, 1):
formatted.append(f"Example {i}:\nInput: {ex['input']}\nOutput: {ex['output']}")
return "\n\n".join(formatted)
Prompt Testing Framework
# tests/test_prompts.py
class PromptTester:
def __init__(self, llm):
self.llm = llm
self.results = []
def test_prompt_variants(self, variants: list, test_input: str):
"""A/B test different prompt versions."""
for variant in variants:
response = self.llm.invoke(variant.format(input=test_input))
self.results.append({
"variant": variant[:50],
"output": response.content,
"tokens": response.usage.total_tokens,
"latency": response.response_time
})
return self.results
def evaluate_output(self, output: str, criteria: dict) -> dict:
"""Score output against criteria."""
scores = {}
scores["length_ok"] = len(output.split()) <= criteria.get("max_words", 200)
scores["has_hook"] = output.split("\n")[0].endswith("?") or len(output.split("\n")[0]) < 100
scores["has_hashtags"] = "#" in output
return scores
Proposed File Structure
Weekly Article Flow/
โโโ .env # API keys
โโโ requirements.txt # Dependencies
โโโ config.py # Configuration + LLM presets
โโโ main.py # Entry point + daily loop
โโโ agents/
โ โโโ __init__.py
โ โโโ orchestrator.py # Main coordinator (ReAct)
โ โโโ research_agent.py # Perplexity-powered research
โ โโโ writer_agent.py # Gemini-powered writing (CoT/ToT)
โ โโโ critic_agent.py # Groq Llama review & improve
โ โโโ prompts.py # Prompt templates
โ โโโ prompt_builder.py # Dynamic prompt construction
โโโ pipelines/
โ โโโ __init__.py
โ โโโ ingestion.py # RSS fetching + web scraping
โ โโโ document_reader.py # PDF extraction (PyPDF2)
โ โโโ medium_reader.py # Medium paywall bypass (optional)
โ โโโ on_demand.py # On-demand URL processing
โ โโโ chunking.py # Text chunking strategies
โ โโโ rag.py # RAG pipeline + context building
โ โโโ recommendation.py # Article ranking
โโโ intelligence/
โ โโโ __init__.py
โ โโโ classifier.py # Topic classification (zero-shot)
โ โโโ trends.py # Simple trend detection
โ โโโ embeddings.py # Sentence transformers
โ โโโ summarizer.py # Extractive/Abstractive
โ โโโ memory_manager.py # Forgetting & aging
โโโ storage/
โ โโโ __init__.py
โ โโโ chroma_store.py # ChromaDB wrapper (Local persistence)
โ โโโ chroma_data/ # Persisted local vectors
โโโ mcp/
โ โโโ __init__.py
โ โโโ tools.py # MCP tool definitions
โโโ utils/
โ โโโ __init__.py
โ โโโ emailer.py # Email delivery
โ โโโ notion_client.py # Notion API integration
โ โโโ retry.py # Retry/backoff logic
โโโ tests/
โโโ __init__.py
โโโ conftest.py # Pytest fixtures
โโโ test_ingestion.py
โโโ test_agents.py
โโโ test_prompts.py
โโโ test_rag.py
12. Main Execution Loop
Purpose: The complete daily workflow as a single orchestrated loop.
graph TB
subgraph "Daily Loop (6 AM)"
Start([Start]) --> Ingest[1. Ingest News + PDFs]
Ingest --> Embed[2. Chunk & Embed]
Embed --> Store[3. Store in VectorDB]
Store --> Classify[4. Classify Topics]
Classify --> Detect[5. Detect Anomalies]
Detect --> Recommend[6. Get Recommendations]
Recommend --> Research[7. Deep Research]
Research --> Draft[8. Draft Post]
Draft --> Critic[9. Critic Review]
Critic -->|Score < 8| Draft
Critic -->|Score >= 8| Image[10. Generate Image]
Image --> Send[11. Send Email + Notion]
Send --> Cleanup[12. Memory Cleanup]
Cleanup --> End([End])
end
subgraph "Feedback Loop"
Send --> Feedback[User Feedback]
Feedback --> Preferences[Update Preferences]
Preferences --> Recommend
end
12. Main Execution Loop
Purpose: The complete daily workflow as a single orchestrated loop.
CLI Implementation
# main.py
import argparse
import schedule
import time
from datetime import datetime
from dotenv import load_dotenv
from pipelines.ingestion import NewsFetcher
from pipelines.chunking import ArticleChunker
from pipelines.rag import RAGPipeline
from pipelines.recommendation import RecommendationEngine
from intelligence.classifier import TopicClassifier
from intelligence.trends import TrendDetector
from intelligence.embeddings import EmbeddingService
from intelligence.memory_manager import MemoryManager
from intelligence.summarizer import ArticleSummarizer
from storage.chroma_store import ChromaStore
from agents.orchestrator import OrchestratorAgent
from agents.critic_agent import CriticAgent
from pipelines.on_demand import OnDemandPipeline
from utils.notion_client import NotionPostSaver
from utils.retry import with_retry
load_dotenv()
class DailyContentPipeline:
def __init__(self):
# Initialize all components
self.chroma_store = ChromaStore()
self.embedder = EmbeddingService()
self.chunker = ArticleChunker()
self.classifier = TopicClassifier()
self.trend_detector = TrendDetector()
self.recommender = RecommendationEngine()
self.memory_manager = MemoryManager(self.chroma_store)
self.summarizer = ArticleSummarizer()
self.rag = RAGPipeline(self.chroma_store, self.summarizer)
self.critic = CriticAgent()
self.orchestrator = OrchestratorAgent(critic=self.critic)
self.notion = NotionPostSaver()
self.fetcher = NewsFetcher()
@with_retry(max_attempts=3, backoff=2.0)
def run_daily_pipeline(self):
"""Execute the complete daily content generation pipeline."""
print(f"[{datetime.now()}] Starting daily pipeline...")
# Phase 1: Ingestion
articles = self.fetcher.fetch_feeds()
# Phase 2: Store & Classify
# ... (Implementation similar to previous steps) ...
# Phase 3: Generate
# ... (Orchestrator run) ...
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--url", help="Process single article URL")
parser.add_argument("--daily", action="store_true", help="Run daily pipeline")
args = parser.parse_args()
pipeline = DailyContentPipeline()
if args.url:
# On-Demand Mode
on_demand = OnDemandPipeline(pipeline.orchestrator, pipeline.chroma_store)
result = on_demand.process_url(args.url)
print(f"Post generated! Notion: {result.get('notion_url')}")
elif args.daily or not args.url:
# Daily Mode
pipeline.run_daily_pipeline()
if __name__ == "__main__":
main()
from storage.chroma_store import ChromaStore from storage.mongo_store import MongoStore from agents.orchestrator import OrchestratorAgent from agents.critic_agent import CriticAgent from utils.emailer import send_email from utils.notion_client import NotionPostSaver from utils.retry import with_retry from config import CONFIG
load_dotenv()
class DailyContentPipeline:
def init(self):
# Initialize all components
self.chroma_store = ChromaStore()
self.embedder = EmbeddingService()
self.chunker = ArticleChunker()
self.classifier = TopicClassifier()
self.trend_detector = TrendDetector()
self.recommender = RecommendationEngine()
self.memory_manager = MemoryManager(self.chroma_store)
self.summarizer = ArticleSummarizer()
self.rag = RAGPipeline(self.chroma_store, self.summarizer)
self.critic = CriticAgent()
self.orchestrator = OrchestratorAgent(critic=self.critic)
self.notion = NotionPostSaver()
self.fetcher = NewsFetcher()
@with_retry(max_attempts=3, backoff=2.0)
def run_daily_pipeline(self):
"""Execute the complete daily content generation pipeline."""
print(f"[{datetime.now()}] Starting daily pipeline...")
# Phase 1: Data Ingestion
print("Phase 1: Ingesting news...")
articles = self.fetcher.fetch_all_feeds()
print(f" Fetched {len(articles)} articles")
# Phase 2: Chunk & Embed
print("Phase 2: Chunking and embedding...")
all_chunks = []
for article in articles:
chunks = self.chunker.chunk_article(article)
all_chunks.extend(chunks)
embeddings = self.embedder.embed_batch([c["text"] for c in all_chunks])
# Phase 3: Store
print("Phase 3: Storing in VectorDB...")
self.vector_store.add_documents(all_chunks, embeddings)
# Phase 4: Classify
print("Phase 4: Classifying topics...")
for article in articles:
article["topic"] = self.classifier.classify(article["content"])
# Phase 5: Anomaly Detection
print("Phase 5: Detecting anomalies...")
trending_topics = self.anomaly_detector.detect(articles)
print(f" Trending: {trending_topics}")
# Phase 6: Recommendations
print("Phase 6: Getting recommendations...")
top_articles = self.recommender.rank(
articles,
trending=trending_topics,
user_prefs=CONFIG.get("user_preferences", {})
)
# Phase 7-9: Agent Orchestration
print("Phase 7-9: Running agent orchestration...")
result = self.orchestrator.run(
articles=top_articles[:3],
rag_pipeline=self.rag,
summarizer=self.summarizer
)
# Phase 10: Quality Check
print("Phase 10: Quality check...")
if not self._quality_check(result):
print(" Quality check failed, regenerating...")
result = self.orchestrator.run(
articles=top_articles[:3],
rag_pipeline=self.rag,
constraints={"retry": True}
)
# Phase 11: Send Email
print("Phase 11: Sending email...")
send_email(
subject=f"Your AI News Digest - {datetime.now().strftime('%B %d')}",
body=result["post"],
image_path=result.get("image_path"),
to_email=CONFIG["email_recipient"]
)
# Phase 12: Cleanup
print("Phase 12: Memory cleanup...")
self.memory_manager.apply_forgetting()
print(f"[{datetime.now()}] Pipeline complete!")
return result
def _quality_check(self, result: dict) -> bool:
"""Validate output quality."""
post = result.get("post", "")
checks = [
len(post.split()) >= 100, # Minimum length
len(post.split()) <= 250, # Maximum length
"#" in post, # Has hashtags
not post.startswith("I"), # Doesn't start with "I"
]
return all(checks)
def main(): pipeline = DailyContentPipeline()
# Schedule daily run at 6 AM
schedule.every().day.at("06:00").do(pipeline.run_daily_pipeline)
# Also run memory cleanup at 2 AM
schedule.every().day.at("02:00").do(
pipeline.memory_manager.apply_forgetting
)
print("Scheduler started. Waiting for scheduled runs...")
print("Press Ctrl+C to exit.")
# For testing, run immediately
if CONFIG.get("run_immediately", False):
pipeline.run_daily_pipeline()
while True:
schedule.run_pending()
time.sleep(60)
if name == "main": main()
---
## 13. Error Handling & Retry Logic
```python
# utils/retry.py
import time
import functools
from typing import Callable, TypeVar
T = TypeVar('T')
def with_retry(
max_attempts: int = 3,
backoff: float = 2.0,
exceptions: tuple = (Exception,)
) -> Callable:
"""Decorator for retry with exponential backoff."""
def decorator(func: Callable[..., T]) -> Callable[..., T]:
@functools.wraps(func)
def wrapper(*args, **kwargs) -> T:
last_exception = None
for attempt in range(max_attempts):
try:
return func(*args, **kwargs)
except exceptions as e:
last_exception = e
wait_time = backoff ** attempt
print(f"Attempt {attempt + 1} failed: {e}. Retrying in {wait_time}s...")
time.sleep(wait_time)
raise last_exception
return wrapper
return decorator
class CircuitBreaker:
"""Circuit breaker for external API calls."""
def __init__(self, failure_threshold: int = 5, reset_timeout: int = 60):
self.failure_count = 0
self.failure_threshold = failure_threshold
self.reset_timeout = reset_timeout
self.last_failure_time = None
self.state = "CLOSED" # CLOSED, OPEN, HALF_OPEN
def call(self, func: Callable, *args, **kwargs):
if self.state == "OPEN":
if time.time() - self.last_failure_time > self.reset_timeout:
self.state = "HALF_OPEN"
else:
raise Exception("Circuit breaker is OPEN")
try:
result = func(*args, **kwargs)
self._on_success()
return result
except Exception as e:
self._on_failure()
raise e
def _on_success(self):
self.failure_count = 0
self.state = "CLOSED"
def _on_failure(self):
self.failure_count += 1
self.last_failure_time = time.time()
if self.failure_count >= self.failure_threshold:
self.state = "OPEN"
14. Testing with Pytest Fixtures
# tests/conftest.py
import pytest
from unittest.mock import Mock, patch
from datetime import datetime, timedelta
# ============ Sample Data Fixtures ============
@pytest.fixture
def sample_article():
"""A single sample article for testing."""
return {
"title": "OpenAI Announces GPT-5 Preview",
"content": "OpenAI has announced a preview of GPT-5, their next-generation language model. The new model shows significant improvements in reasoning and code generation. Sam Altman stated that GPT-5 represents a major leap forward in AI capabilities.",
"source": "TechCrunch",
"url": "https://techcrunch.com/2026/01/20/openai-gpt-5",
"date": datetime.now().isoformat(),
"topic": None
}
@pytest.fixture
def sample_articles(sample_article):
"""Multiple sample articles for batch testing."""
articles = [sample_article]
articles.append({
"title": "Google Releases Gemini 3.0",
"content": "Google has released Gemini 3.0, featuring improved multimodal capabilities and faster inference times.",
"source": "The Verge",
"url": "https://theverge.com/2026/01/20/gemini-3",
"date": (datetime.now() - timedelta(hours=2)).isoformat(),
"topic": None
})
return articles
@pytest.fixture
def sample_chunks(sample_article):
"""Pre-chunked article data."""
return [
{"text": "OpenAI has announced a preview of GPT-5.", "metadata": {"source": "TechCrunch", "chunk_index": 0}},
{"text": "The new model shows significant improvements.", "metadata": {"source": "TechCrunch", "chunk_index": 1}},
]
# ============ Mock Service Fixtures ============
@pytest.fixture
def mock_vector_store():
"""Mock ChromaDB vector store."""
store = Mock()
store.add_documents = Mock(return_value=None)
store.similarity_search = Mock(return_value=[
Mock(page_content="Sample content", metadata={"source": "Test"}, score=0.95)
])
store.delete = Mock(return_value=None)
return store
@pytest.fixture
def mock_embedding_service():
"""Mock sentence transformer embeddings."""
service = Mock()
service.embed = Mock(return_value=[0.1] * 384)
service.embed_batch = Mock(return_value=[[0.1] * 384, [0.2] * 384])
return service
@pytest.fixture
def mock_llm():
"""Mock LLM for testing agents."""
llm = Mock()
llm.invoke = Mock(return_value=Mock(
content="This is a test LinkedIn post about AI. #AI #Tech",
usage=Mock(total_tokens=100),
response_time=0.5
))
return llm
@pytest.fixture
def mock_perplexity():
"""Mock Perplexity API for research agent."""
with patch('agents.research_agent.PerplexityClient') as mock:
mock.return_value.search = Mock(return_value={
"answer": "GPT-5 is expected to be released in Q2 2026.",
"sources": ["techcrunch.com", "theverge.com"]
})
yield mock
@pytest.fixture
def mock_groq():
"""Mock Groq API for critic agent."""
with patch('agents.critic_agent.Groq') as mock:
mock.return_value.chat.completions.create = Mock(return_value=Mock(
choices=[Mock(message=Mock(content='''{
"score": 8,
"strengths": ["Strong hook", "Clear value"],
"improvements": ["Add more data points"],
"revised_post": "Improved test LinkedIn post about AI. #AI #Tech"
}'''))]
))
yield mock
# ============ Component Fixtures ============
@pytest.fixture
def chunker():
"""Real ArticleChunker instance."""
from pipelines.chunking import ArticleChunker
return ArticleChunker()
@pytest.fixture
def classifier():
"""Real TopicClassifier (may need model download)."""
from intelligence.classifier import TopicClassifier
return TopicClassifier()
@pytest.fixture
def memory_manager(mock_vector_store):
"""MemoryManager with mocked stores."""
from intelligence.memory_manager import MemoryManager
mock_mongo = Mock()
mock_mongo.add_documents = Mock(return_value=None)
return MemoryManager(mock_vector_store, mock_mongo)
@pytest.fixture
def mock_notion():
"""Mock Notion API for post saving."""
with patch('utils.notion_client.Client') as mock:
mock.return_value.pages.create = Mock(return_value={
"url": "https://notion.so/test-page-123",
"id": "test-page-123"
})
yield mock
@pytest.fixture
def mock_mongo_store():
"""Mock MongoDB Atlas store."""
store = Mock()
store.add_documents = Mock(return_value=None)
store.similarity_search = Mock(return_value=[])
store.delete = Mock(return_value=None)
store.migrate_from_chroma = Mock(return_value=10)
return store
@pytest.fixture
def mock_medium_reader():
"""Mock Medium reader for paywall bypass."""
with patch('pipelines.medium_reader.requests.get') as mock:
mock.return_value.text = '<h1>Test Article</h1><article>Content here</article>'
yield mock
@pytest.fixture
def on_demand_pipeline(mock_vector_store, mock_llm, mock_groq, mock_notion):
"""Fully mocked on-demand pipeline for testing URL processing."""
from pipelines.on_demand import OnDemandPipeline
from agents.orchestrator import OrchestratorAgent
orchestrator = Mock()
orchestrator.run = Mock(return_value={
"post": "Generated LinkedIn post",
"image_url": "https://example.com/image.png",
"notion_url": "https://notion.so/test"
})
pipeline = OnDemandPipeline(orchestrator, mock_vector_store)
return pipeline
# ============ Integration Fixtures ============
@pytest.fixture
def pipeline(mock_vector_store, mock_embedding_service, mock_llm):
"""Fully mocked pipeline for integration testing."""
from main import DailyContentPipeline
pipeline = DailyContentPipeline()
pipeline.vector_store = mock_vector_store
pipeline.embedder = mock_embedding_service
pipeline.orchestrator.llm = mock_llm
return pipeline
Example Tests
# tests/test_ingestion.py
import pytest
class TestChunking:
def test_chunk_article_returns_list(self, chunker, sample_article):
chunks = chunker.chunk_article(sample_article)
assert isinstance(chunks, list)
assert len(chunks) > 0
def test_chunks_have_metadata(self, chunker, sample_article):
chunks = chunker.chunk_article(sample_article)
for chunk in chunks:
assert "metadata" in chunk
assert "source" in chunk["metadata"]
assert "chunk_index" in chunk["metadata"]
class TestMemoryManager:
def test_calculate_relevance_today(self, memory_manager):
from datetime import datetime
score = memory_manager.calculate_relevance(datetime.now())
assert 0.9 < score <= 1.0 # Very recent = high score
def test_calculate_relevance_old(self, memory_manager):
from datetime import datetime, timedelta
old_date = datetime.now() - timedelta(days=30)
score = memory_manager.calculate_relevance(old_date)
assert score < 0.5 # Old = low score
Technology Stack
[Requirements]
[NEW] requirements.txt
# Core
python-dotenv
feedparser
schedule
# LLMs & Agents
google-genai
groq # Critic Agent (free Llama 3.3 70B)
langchain>=0.3.0
langgraph # State Machine Orchestration
langchain-google-genai
langchain-community
langchain-core
# Observability
langsmith
# Embeddings & Vector DB (Hybrid)
sentence-transformers
chromadb # Hot layer (local, fast)
pymongo # Archival layer (MongoDB Atlas)
langchain-mongodb
torch
# HuggingFace Models
transformers
accelerate
# Classification (zero-shot)
# Uses transformers above
# Utilities
requests
beautifulsoup4
notion-client # Notion API integration
# PDF Processing
PyPDF2 # Text-based PDF extraction (free, covers 95% of cases)
# google-cloud-vision # Optional: for scanned PDFs only
RSS Sources (12 Feeds)
| Source | RSS URL |
|---|---|
| Artificial Intelligence News | artificialintelligence-news.com/feed/ |
| Analytics Insight | analyticsinsight.net/feed/ |
| TechCrunch AI | techcrunch.com/category/artificial-intelligence/feed/ |
| MIT Tech Review AI | technologyreview.com/topic/artificial-intelligence/feed |
| VentureBeat AI | venturebeat.com/category/ai/feed/ |
| ScienceDaily AI | sciencedaily.com/rss/computers_math/artificial_intelligence.xml |
| AWS ML Blog | aws.amazon.com/blogs/machine-learning/feed/ |
| The Verge AI | theverge.com/rss/ai-artificial-intelligence/index.xml |
| Wired AI | wired.com/feed/category/ai/latest/rss |
| MIT News AI | news.mit.edu/rss/topic/artificial-intelligence2 |
| Forbes AI | forbes.com/ai/feed/ |
| Just AI News | justainews.com/feed |
| The Rundown AI | rss.beehiiv.com/feeds/2R3C6Bt5wj.xml |
Verification Plan
Phase 1: Infrastructure & Ingestion
- Test RSS fetching from all 12 sources
- Verify PDF text extraction (PyPDF2)
- Test Medium paywall bypass (Freedium)
- Verify Hybrid DB storage (Chroma hot + Mongo archival)
Phase 2: Intelligence & Logic
- Test topic classifier accuracy
- Verify anomaly detection on sample data
- Test recommendation ranking
- Analyze memory manager aging logic (7-day cutoff)
Phase 3: Agents & LLMs
- Test Research Agent (Perplexity) search depth
- Test Writer Agent (Gemini) draft quality
- Test Critic Agent (Groq) review loop & scoring
- Verify Image Agent generation (Gemini)
Phase 4: Delivery & Integration
- Test specific On-Demand URL processing
- Verify Notion page creation & property formatting
- Test Email delivery with attachments
- Full end-to-end daily simulation
Phase 5: Resilience
- Test retry logic on API failures
- Verify behavior when Freedium is down
- Check MongoDB free tier limits handling
Source code: https://github.com/rudzz1950/AI-Content-Generation-for-LinkedIn
Run locally
pip install -r requirements.txt
python app.py
app.py is a Gradio app; the original project scripts are unchanged alongside it.