Create docs/DATA_ENRICHMENT.md

docs/DATA_ENRICHMENT.md

@@ -0,0 +1,480 @@
+# Data Enrichment Module Documentation
+
+## Overview
+
+The `data_enrichment.py` module provides comprehensive contact and lead enrichment capabilities for the StrategyStack platform. It integrates 14 different data sources through RapidAPI to collect, enrich, and cache contact information.
+
+## Features
+
+### 🔌 Multi-Provider Integration
+- **LinkedIn APIs** (4 providers with fallback)
+- **Apollo.io** - B2B contact enrichment  
+- **Social Media** - Twitter & Facebook profile data
+- **Mediafy API** - Media and social data
+- **Google Search** - Web search enrichment
+- **LetScrape** - Local business data & website contacts
+
+### 💾 Intelligent Caching
+- Per-user data isolation (`/data/users/{username}/enrichment_cache/`)
+- 30-day cache TTL
+- Automatic cache invalidation
+- Privacy-first design - users own their data
+
+### 🔄 Fallback & Redundancy
+- Automatic failover between providers
+- Graceful error handling
+- Partial data return on API failures
+
+## Architecture
+
+```
+DataEnrichmentManager
+├── API Keys Management (11 secrets)
+├── Request Handler (rate limiting, retries)
+├── Cache Manager (read/write/invalidate)
+└── Enrichment Methods
+    ├── LinkedIn Profile
+    ├── Company Data  
+    ├── Apollo.io
+    ├── Social Media
+    ├── Local Business
+    └── Website Contacts
+```
+
+## Installation & Setup
+
+### 1. Environment Variables
+
+Add the following secrets to your Hugging Face Space:
+
+```bash
+RAPIDAPI_KEY_LINKEDIN_FRESH
+RAPIDAPI_KEY_LINKEDIN_ROCKAPIS
+RAPIDAPI_KEY_LISCRAPER
+RAPIDAPI_KEY_LINKEDIN_SCRAPER
+RAPIDAPI_KEY_APOLLO
+RAPIDAPI_KEY_MEDIAFY
+RAPIDAPI_KEY_TWITTER
+RAPIDAPI_KEY_FACEBOOK  
+RAPIDAPI_KEY_GOOGLE
+RAPIDAPI_KEY_LETSCRAPE
+```
+
+### 2. Import Module
+
+```python
+from data_enrichment import DataEnrichmentManager, enrich_contact
+
+# Initialize manager
+manager = DataEnrichmentManager(username="user123")
+```
+
+## Usage Examples
+
+### Basic Contact Enrichment
+
+```python
+# Enrich a single contact
+contact_data = {
+    'name': 'John Doe',
+    'email': 'john@example.com',
+    'linkedin_url': 'https://linkedin.com/in/johndoe',
+    'domain': 'example.com'
+}
+
+result = manager.enrich_contact_full(contact_data)
+
+print(f"Enriched from {result['total_sources']} sources")
+print(f"Sources used: {result['sources']}")
+```
+
+### LinkedIn Profile Enrichment
+
+```python
+# Get LinkedIn profile data (with fallback)
+profile = manager.enrich_linkedin_profile(
+    'https://linkedin.com/in/johndoe'
+)
+
+if profile:
+    print(f"Name: {profile.get('name')}")
+    print(f"Title: {profile.get('title')}")
+    print(f"Company: {profile.get('company')}")
+```
+
+### Company Enrichment
+
+```python
+# Enrich company by domain
+company = manager.enrich_company('example.com')
+
+if company:
+    print(f"Company: {company.get('name')}")
+    print(f"Industry: {company.get('industry')}")
+    print(f"Size: {company.get('employee_count')}")
+```
+
+### LetScrape: Local Business Data
+
+```python
+# Search for local businesses
+businesses = manager.scrape_local_business(
+    query='pizza restaurants',
+    location='New York, NY',
+    limit=20
+)
+
+for business in businesses.get('data', []):
+    print(f"Name: {business.get('name')}")
+    print(f"Address: {business.get('address')}")
+    print(f"Phone: {business.get('phone')}")
+    print(f"Rating: {business.get('rating')}")
+```
+
+### LetScrape: Website Contacts
+
+```python
+# Extract contacts from a website
+contacts = manager.scrape_website_contacts('example.com')
+
+if contacts:
+    print(f"Emails: {contacts.get('emails', [])}")
+    print(f"Phones: {contacts.get('phones', [])}")
+    print(f"Social: {contacts.get('social_links', {})}")
+```
+
+### Batch Enrichment
+
+```python
+# Enrich multiple contacts
+contacts = [
+    {'name': 'John Doe', 'email': 'john@example.com'},
+    {'name': 'Jane Smith', 'domain': 'acme.com'}
+]
+
+for contact in contacts:
+    enriched = manager.enrich_contact_full(contact)
+    print(f"Enriched {contact['name']} from {enriched['total_sources']} sources")
+```
+
+## API Methods Reference
+
+### Core Methods
+
+#### `__init__(username: str)`
+Initialize the enrichment manager for a specific user.
+
+**Parameters:**
+- `username` (str): User identifier for data isolation
+
+#### `enrich_contact_full(contact_data: Dict) -> Dict`
+Perform comprehensive enrichment using all available sources.
+
+**Parameters:**
+- `contact_data` (dict): Contact information
+  - `name` (str, optional)
+  - `email` (str, optional)
+  - `linkedin_url` (str, optional)
+  - `domain` (str, optional)
+  - `company` (str, optional)
+  - `location` (str, optional)
+  - `twitter_username` (str, optional)
+  - `facebook_id` (str, optional)
+
+**Returns:**
+```python
+{
+    'original': {...},           # Original input data
+    'enrichments': {            # Enriched data by source
+        'linkedin': {...},
+        'apollo': {...},
+        'company': {...},
+        'website_contacts': {...},
+        'local_business': {...}
+    },
+    'sources': ['linkedin', 'apollo', ...],  # Sources used
+    'total_sources': 5,         # Count of successful enrichments
+    'enriched_at': '2025-11-06T14:00:00'  # Timestamp
+}
+```
+
+### LinkedIn Methods
+
+#### `enrich_linkedin_profile(linkedin_url: str) -> Optional[Dict]`
+Enrich LinkedIn profile with automatic fallback across 4 providers.
+
+#### `enrich_company(domain: str) -> Optional[Dict]`
+Get company information from LinkedIn by domain.
+
+### Apollo.io
+
+#### `enrich_with_apollo(email: str = None, domain: str = None) -> Optional[Dict]`
+Enrich contact using Apollo.io B2B database.
+
+### Social Media
+
+#### `enrich_twitter_profile(username: str) -> Optional[Dict]`
+Get Twitter profile data.
+
+#### `enrich_facebook_profile(profile_id: str) -> Optional[Dict]`
+Get Facebook profile data.
+
+### Search & Media
+
+#### `search_google(query: str, num_results: int = 10) -> Optional[Dict]`
+Perform Google search for contact information.
+
+#### `enrich_with_mediafy(query: str) -> Optional[Dict]`
+Get media and social data from Mediafy API.
+
+### LetScrape Methods
+
+#### `scrape_local_business(query: str, location: str = None, limit: int = 20) -> Optional[Dict]`
+Scrape local business data from Google Maps.
+
+**Parameters:**
+- `query` (str): Business search query (e.g., "pizza restaurants")
+- `location` (str, optional): Geographic location
+- `limit` (int): Maximum results (default: 20)
+
+**Returns:**
+```python
+{
+    'data': [
+        {
+            'name': 'Business Name',
+            'address': '123 Main St',
+            'phone': '+1234567890',
+            'email': 'contact@business.com',
+            'website': 'https://business.com',
+            'rating': 4.5,
+            'review_count': 150,
+            'category': 'Restaurant',
+            'hours': {...}
+        }
+    ]
+}
+```
+
+#### `scrape_website_contacts(domain: str) -> Optional[Dict]`
+Extract emails, phone numbers, and social links from a website.
+
+**Parameters:**
+- `domain` (str): Website domain (e.g., "example.com")
+
+**Returns:**
+```python
+{
+    'emails': ['contact@example.com', 'info@example.com'],
+    'phones': ['+1234567890'],
+    'social_links': {
+        'facebook': 'https://facebook.com/example',
+        'linkedin': 'https://linkedin.com/company/example',
+        'twitter': 'https://twitter.com/example',
+        'instagram': 'https://instagram.com/example'
+    }
+}
+```
+
+### Utility Methods
+
+#### `get_enrichment_stats() -> Dict`
+Get cache statistics and API usage information.
+
+**Returns:**
+```python
+{
+    'cache_directory': '/data/users/user123/enrichment_cache',
+    'cached_profiles': 45,
+    'cache_types': {
+        'linkedin': 15,
+        'company': 10,
+        'apollo': 12,
+        'website_contacts': 8
+    },
+    'configured_apis': ['linkedin_fresh', 'apollo', 'letscrape'],
+    'total_apis': 10
+}
+```
+
+## Data Privacy & Security
+
+### User Data Isolation
+- Each user has separate cache directory
+- No cross-user data access
+- Users own their enrichment data
+
+### Cache Management
+- 30-day automatic expiration
+- Stored in user-specific directories
+- Can be manually cleared
+
+### API Key Security
+- All keys stored as HuggingFace Secrets
+- Never exposed in logs or responses
+- Separate keys per API provider
+
+## Performance Optimization
+
+### Caching Strategy
+1. Check cache first (instant response)
+2. If cache miss, call API
+3. Store result in cache
+4. Return data
+
+### Fallback Logic
+```python
+Providers = [Primary, Secondary, Tertiary, Fallback]
+for provider in Providers:
+    try:
+        data = provider.fetch()
+        if data: return data
+    except:
+        continue
+return None
+```
+
+### Rate Limiting
+- 30-second timeout per request
+- Automatic retry on failures
+- Respects API provider limits
+
+## Error Handling
+
+All methods return `None` on failure and log errors:
+
+```python
+try:
+    result = manager.enrich_linkedin_profile(url)
+    if result:
+        # Success
+    else:
+        # Failed - check logs
+except Exception as e:
+    # Handle exception
+    logger.error(f"Enrichment failed: {e}")
+```
+
+## Integration Examples
+
+### With Contact Management
+
+```python
+def create_contact(user, contact_data):
+    # Create contact
+    contact = Contact.create(**contact_data)
+    
+    # Enrich asynchronously
+    manager = DataEnrichmentManager(user.username)
+    enriched = manager.enrich_contact_full(contact_data)
+    
+    # Store enrichment
+    contact.enrichment_data = enriched
+    contact.save()
+    
+    return contact
+```
+
+### With Outreach Agent
+
+```python
+def prepare_outreach(contact):
+    # Enrich contact before outreach
+    manager = DataEnrichmentManager(current_user)
+    enriched = manager.enrich_contact_full({
+        'email': contact.email,
+        'linkedin_url': contact.linkedin_url,
+        'domain': contact.company_domain
+    })
+    
+    # Use enriched data for personalization
+    if enriched['total_sources'] > 0:
+        contact.enriched = True
+        contact.enrichment_score = enriched['total_sources']
+    
+    return enriched
+```
+
+## Troubleshooting
+
+### Common Issues
+
+**Issue: No data returned**
+- Check API keys are configured
+- Verify network connectivity
+- Check rate limits
+
+**Issue: Slow responses**
+- Cache may be cold (first request)
+- API provider may be slow
+- Check timeout settings
+
+**Issue: Missing enrichments**
+- Some providers may not have data
+- Check logs for API errors
+- Verify input data format
+
+### Debug Mode
+
+```python
+import logging
+logging.basicConfig(level=logging.DEBUG)
+
+manager = DataEnrichmentManager(username)
+result = manager.enrich_contact_full(contact_data)
+# Check logs for detailed information
+```
+
+## API Provider Details
+
+### RapidAPI Configuration
+
+All APIs are accessed through RapidAPI:
+- Host: `*.p.rapidapi.com`
+- Authentication: `x-rapidapi-key` header
+- Format: JSON
+- Protocol: HTTPS
+
+### Provider URLs
+
+- **LinkedIn Fresh**: `fresh-linkedin-profile-data.p.rapidapi.com`
+- **LinkedIn RockAPIs**: `linkedin-data-api.p.rapidapi.com`
+- **LI Data Scraper**: `li-data-scraper.p.rapidapi.com`
+- **LinkedIn Scraper**: `linkedin-data-scraper.p.rapidapi.com`
+- **Apollo.io**: `apollo-io-no-cookies-required.p.rapidapi.com`
+- **Mediafy**: `mediafy-api.p.rapidapi.com`
+- **Twitter**: `twitter241.p.rapidapi.com`
+- **Facebook**: `facebook-scraper3.p.rapidapi.com`
+- **Google**: `google-search74.p.rapidapi.com`
+- **LetScrape Business**: `local-business-data.p.rapidapi.com`
+- **LetScrape Contacts**: `website-contacts-scraper.p.rapidapi.com`
+
+## Changelog
+
+### Version 2.0.0 (2025-11-06)
+- ✨ Added LetScrape Local Business Data API
+- ✨ Added LetScrape Website Contacts Scraper API
+- 🔄 Enhanced fallback logic
+- 📚 Comprehensive documentation
+- 🐛 Bug fixes and improvements
+
+### Version 1.0.0 (2025-11-05)
+- 🎉 Initial release
+- LinkedIn enrichment (4 providers)
+- Apollo.io integration
+- Social media enrichment
+- Caching system
+- Per-user data isolation
+
+## Support & Contact
+
+For issues or questions:
+- Check logs in `/data/users/{username}/enrichment_cache/`
+- Review API provider documentation
+- Contact StrategyStack support
+
+---
+
+**Last Updated:** November 6, 2025  
+**Version:** 2.0.0  
+**Maintainer:** StrategyStack Team