# Data Enrichment Module Documentation

## Overview

The `data_enrichment.py` module provides comprehensive contact and lead enrichment capabilities for the StrategyStack platform. It integrates 14 different data sources through RapidAPI to collect, enrich, and cache contact information.

## Features

### 🔌 Multi-Provider Integration
- **LinkedIn APIs** (4 providers with fallback)
- **Apollo.io** - B2B contact enrichment  
- **Social Media** - Twitter & Facebook profile data
- **Mediafy API** - Media and social data
- **Google Search** - Web search enrichment
- **LetScrape** - Local business data & website contacts

### 💾 Intelligent Caching
- Per-user data isolation (`/data/users/{username}/enrichment_cache/`)
- 30-day cache TTL
- Automatic cache invalidation
- Privacy-first design - users own their data

### 🔄 Fallback & Redundancy
- Automatic failover between providers
- Graceful error handling
- Partial data return on API failures

## Architecture

```
DataEnrichmentManager
├── API Keys Management (11 secrets)
├── Request Handler (rate limiting, retries)
├── Cache Manager (read/write/invalidate)
└── Enrichment Methods
    ├── LinkedIn Profile
    ├── Company Data  
    ├── Apollo.io
    ├── Social Media
    ├── Local Business
    └── Website Contacts
```

## Installation & Setup

### 1. Environment Variables

Add the following secrets to your Hugging Face Space:

```bash
RAPIDAPI_KEY_LINKEDIN_FRESH
RAPIDAPI_KEY_LINKEDIN_ROCKAPIS
RAPIDAPI_KEY_LISCRAPER
RAPIDAPI_KEY_LINKEDIN_SCRAPER
RAPIDAPI_KEY_APOLLO
RAPIDAPI_KEY_MEDIAFY
RAPIDAPI_KEY_TWITTER
RAPIDAPI_KEY_FACEBOOK  
RAPIDAPI_KEY_GOOGLE
RAPIDAPI_KEY_LETSCRAPE
```

### 2. Import Module

```python
from data_enrichment import DataEnrichmentManager, enrich_contact

# Initialize manager
manager = DataEnrichmentManager(username="user123")
```

## Usage Examples

### Basic Contact Enrichment

```python
# Enrich a single contact
contact_data = {
    'name': 'John Doe',
    'email': 'john@example.com',
    'linkedin_url': 'https://linkedin.com/in/johndoe',
    'domain': 'example.com'
}

result = manager.enrich_contact_full(contact_data)

print(f"Enriched from {result['total_sources']} sources")
print(f"Sources used: {result['sources']}")
```

### LinkedIn Profile Enrichment

```python
# Get LinkedIn profile data (with fallback)
profile = manager.enrich_linkedin_profile(
    'https://linkedin.com/in/johndoe'
)

if profile:
    print(f"Name: {profile.get('name')}")
    print(f"Title: {profile.get('title')}")
    print(f"Company: {profile.get('company')}")
```

### Company Enrichment

```python
# Enrich company by domain
company = manager.enrich_company('example.com')

if company:
    print(f"Company: {company.get('name')}")
    print(f"Industry: {company.get('industry')}")
    print(f"Size: {company.get('employee_count')}")
```

### LetScrape: Local Business Data

```python
# Search for local businesses
businesses = manager.scrape_local_business(
    query='pizza restaurants',
    location='New York, NY',
    limit=20
)

for business in businesses.get('data', []):
    print(f"Name: {business.get('name')}")
    print(f"Address: {business.get('address')}")
    print(f"Phone: {business.get('phone')}")
    print(f"Rating: {business.get('rating')}")
```

### LetScrape: Website Contacts

```python
# Extract contacts from a website
contacts = manager.scrape_website_contacts('example.com')

if contacts:
    print(f"Emails: {contacts.get('emails', [])}")
    print(f"Phones: {contacts.get('phones', [])}")
    print(f"Social: {contacts.get('social_links', {})}")
```

### Batch Enrichment

```python
# Enrich multiple contacts
contacts = [
    {'name': 'John Doe', 'email': 'john@example.com'},
    {'name': 'Jane Smith', 'domain': 'acme.com'}
]

for contact in contacts:
    enriched = manager.enrich_contact_full(contact)
    print(f"Enriched {contact['name']} from {enriched['total_sources']} sources")
```

## API Methods Reference

### Core Methods

#### `__init__(username: str)`
Initialize the enrichment manager for a specific user.

**Parameters:**
- `username` (str): User identifier for data isolation

#### `enrich_contact_full(contact_data: Dict) -> Dict`
Perform comprehensive enrichment using all available sources.

**Parameters:**
- `contact_data` (dict): Contact information
  - `name` (str, optional)
  - `email` (str, optional)
  - `linkedin_url` (str, optional)
  - `domain` (str, optional)
  - `company` (str, optional)
  - `location` (str, optional)
  - `twitter_username` (str, optional)
  - `facebook_id` (str, optional)

**Returns:**
```python
{
    'original': {...},           # Original input data
    'enrichments': {            # Enriched data by source
        'linkedin': {...},
        'apollo': {...},
        'company': {...},
        'website_contacts': {...},
        'local_business': {...}
    },
    'sources': ['linkedin', 'apollo', ...],  # Sources used
    'total_sources': 5,         # Count of successful enrichments
    'enriched_at': '2025-11-06T14:00:00'  # Timestamp
}
```

### LinkedIn Methods

#### `enrich_linkedin_profile(linkedin_url: str) -> Optional[Dict]`
Enrich LinkedIn profile with automatic fallback across 4 providers.

#### `enrich_company(domain: str) -> Optional[Dict]`
Get company information from LinkedIn by domain.

### Apollo.io

#### `enrich_with_apollo(email: str = None, domain: str = None) -> Optional[Dict]`
Enrich contact using Apollo.io B2B database.

### Social Media

#### `enrich_twitter_profile(username: str) -> Optional[Dict]`
Get Twitter profile data.

#### `enrich_facebook_profile(profile_id: str) -> Optional[Dict]`
Get Facebook profile data.

### Search & Media

#### `search_google(query: str, num_results: int = 10) -> Optional[Dict]`
Perform Google search for contact information.

#### `enrich_with_mediafy(query: str) -> Optional[Dict]`
Get media and social data from Mediafy API.

### LetScrape Methods

#### `scrape_local_business(query: str, location: str = None, limit: int = 20) -> Optional[Dict]`
Scrape local business data from Google Maps.

**Parameters:**
- `query` (str): Business search query (e.g., "pizza restaurants")
- `location` (str, optional): Geographic location
- `limit` (int): Maximum results (default: 20)

**Returns:**
```python
{
    'data': [
        {
            'name': 'Business Name',
            'address': '123 Main St',
            'phone': '+1234567890',
            'email': 'contact@business.com',
            'website': 'https://business.com',
            'rating': 4.5,
            'review_count': 150,
            'category': 'Restaurant',
            'hours': {...}
        }
    ]
}
```

#### `scrape_website_contacts(domain: str) -> Optional[Dict]`
Extract emails, phone numbers, and social links from a website.

**Parameters:**
- `domain` (str): Website domain (e.g., "example.com")

**Returns:**
```python
{
    'emails': ['contact@example.com', 'info@example.com'],
    'phones': ['+1234567890'],
    'social_links': {
        'facebook': 'https://facebook.com/example',
        'linkedin': 'https://linkedin.com/company/example',
        'twitter': 'https://twitter.com/example',
        'instagram': 'https://instagram.com/example'
    }
}
```

### Utility Methods

#### `get_enrichment_stats() -> Dict`
Get cache statistics and API usage information.

**Returns:**
```python
{
    'cache_directory': '/data/users/user123/enrichment_cache',
    'cached_profiles': 45,
    'cache_types': {
        'linkedin': 15,
        'company': 10,
        'apollo': 12,
        'website_contacts': 8
    },
    'configured_apis': ['linkedin_fresh', 'apollo', 'letscrape'],
    'total_apis': 10
}
```

## Data Privacy & Security

### User Data Isolation
- Each user has separate cache directory
- No cross-user data access
- Users own their enrichment data

### Cache Management
- 30-day automatic expiration
- Stored in user-specific directories
- Can be manually cleared

### API Key Security
- All keys stored as HuggingFace Secrets
- Never exposed in logs or responses
- Separate keys per API provider

## Performance Optimization

### Caching Strategy
1. Check cache first (instant response)
2. If cache miss, call API
3. Store result in cache
4. Return data

### Fallback Logic
```python
Providers = [Primary, Secondary, Tertiary, Fallback]
for provider in Providers:
    try:
        data = provider.fetch()
        if data: return data
    except:
        continue
return None
```

### Rate Limiting
- 30-second timeout per request
- Automatic retry on failures
- Respects API provider limits

## Error Handling

All methods return `None` on failure and log errors:

```python
try:
    result = manager.enrich_linkedin_profile(url)
    if result:
        # Success
    else:
        # Failed - check logs
except Exception as e:
    # Handle exception
    logger.error(f"Enrichment failed: {e}")
```

## Integration Examples

### With Contact Management

```python
def create_contact(user, contact_data):
    # Create contact
    contact = Contact.create(**contact_data)
    
    # Enrich asynchronously
    manager = DataEnrichmentManager(user.username)
    enriched = manager.enrich_contact_full(contact_data)
    
    # Store enrichment
    contact.enrichment_data = enriched
    contact.save()
    
    return contact
```

### With Outreach Agent

```python
def prepare_outreach(contact):
    # Enrich contact before outreach
    manager = DataEnrichmentManager(current_user)
    enriched = manager.enrich_contact_full({
        'email': contact.email,
        'linkedin_url': contact.linkedin_url,
        'domain': contact.company_domain
    })
    
    # Use enriched data for personalization
    if enriched['total_sources'] > 0:
        contact.enriched = True
        contact.enrichment_score = enriched['total_sources']
    
    return enriched
```

## Troubleshooting

### Common Issues

**Issue: No data returned**
- Check API keys are configured
- Verify network connectivity
- Check rate limits

**Issue: Slow responses**
- Cache may be cold (first request)
- API provider may be slow
- Check timeout settings

**Issue: Missing enrichments**
- Some providers may not have data
- Check logs for API errors
- Verify input data format

### Debug Mode

```python
import logging
logging.basicConfig(level=logging.DEBUG)

manager = DataEnrichmentManager(username)
result = manager.enrich_contact_full(contact_data)
# Check logs for detailed information
```

## API Provider Details

### RapidAPI Configuration

All APIs are accessed through RapidAPI:
- Host: `*.p.rapidapi.com`
- Authentication: `x-rapidapi-key` header
- Format: JSON
- Protocol: HTTPS

### Provider URLs

- **LinkedIn Fresh**: `fresh-linkedin-profile-data.p.rapidapi.com`
- **LinkedIn RockAPIs**: `linkedin-data-api.p.rapidapi.com`
- **LI Data Scraper**: `li-data-scraper.p.rapidapi.com`
- **LinkedIn Scraper**: `linkedin-data-scraper.p.rapidapi.com`
- **Apollo.io**: `apollo-io-no-cookies-required.p.rapidapi.com`
- **Mediafy**: `mediafy-api.p.rapidapi.com`
- **Twitter**: `twitter241.p.rapidapi.com`
- **Facebook**: `facebook-scraper3.p.rapidapi.com`
- **Google**: `google-search74.p.rapidapi.com`
- **LetScrape Business**: `local-business-data.p.rapidapi.com`
- **LetScrape Contacts**: `website-contacts-scraper.p.rapidapi.com`

## Changelog

### Version 2.0.0 (2025-11-06)
- ✨ Added LetScrape Local Business Data API
- ✨ Added LetScrape Website Contacts Scraper API
- 🔄 Enhanced fallback logic
- 📚 Comprehensive documentation
- 🐛 Bug fixes and improvements

### Version 1.0.0 (2025-11-05)
- 🎉 Initial release
- LinkedIn enrichment (4 providers)
- Apollo.io integration
- Social media enrichment
- Caching system
- Per-user data isolation

## Support & Contact

For issues or questions:
- Check logs in `/data/users/{username}/enrichment_cache/`
- Review API provider documentation
- Contact StrategyStack support

---

**Last Updated:** November 6, 2025  
**Version:** 2.0.0  
**Maintainer:** StrategyStack Team