README / docs /DATA_ENRICHMENT.md
stackConsult's picture
Create docs/DATA_ENRICHMENT.md
89ae4ff verified
|
Raw History Blame Contribute Delete
12 kB
# Data Enrichment Module Documentation
## Overview
The `data_enrichment.py` module provides comprehensive contact and lead enrichment capabilities for the StrategyStack platform. It integrates 14 different data sources through RapidAPI to collect, enrich, and cache contact information.
## Features
### πŸ”Œ Multi-Provider Integration
- **LinkedIn APIs** (4 providers with fallback)
- **Apollo.io** - B2B contact enrichment
- **Social Media** - Twitter & Facebook profile data
- **Mediafy API** - Media and social data
- **Google Search** - Web search enrichment
- **LetScrape** - Local business data & website contacts
### πŸ’Ύ Intelligent Caching
- Per-user data isolation (`/data/users/{username}/enrichment_cache/`)
- 30-day cache TTL
- Automatic cache invalidation
- Privacy-first design - users own their data
### πŸ”„ Fallback & Redundancy
- Automatic failover between providers
- Graceful error handling
- Partial data return on API failures
## Architecture
```
DataEnrichmentManager
β”œβ”€β”€ API Keys Management (11 secrets)
β”œβ”€β”€ Request Handler (rate limiting, retries)
β”œβ”€β”€ Cache Manager (read/write/invalidate)
└── Enrichment Methods
β”œβ”€β”€ LinkedIn Profile
β”œβ”€β”€ Company Data
β”œβ”€β”€ Apollo.io
β”œβ”€β”€ Social Media
β”œβ”€β”€ Local Business
└── Website Contacts
```
## Installation & Setup
### 1. Environment Variables
Add the following secrets to your Hugging Face Space:
```bash
RAPIDAPI_KEY_LINKEDIN_FRESH
RAPIDAPI_KEY_LINKEDIN_ROCKAPIS
RAPIDAPI_KEY_LISCRAPER
RAPIDAPI_KEY_LINKEDIN_SCRAPER
RAPIDAPI_KEY_APOLLO
RAPIDAPI_KEY_MEDIAFY
RAPIDAPI_KEY_TWITTER
RAPIDAPI_KEY_FACEBOOK
RAPIDAPI_KEY_GOOGLE
RAPIDAPI_KEY_LETSCRAPE
```
### 2. Import Module
```python
from data_enrichment import DataEnrichmentManager, enrich_contact
# Initialize manager
manager = DataEnrichmentManager(username="user123")
```
## Usage Examples
### Basic Contact Enrichment
```python
# Enrich a single contact
contact_data = {
'name': 'John Doe',
'email': 'john@example.com',
'linkedin_url': 'https://linkedin.com/in/johndoe',
'domain': 'example.com'
}
result = manager.enrich_contact_full(contact_data)
print(f"Enriched from {result['total_sources']} sources")
print(f"Sources used: {result['sources']}")
```
### LinkedIn Profile Enrichment
```python
# Get LinkedIn profile data (with fallback)
profile = manager.enrich_linkedin_profile(
'https://linkedin.com/in/johndoe'
)
if profile:
print(f"Name: {profile.get('name')}")
print(f"Title: {profile.get('title')}")
print(f"Company: {profile.get('company')}")
```
### Company Enrichment
```python
# Enrich company by domain
company = manager.enrich_company('example.com')
if company:
print(f"Company: {company.get('name')}")
print(f"Industry: {company.get('industry')}")
print(f"Size: {company.get('employee_count')}")
```
### LetScrape: Local Business Data
```python
# Search for local businesses
businesses = manager.scrape_local_business(
query='pizza restaurants',
location='New York, NY',
limit=20
)
for business in businesses.get('data', []):
print(f"Name: {business.get('name')}")
print(f"Address: {business.get('address')}")
print(f"Phone: {business.get('phone')}")
print(f"Rating: {business.get('rating')}")
```
### LetScrape: Website Contacts
```python
# Extract contacts from a website
contacts = manager.scrape_website_contacts('example.com')
if contacts:
print(f"Emails: {contacts.get('emails', [])}")
print(f"Phones: {contacts.get('phones', [])}")
print(f"Social: {contacts.get('social_links', {})}")
```
### Batch Enrichment
```python
# Enrich multiple contacts
contacts = [
{'name': 'John Doe', 'email': 'john@example.com'},
{'name': 'Jane Smith', 'domain': 'acme.com'}
]
for contact in contacts:
enriched = manager.enrich_contact_full(contact)
print(f"Enriched {contact['name']} from {enriched['total_sources']} sources")
```
## API Methods Reference
### Core Methods
#### `__init__(username: str)`
Initialize the enrichment manager for a specific user.
**Parameters:**
- `username` (str): User identifier for data isolation
#### `enrich_contact_full(contact_data: Dict) -> Dict`
Perform comprehensive enrichment using all available sources.
**Parameters:**
- `contact_data` (dict): Contact information
- `name` (str, optional)
- `email` (str, optional)
- `linkedin_url` (str, optional)
- `domain` (str, optional)
- `company` (str, optional)
- `location` (str, optional)
- `twitter_username` (str, optional)
- `facebook_id` (str, optional)
**Returns:**
```python
{
'original': {...}, # Original input data
'enrichments': { # Enriched data by source
'linkedin': {...},
'apollo': {...},
'company': {...},
'website_contacts': {...},
'local_business': {...}
},
'sources': ['linkedin', 'apollo', ...], # Sources used
'total_sources': 5, # Count of successful enrichments
'enriched_at': '2025-11-06T14:00:00' # Timestamp
}
```
### LinkedIn Methods
#### `enrich_linkedin_profile(linkedin_url: str) -> Optional[Dict]`
Enrich LinkedIn profile with automatic fallback across 4 providers.
#### `enrich_company(domain: str) -> Optional[Dict]`
Get company information from LinkedIn by domain.
### Apollo.io
#### `enrich_with_apollo(email: str = None, domain: str = None) -> Optional[Dict]`
Enrich contact using Apollo.io B2B database.
### Social Media
#### `enrich_twitter_profile(username: str) -> Optional[Dict]`
Get Twitter profile data.
#### `enrich_facebook_profile(profile_id: str) -> Optional[Dict]`
Get Facebook profile data.
### Search & Media
#### `search_google(query: str, num_results: int = 10) -> Optional[Dict]`
Perform Google search for contact information.
#### `enrich_with_mediafy(query: str) -> Optional[Dict]`
Get media and social data from Mediafy API.
### LetScrape Methods
#### `scrape_local_business(query: str, location: str = None, limit: int = 20) -> Optional[Dict]`
Scrape local business data from Google Maps.
**Parameters:**
- `query` (str): Business search query (e.g., "pizza restaurants")
- `location` (str, optional): Geographic location
- `limit` (int): Maximum results (default: 20)
**Returns:**
```python
{
'data': [
{
'name': 'Business Name',
'address': '123 Main St',
'phone': '+1234567890',
'email': 'contact@business.com',
'website': 'https://business.com',
'rating': 4.5,
'review_count': 150,
'category': 'Restaurant',
'hours': {...}
}
]
}
```
#### `scrape_website_contacts(domain: str) -> Optional[Dict]`
Extract emails, phone numbers, and social links from a website.
**Parameters:**
- `domain` (str): Website domain (e.g., "example.com")
**Returns:**
```python
{
'emails': ['contact@example.com', 'info@example.com'],
'phones': ['+1234567890'],
'social_links': {
'facebook': 'https://facebook.com/example',
'linkedin': 'https://linkedin.com/company/example',
'twitter': 'https://twitter.com/example',
'instagram': 'https://instagram.com/example'
}
}
```
### Utility Methods
#### `get_enrichment_stats() -> Dict`
Get cache statistics and API usage information.
**Returns:**
```python
{
'cache_directory': '/data/users/user123/enrichment_cache',
'cached_profiles': 45,
'cache_types': {
'linkedin': 15,
'company': 10,
'apollo': 12,
'website_contacts': 8
},
'configured_apis': ['linkedin_fresh', 'apollo', 'letscrape'],
'total_apis': 10
}
```
## Data Privacy & Security
### User Data Isolation
- Each user has separate cache directory
- No cross-user data access
- Users own their enrichment data
### Cache Management
- 30-day automatic expiration
- Stored in user-specific directories
- Can be manually cleared
### API Key Security
- All keys stored as HuggingFace Secrets
- Never exposed in logs or responses
- Separate keys per API provider
## Performance Optimization
### Caching Strategy
1. Check cache first (instant response)
2. If cache miss, call API
3. Store result in cache
4. Return data
### Fallback Logic
```python
Providers = [Primary, Secondary, Tertiary, Fallback]
for provider in Providers:
try:
data = provider.fetch()
if data: return data
except:
continue
return None
```
### Rate Limiting
- 30-second timeout per request
- Automatic retry on failures
- Respects API provider limits
## Error Handling
All methods return `None` on failure and log errors:
```python
try:
result = manager.enrich_linkedin_profile(url)
if result:
# Success
else:
# Failed - check logs
except Exception as e:
# Handle exception
logger.error(f"Enrichment failed: {e}")
```
## Integration Examples
### With Contact Management
```python
def create_contact(user, contact_data):
# Create contact
contact = Contact.create(**contact_data)
# Enrich asynchronously
manager = DataEnrichmentManager(user.username)
enriched = manager.enrich_contact_full(contact_data)
# Store enrichment
contact.enrichment_data = enriched
contact.save()
return contact
```
### With Outreach Agent
```python
def prepare_outreach(contact):
# Enrich contact before outreach
manager = DataEnrichmentManager(current_user)
enriched = manager.enrich_contact_full({
'email': contact.email,
'linkedin_url': contact.linkedin_url,
'domain': contact.company_domain
})
# Use enriched data for personalization
if enriched['total_sources'] > 0:
contact.enriched = True
contact.enrichment_score = enriched['total_sources']
return enriched
```
## Troubleshooting
### Common Issues
**Issue: No data returned**
- Check API keys are configured
- Verify network connectivity
- Check rate limits
**Issue: Slow responses**
- Cache may be cold (first request)
- API provider may be slow
- Check timeout settings
**Issue: Missing enrichments**
- Some providers may not have data
- Check logs for API errors
- Verify input data format
### Debug Mode
```python
import logging
logging.basicConfig(level=logging.DEBUG)
manager = DataEnrichmentManager(username)
result = manager.enrich_contact_full(contact_data)
# Check logs for detailed information
```
## API Provider Details
### RapidAPI Configuration
All APIs are accessed through RapidAPI:
- Host: `*.p.rapidapi.com`
- Authentication: `x-rapidapi-key` header
- Format: JSON
- Protocol: HTTPS
### Provider URLs
- **LinkedIn Fresh**: `fresh-linkedin-profile-data.p.rapidapi.com`
- **LinkedIn RockAPIs**: `linkedin-data-api.p.rapidapi.com`
- **LI Data Scraper**: `li-data-scraper.p.rapidapi.com`
- **LinkedIn Scraper**: `linkedin-data-scraper.p.rapidapi.com`
- **Apollo.io**: `apollo-io-no-cookies-required.p.rapidapi.com`
- **Mediafy**: `mediafy-api.p.rapidapi.com`
- **Twitter**: `twitter241.p.rapidapi.com`
- **Facebook**: `facebook-scraper3.p.rapidapi.com`
- **Google**: `google-search74.p.rapidapi.com`
- **LetScrape Business**: `local-business-data.p.rapidapi.com`
- **LetScrape Contacts**: `website-contacts-scraper.p.rapidapi.com`
## Changelog
### Version 2.0.0 (2025-11-06)
- ✨ Added LetScrape Local Business Data API
- ✨ Added LetScrape Website Contacts Scraper API
- πŸ”„ Enhanced fallback logic
- πŸ“š Comprehensive documentation
- πŸ› Bug fixes and improvements
### Version 1.0.0 (2025-11-05)
- πŸŽ‰ Initial release
- LinkedIn enrichment (4 providers)
- Apollo.io integration
- Social media enrichment
- Caching system
- Per-user data isolation
## Support & Contact
For issues or questions:
- Check logs in `/data/users/{username}/enrichment_cache/`
- Review API provider documentation
- Contact StrategyStack support
---
**Last Updated:** November 6, 2025
**Version:** 2.0.0
**Maintainer:** StrategyStack Team