docs/DATA_ENRICHMENT.md

Data Enrichment Module Documentation

Overview

The data_enrichment.py module provides comprehensive contact and lead enrichment capabilities for the StrategyStack platform. It integrates 14 different data sources through RapidAPI to collect, enrich, and cache contact information.

Features

🔌 Multi-Provider Integration

• LinkedIn APIs (4 providers with fallback)
• Apollo.io - B2B contact enrichment
• Social Media - Twitter & Facebook profile data
• Mediafy API - Media and social data
• Google Search - Web search enrichment
• LetScrape - Local business data & website contacts

💾 Intelligent Caching

• Per-user data isolation (/data/users/{username}/enrichment_cache/)
• 30-day cache TTL
• Automatic cache invalidation
• Privacy-first design - users own their data

🔄 Fallback & Redundancy

• Automatic failover between providers
• Graceful error handling
• Partial data return on API failures

Architecture

DataEnrichmentManager
├── API Keys Management (11 secrets)
├── Request Handler (rate limiting, retries)
├── Cache Manager (read/write/invalidate)
└── Enrichment Methods
    ├── LinkedIn Profile
    ├── Company Data  
    ├── Apollo.io
    ├── Social Media
    ├── Local Business
    └── Website Contacts

Installation & Setup

1. Environment Variables

Add the following secrets to your Hugging Face Space:

RAPIDAPI_KEY_LINKEDIN_FRESH
RAPIDAPI_KEY_LINKEDIN_ROCKAPIS
RAPIDAPI_KEY_LISCRAPER
RAPIDAPI_KEY_LINKEDIN_SCRAPER
RAPIDAPI_KEY_APOLLO
RAPIDAPI_KEY_MEDIAFY
RAPIDAPI_KEY_TWITTER
RAPIDAPI_KEY_FACEBOOK  
RAPIDAPI_KEY_GOOGLE
RAPIDAPI_KEY_LETSCRAPE

2. Import Module

from data_enrichment import DataEnrichmentManager, enrich_contact

# Initialize manager
manager = DataEnrichmentManager(username="user123")

Usage Examples

Basic Contact Enrichment

# Enrich a single contact
contact_data = {
    'name': 'John Doe',
    'email': 'john@example.com',
    'linkedin_url': 'https://linkedin.com/in/johndoe',
    'domain': 'example.com'
}

result = manager.enrich_contact_full(contact_data)

print(f"Enriched from {result['total_sources']} sources")
print(f"Sources used: {result['sources']}")

LinkedIn Profile Enrichment

# Get LinkedIn profile data (with fallback)
profile = manager.enrich_linkedin_profile(
    'https://linkedin.com/in/johndoe'
)

if profile:
    print(f"Name: {profile.get('name')}")
    print(f"Title: {profile.get('title')}")
    print(f"Company: {profile.get('company')}")

Company Enrichment

# Enrich company by domain
company = manager.enrich_company('example.com')

if company:
    print(f"Company: {company.get('name')}")
    print(f"Industry: {company.get('industry')}")
    print(f"Size: {company.get('employee_count')}")

LetScrape: Local Business Data

# Search for local businesses
businesses = manager.scrape_local_business(
    query='pizza restaurants',
    location='New York, NY',
    limit=20
)

for business in businesses.get('data', []):
    print(f"Name: {business.get('name')}")
    print(f"Address: {business.get('address')}")
    print(f"Phone: {business.get('phone')}")
    print(f"Rating: {business.get('rating')}")

LetScrape: Website Contacts

# Extract contacts from a website
contacts = manager.scrape_website_contacts('example.com')

if contacts:
    print(f"Emails: {contacts.get('emails', [])}")
    print(f"Phones: {contacts.get('phones', [])}")
    print(f"Social: {contacts.get('social_links', {})}")

Batch Enrichment

# Enrich multiple contacts
contacts = [
    {'name': 'John Doe', 'email': 'john@example.com'},
    {'name': 'Jane Smith', 'domain': 'acme.com'}
]

for contact in contacts:
    enriched = manager.enrich_contact_full(contact)
    print(f"Enriched {contact['name']} from {enriched['total_sources']} sources")

API Methods Reference

Core Methods

__init__(username: str)

Initialize the enrichment manager for a specific user.

Parameters:

• username (str): User identifier for data isolation

enrich_contact_full(contact_data: Dict) -> Dict

Perform comprehensive enrichment using all available sources.

Parameters:

• contact_data (dict): Contact information
  • name (str, optional)
  • email (str, optional)
  • linkedin_url (str, optional)
  • domain (str, optional)
  • company (str, optional)
  • location (str, optional)
  • twitter_username (str, optional)
  • facebook_id (str, optional)

Returns:

{
    'original': {...},           # Original input data
    'enrichments': {            # Enriched data by source
        'linkedin': {...},
        'apollo': {...},
        'company': {...},
        'website_contacts': {...},
        'local_business': {...}
    },
    'sources': ['linkedin', 'apollo', ...],  # Sources used
    'total_sources': 5,         # Count of successful enrichments
    'enriched_at': '2025-11-06T14:00:00'  # Timestamp
}

LinkedIn Methods

enrich_linkedin_profile(linkedin_url: str) -> Optional[Dict]

Enrich LinkedIn profile with automatic fallback across 4 providers.

enrich_company(domain: str) -> Optional[Dict]

Get company information from LinkedIn by domain.

Apollo.io

enrich_with_apollo(email: str = None, domain: str = None) -> Optional[Dict]

Enrich contact using Apollo.io B2B database.

Social Media

enrich_twitter_profile(username: str) -> Optional[Dict]

Get Twitter profile data.

enrich_facebook_profile(profile_id: str) -> Optional[Dict]

Get Facebook profile data.

Search & Media

search_google(query: str, num_results: int = 10) -> Optional[Dict]

Perform Google search for contact information.

enrich_with_mediafy(query: str) -> Optional[Dict]

Get media and social data from Mediafy API.

LetScrape Methods

scrape_local_business(query: str, location: str = None, limit: int = 20) -> Optional[Dict]

Scrape local business data from Google Maps.

Parameters:

• query (str): Business search query (e.g., "pizza restaurants")
• location (str, optional): Geographic location
• limit (int): Maximum results (default: 20)

Returns:

{
    'data': [
        {
            'name': 'Business Name',
            'address': '123 Main St',
            'phone': '+1234567890',
            'email': 'contact@business.com',
            'website': 'https://business.com',
            'rating': 4.5,
            'review_count': 150,
            'category': 'Restaurant',
            'hours': {...}
        }
    ]
}

scrape_website_contacts(domain: str) -> Optional[Dict]

Extract emails, phone numbers, and social links from a website.

Parameters:

• domain (str): Website domain (e.g., "example.com")

Returns:

{
    'emails': ['contact@example.com', 'info@example.com'],
    'phones': ['+1234567890'],
    'social_links': {
        'facebook': 'https://facebook.com/example',
        'linkedin': 'https://linkedin.com/company/example',
        'twitter': 'https://twitter.com/example',
        'instagram': 'https://instagram.com/example'
    }
}

Utility Methods

get_enrichment_stats() -> Dict

Get cache statistics and API usage information.

Returns:

{
    'cache_directory': '/data/users/user123/enrichment_cache',
    'cached_profiles': 45,
    'cache_types': {
        'linkedin': 15,
        'company': 10,
        'apollo': 12,
        'website_contacts': 8
    },
    'configured_apis': ['linkedin_fresh', 'apollo', 'letscrape'],
    'total_apis': 10
}

Data Privacy & Security

User Data Isolation

• Each user has separate cache directory
• No cross-user data access
• Users own their enrichment data

Cache Management

• 30-day automatic expiration
• Stored in user-specific directories
• Can be manually cleared

API Key Security

• All keys stored as HuggingFace Secrets
• Never exposed in logs or responses
• Separate keys per API provider

Performance Optimization

Caching Strategy

1. Check cache first (instant response)
2. If cache miss, call API
3. Store result in cache
4. Return data

Fallback Logic

Providers = [Primary, Secondary, Tertiary, Fallback]
for provider in Providers:
    try:
        data = provider.fetch()
        if data: return data
    except:
        continue
return None

Rate Limiting

• 30-second timeout per request
• Automatic retry on failures
• Respects API provider limits

Error Handling

All methods return None on failure and log errors:

try:
    result = manager.enrich_linkedin_profile(url)
    if result:
        # Success
    else:
        # Failed - check logs
except Exception as e:
    # Handle exception
    logger.error(f"Enrichment failed: {e}")

Integration Examples

With Contact Management

def create_contact(user, contact_data):
    # Create contact
    contact = Contact.create(**contact_data)
    
    # Enrich asynchronously
    manager = DataEnrichmentManager(user.username)
    enriched = manager.enrich_contact_full(contact_data)
    
    # Store enrichment
    contact.enrichment_data = enriched
    contact.save()
    
    return contact

With Outreach Agent

def prepare_outreach(contact):
    # Enrich contact before outreach
    manager = DataEnrichmentManager(current_user)
    enriched = manager.enrich_contact_full({
        'email': contact.email,
        'linkedin_url': contact.linkedin_url,
        'domain': contact.company_domain
    })
    
    # Use enriched data for personalization
    if enriched['total_sources'] > 0:
        contact.enriched = True
        contact.enrichment_score = enriched['total_sources']
    
    return enriched

Troubleshooting

Common Issues

Issue: No data returned

• Check API keys are configured
• Verify network connectivity
• Check rate limits

Issue: Slow responses

• Cache may be cold (first request)
• API provider may be slow
• Check timeout settings

Issue: Missing enrichments

• Some providers may not have data
• Check logs for API errors
• Verify input data format

Debug Mode

import logging
logging.basicConfig(level=logging.DEBUG)

manager = DataEnrichmentManager(username)
result = manager.enrich_contact_full(contact_data)
# Check logs for detailed information

API Provider Details

RapidAPI Configuration

All APIs are accessed through RapidAPI:

• Host: *.p.rapidapi.com
• Authentication: x-rapidapi-key header
• Format: JSON
• Protocol: HTTPS

Provider URLs

• LinkedIn Fresh: fresh-linkedin-profile-data.p.rapidapi.com
• LinkedIn RockAPIs: linkedin-data-api.p.rapidapi.com
• LI Data Scraper: li-data-scraper.p.rapidapi.com
• LinkedIn Scraper: linkedin-data-scraper.p.rapidapi.com
• Apollo.io: apollo-io-no-cookies-required.p.rapidapi.com
• Mediafy: mediafy-api.p.rapidapi.com
• Twitter: twitter241.p.rapidapi.com
• Facebook: facebook-scraper3.p.rapidapi.com
• Google: google-search74.p.rapidapi.com
• LetScrape Business: local-business-data.p.rapidapi.com
• LetScrape Contacts: website-contacts-scraper.p.rapidapi.com

Changelog

Version 2.0.0 (2025-11-06)

• ✨ Added LetScrape Local Business Data API
• ✨ Added LetScrape Website Contacts Scraper API
• 🔄 Enhanced fallback logic
• 📚 Comprehensive documentation
• 🐛 Bug fixes and improvements

Version 1.0.0 (2025-11-05)

• 🎉 Initial release
• LinkedIn enrichment (4 providers)
• Apollo.io integration
• Social media enrichment
• Caching system
• Per-user data isolation

Support & Contact

For issues or questions:

• Check logs in /data/users/{username}/enrichment_cache/
• Review API provider documentation
• Contact StrategyStack support

---

Last Updated: November 6, 2025
Version: 2.0.0
Maintainer: StrategyStack Team