Spaces:
Sleeping
Sleeping
|
Download docs/DATA_ENRICHMENT.md from StrategyStack/README: direct link, hf CLI and curl.
- Browser
- Download file 12 kB
-
https://huggingface.co/spaces/StrategyStack/README/resolve/main/docs/DATA_ENRICHMENT.md
- Command line
-
hf download hf://spaces/StrategyStack/README/docs/DATA_ENRICHMENT.md
-
curl -L -o DATA_ENRICHMENT.md https://huggingface.co/spaces/StrategyStack/README/resolve/main/docs/DATA_ENRICHMENT.md
12 kB
| # Data Enrichment Module Documentation | |
| ## Overview | |
| The `data_enrichment.py` module provides comprehensive contact and lead enrichment capabilities for the StrategyStack platform. It integrates 14 different data sources through RapidAPI to collect, enrich, and cache contact information. | |
| ## Features | |
| ### π Multi-Provider Integration | |
| - **LinkedIn APIs** (4 providers with fallback) | |
| - **Apollo.io** - B2B contact enrichment | |
| - **Social Media** - Twitter & Facebook profile data | |
| - **Mediafy API** - Media and social data | |
| - **Google Search** - Web search enrichment | |
| - **LetScrape** - Local business data & website contacts | |
| ### πΎ Intelligent Caching | |
| - Per-user data isolation (`/data/users/{username}/enrichment_cache/`) | |
| - 30-day cache TTL | |
| - Automatic cache invalidation | |
| - Privacy-first design - users own their data | |
| ### π Fallback & Redundancy | |
| - Automatic failover between providers | |
| - Graceful error handling | |
| - Partial data return on API failures | |
| ## Architecture | |
| ``` | |
| DataEnrichmentManager | |
| βββ API Keys Management (11 secrets) | |
| βββ Request Handler (rate limiting, retries) | |
| βββ Cache Manager (read/write/invalidate) | |
| βββ Enrichment Methods | |
| βββ LinkedIn Profile | |
| βββ Company Data | |
| βββ Apollo.io | |
| βββ Social Media | |
| βββ Local Business | |
| βββ Website Contacts | |
| ``` | |
| ## Installation & Setup | |
| ### 1. Environment Variables | |
| Add the following secrets to your Hugging Face Space: | |
| ```bash | |
| RAPIDAPI_KEY_LINKEDIN_FRESH | |
| RAPIDAPI_KEY_LINKEDIN_ROCKAPIS | |
| RAPIDAPI_KEY_LISCRAPER | |
| RAPIDAPI_KEY_LINKEDIN_SCRAPER | |
| RAPIDAPI_KEY_APOLLO | |
| RAPIDAPI_KEY_MEDIAFY | |
| RAPIDAPI_KEY_TWITTER | |
| RAPIDAPI_KEY_FACEBOOK | |
| RAPIDAPI_KEY_GOOGLE | |
| RAPIDAPI_KEY_LETSCRAPE | |
| ``` | |
| ### 2. Import Module | |
| ```python | |
| from data_enrichment import DataEnrichmentManager, enrich_contact | |
| # Initialize manager | |
| manager = DataEnrichmentManager(username="user123") | |
| ``` | |
| ## Usage Examples | |
| ### Basic Contact Enrichment | |
| ```python | |
| # Enrich a single contact | |
| contact_data = { | |
| 'name': 'John Doe', | |
| 'email': 'john@example.com', | |
| 'linkedin_url': 'https://linkedin.com/in/johndoe', | |
| 'domain': 'example.com' | |
| } | |
| result = manager.enrich_contact_full(contact_data) | |
| print(f"Enriched from {result['total_sources']} sources") | |
| print(f"Sources used: {result['sources']}") | |
| ``` | |
| ### LinkedIn Profile Enrichment | |
| ```python | |
| # Get LinkedIn profile data (with fallback) | |
| profile = manager.enrich_linkedin_profile( | |
| 'https://linkedin.com/in/johndoe' | |
| ) | |
| if profile: | |
| print(f"Name: {profile.get('name')}") | |
| print(f"Title: {profile.get('title')}") | |
| print(f"Company: {profile.get('company')}") | |
| ``` | |
| ### Company Enrichment | |
| ```python | |
| # Enrich company by domain | |
| company = manager.enrich_company('example.com') | |
| if company: | |
| print(f"Company: {company.get('name')}") | |
| print(f"Industry: {company.get('industry')}") | |
| print(f"Size: {company.get('employee_count')}") | |
| ``` | |
| ### LetScrape: Local Business Data | |
| ```python | |
| # Search for local businesses | |
| businesses = manager.scrape_local_business( | |
| query='pizza restaurants', | |
| location='New York, NY', | |
| limit=20 | |
| ) | |
| for business in businesses.get('data', []): | |
| print(f"Name: {business.get('name')}") | |
| print(f"Address: {business.get('address')}") | |
| print(f"Phone: {business.get('phone')}") | |
| print(f"Rating: {business.get('rating')}") | |
| ``` | |
| ### LetScrape: Website Contacts | |
| ```python | |
| # Extract contacts from a website | |
| contacts = manager.scrape_website_contacts('example.com') | |
| if contacts: | |
| print(f"Emails: {contacts.get('emails', [])}") | |
| print(f"Phones: {contacts.get('phones', [])}") | |
| print(f"Social: {contacts.get('social_links', {})}") | |
| ``` | |
| ### Batch Enrichment | |
| ```python | |
| # Enrich multiple contacts | |
| contacts = [ | |
| {'name': 'John Doe', 'email': 'john@example.com'}, | |
| {'name': 'Jane Smith', 'domain': 'acme.com'} | |
| ] | |
| for contact in contacts: | |
| enriched = manager.enrich_contact_full(contact) | |
| print(f"Enriched {contact['name']} from {enriched['total_sources']} sources") | |
| ``` | |
| ## API Methods Reference | |
| ### Core Methods | |
| #### `__init__(username: str)` | |
| Initialize the enrichment manager for a specific user. | |
| **Parameters:** | |
| - `username` (str): User identifier for data isolation | |
| #### `enrich_contact_full(contact_data: Dict) -> Dict` | |
| Perform comprehensive enrichment using all available sources. | |
| **Parameters:** | |
| - `contact_data` (dict): Contact information | |
| - `name` (str, optional) | |
| - `email` (str, optional) | |
| - `linkedin_url` (str, optional) | |
| - `domain` (str, optional) | |
| - `company` (str, optional) | |
| - `location` (str, optional) | |
| - `twitter_username` (str, optional) | |
| - `facebook_id` (str, optional) | |
| **Returns:** | |
| ```python | |
| { | |
| 'original': {...}, # Original input data | |
| 'enrichments': { # Enriched data by source | |
| 'linkedin': {...}, | |
| 'apollo': {...}, | |
| 'company': {...}, | |
| 'website_contacts': {...}, | |
| 'local_business': {...} | |
| }, | |
| 'sources': ['linkedin', 'apollo', ...], # Sources used | |
| 'total_sources': 5, # Count of successful enrichments | |
| 'enriched_at': '2025-11-06T14:00:00' # Timestamp | |
| } | |
| ``` | |
| ### LinkedIn Methods | |
| #### `enrich_linkedin_profile(linkedin_url: str) -> Optional[Dict]` | |
| Enrich LinkedIn profile with automatic fallback across 4 providers. | |
| #### `enrich_company(domain: str) -> Optional[Dict]` | |
| Get company information from LinkedIn by domain. | |
| ### Apollo.io | |
| #### `enrich_with_apollo(email: str = None, domain: str = None) -> Optional[Dict]` | |
| Enrich contact using Apollo.io B2B database. | |
| ### Social Media | |
| #### `enrich_twitter_profile(username: str) -> Optional[Dict]` | |
| Get Twitter profile data. | |
| #### `enrich_facebook_profile(profile_id: str) -> Optional[Dict]` | |
| Get Facebook profile data. | |
| ### Search & Media | |
| #### `search_google(query: str, num_results: int = 10) -> Optional[Dict]` | |
| Perform Google search for contact information. | |
| #### `enrich_with_mediafy(query: str) -> Optional[Dict]` | |
| Get media and social data from Mediafy API. | |
| ### LetScrape Methods | |
| #### `scrape_local_business(query: str, location: str = None, limit: int = 20) -> Optional[Dict]` | |
| Scrape local business data from Google Maps. | |
| **Parameters:** | |
| - `query` (str): Business search query (e.g., "pizza restaurants") | |
| - `location` (str, optional): Geographic location | |
| - `limit` (int): Maximum results (default: 20) | |
| **Returns:** | |
| ```python | |
| { | |
| 'data': [ | |
| { | |
| 'name': 'Business Name', | |
| 'address': '123 Main St', | |
| 'phone': '+1234567890', | |
| 'email': 'contact@business.com', | |
| 'website': 'https://business.com', | |
| 'rating': 4.5, | |
| 'review_count': 150, | |
| 'category': 'Restaurant', | |
| 'hours': {...} | |
| } | |
| ] | |
| } | |
| ``` | |
| #### `scrape_website_contacts(domain: str) -> Optional[Dict]` | |
| Extract emails, phone numbers, and social links from a website. | |
| **Parameters:** | |
| - `domain` (str): Website domain (e.g., "example.com") | |
| **Returns:** | |
| ```python | |
| { | |
| 'emails': ['contact@example.com', 'info@example.com'], | |
| 'phones': ['+1234567890'], | |
| 'social_links': { | |
| 'facebook': 'https://facebook.com/example', | |
| 'linkedin': 'https://linkedin.com/company/example', | |
| 'twitter': 'https://twitter.com/example', | |
| 'instagram': 'https://instagram.com/example' | |
| } | |
| } | |
| ``` | |
| ### Utility Methods | |
| #### `get_enrichment_stats() -> Dict` | |
| Get cache statistics and API usage information. | |
| **Returns:** | |
| ```python | |
| { | |
| 'cache_directory': '/data/users/user123/enrichment_cache', | |
| 'cached_profiles': 45, | |
| 'cache_types': { | |
| 'linkedin': 15, | |
| 'company': 10, | |
| 'apollo': 12, | |
| 'website_contacts': 8 | |
| }, | |
| 'configured_apis': ['linkedin_fresh', 'apollo', 'letscrape'], | |
| 'total_apis': 10 | |
| } | |
| ``` | |
| ## Data Privacy & Security | |
| ### User Data Isolation | |
| - Each user has separate cache directory | |
| - No cross-user data access | |
| - Users own their enrichment data | |
| ### Cache Management | |
| - 30-day automatic expiration | |
| - Stored in user-specific directories | |
| - Can be manually cleared | |
| ### API Key Security | |
| - All keys stored as HuggingFace Secrets | |
| - Never exposed in logs or responses | |
| - Separate keys per API provider | |
| ## Performance Optimization | |
| ### Caching Strategy | |
| 1. Check cache first (instant response) | |
| 2. If cache miss, call API | |
| 3. Store result in cache | |
| 4. Return data | |
| ### Fallback Logic | |
| ```python | |
| Providers = [Primary, Secondary, Tertiary, Fallback] | |
| for provider in Providers: | |
| try: | |
| data = provider.fetch() | |
| if data: return data | |
| except: | |
| continue | |
| return None | |
| ``` | |
| ### Rate Limiting | |
| - 30-second timeout per request | |
| - Automatic retry on failures | |
| - Respects API provider limits | |
| ## Error Handling | |
| All methods return `None` on failure and log errors: | |
| ```python | |
| try: | |
| result = manager.enrich_linkedin_profile(url) | |
| if result: | |
| # Success | |
| else: | |
| # Failed - check logs | |
| except Exception as e: | |
| # Handle exception | |
| logger.error(f"Enrichment failed: {e}") | |
| ``` | |
| ## Integration Examples | |
| ### With Contact Management | |
| ```python | |
| def create_contact(user, contact_data): | |
| # Create contact | |
| contact = Contact.create(**contact_data) | |
| # Enrich asynchronously | |
| manager = DataEnrichmentManager(user.username) | |
| enriched = manager.enrich_contact_full(contact_data) | |
| # Store enrichment | |
| contact.enrichment_data = enriched | |
| contact.save() | |
| return contact | |
| ``` | |
| ### With Outreach Agent | |
| ```python | |
| def prepare_outreach(contact): | |
| # Enrich contact before outreach | |
| manager = DataEnrichmentManager(current_user) | |
| enriched = manager.enrich_contact_full({ | |
| 'email': contact.email, | |
| 'linkedin_url': contact.linkedin_url, | |
| 'domain': contact.company_domain | |
| }) | |
| # Use enriched data for personalization | |
| if enriched['total_sources'] > 0: | |
| contact.enriched = True | |
| contact.enrichment_score = enriched['total_sources'] | |
| return enriched | |
| ``` | |
| ## Troubleshooting | |
| ### Common Issues | |
| **Issue: No data returned** | |
| - Check API keys are configured | |
| - Verify network connectivity | |
| - Check rate limits | |
| **Issue: Slow responses** | |
| - Cache may be cold (first request) | |
| - API provider may be slow | |
| - Check timeout settings | |
| **Issue: Missing enrichments** | |
| - Some providers may not have data | |
| - Check logs for API errors | |
| - Verify input data format | |
| ### Debug Mode | |
| ```python | |
| import logging | |
| logging.basicConfig(level=logging.DEBUG) | |
| manager = DataEnrichmentManager(username) | |
| result = manager.enrich_contact_full(contact_data) | |
| # Check logs for detailed information | |
| ``` | |
| ## API Provider Details | |
| ### RapidAPI Configuration | |
| All APIs are accessed through RapidAPI: | |
| - Host: `*.p.rapidapi.com` | |
| - Authentication: `x-rapidapi-key` header | |
| - Format: JSON | |
| - Protocol: HTTPS | |
| ### Provider URLs | |
| - **LinkedIn Fresh**: `fresh-linkedin-profile-data.p.rapidapi.com` | |
| - **LinkedIn RockAPIs**: `linkedin-data-api.p.rapidapi.com` | |
| - **LI Data Scraper**: `li-data-scraper.p.rapidapi.com` | |
| - **LinkedIn Scraper**: `linkedin-data-scraper.p.rapidapi.com` | |
| - **Apollo.io**: `apollo-io-no-cookies-required.p.rapidapi.com` | |
| - **Mediafy**: `mediafy-api.p.rapidapi.com` | |
| - **Twitter**: `twitter241.p.rapidapi.com` | |
| - **Facebook**: `facebook-scraper3.p.rapidapi.com` | |
| - **Google**: `google-search74.p.rapidapi.com` | |
| - **LetScrape Business**: `local-business-data.p.rapidapi.com` | |
| - **LetScrape Contacts**: `website-contacts-scraper.p.rapidapi.com` | |
| ## Changelog | |
| ### Version 2.0.0 (2025-11-06) | |
| - β¨ Added LetScrape Local Business Data API | |
| - β¨ Added LetScrape Website Contacts Scraper API | |
| - π Enhanced fallback logic | |
| - π Comprehensive documentation | |
| - π Bug fixes and improvements | |
| ### Version 1.0.0 (2025-11-05) | |
| - π Initial release | |
| - LinkedIn enrichment (4 providers) | |
| - Apollo.io integration | |
| - Social media enrichment | |
| - Caching system | |
| - Per-user data isolation | |
| ## Support & Contact | |
| For issues or questions: | |
| - Check logs in `/data/users/{username}/enrichment_cache/` | |
| - Review API provider documentation | |
| - Contact StrategyStack support | |
| --- | |
| **Last Updated:** November 6, 2025 | |
| **Version:** 2.0.0 | |
| **Maintainer:** StrategyStack Team |