Commit History
-
🚀 Transform scraper into Lead Generation powerhouse Running
MAJOR FEATURES ADDED: 📧 Email extraction with regex pattern matching 📞 Phone number detection and formatting 🏢 Company name identification (multiple methods) 📍 Address extraction with US format regex 🔗 Social media profile discovery 📝 Contact form detection ⚡ Technology stack identification 🎯 Industry keyword classification 📊 Intelligent lead scoring system (0-100) LEAD SCORING ALGORITHM: - Emails found: +30 points - Phone numbers: +25 points - Contact forms: +20 points - Social media: +15 points - Address: +15 points - Company name: +10 points - Technologies: +10 points - Industry keywords: +5 points BUSINESS APPLICATIONS: - B2B lead generation - Sales prospecting - Market research - Competitor analysis - Contact discovery Example: /scrape?url=https://company.com&lead_generation=true35b24cc
-
Focus scraper on body content, links, and webpage images
Features: - Extract clean body text content (removes scripts/styles) - Get all meaningful links with text and URLs - Take full page screenshots (not just viewport) - Extract page title and meta description - Filter links to only include valid HTTP URLs - Limit link text to 200 characters for better performance - Changed parameter from get_content to get_body for clarity Example: /scrape?url=https://example.com&screenshot=true&get_links=true&get_body=true9e4b598 -
Add missing system dependencies for Playwright browsers
- Add libgbm1 to system dependencies - Run 'playwright install-deps' to install all required system libraries - Should resolve 'Host system is missing dependencies' error - This installs all dependencies needed for Chromium to run in container0540d86 -
Fix Playwright browser installation for appuser
- Install browsers as appuser instead of root - Add proper browser launch args for containerized environment - Add comprehensive logging for debugging - Should resolve 'Executable doesn't exist' error02554df -
Convert FastAPI to use GET requests for easier testing
- Changed /scrape endpoint from POST to GET - Added query parameters: url, screenshot, get_links, get_content - Added root endpoint (/) with API information and examples - Made get_content default to false to avoid large responses - Now testable directly in browser with URL parameters - Example: /scrape?url=https://example.com&screenshot=true05c92cc -
Switch to python:3.11-slim base image for better compatibility
- Replace zenika/alpine-chrome with standard Python image - Install all required system dependencies for Playwright - Install Playwright browsers during build - Create non-root user for security - Should resolve all import and compatibility issues9b0dc4d -
Fix Playwright module import error
- Add playwright back to requirements.txt (needed for Python import) - Set PLAYWRIGHT_BROWSERS_PATH to use existing browsers from base image - This should resolve 'No module named playwright' error - Playwright package provides Python API, browsers are from base image4abb273 -
Remove playwright from requirements.txt
- Playwright is already installed in the base image zenika/alpine-chrome:with-playwright - Avoids version conflicts and installation errors - Only install FastAPI, Uvicorn, and Pydantic which are needed1864ecb -
updated reuirements
67add6a -
Fix PEP 668 externally-managed-environment error
- Add --break-system-packages flag to pip install - This resolves the Alpine Linux PEP 668 restriction - Allows installation of Python packages in Docker container23a3391 -
Fix Docker permission issues
- Switch to root user for package installation - Use python3 explicitly instead of python - Switch back to chrome user for security - Should resolve apk permission denied errors41e802f -
Fix Python syntax errors and add requirements.txt
- Fixed all syntax errors in scrape.py (imports, indentation, class definitions) - Created requirements.txt with necessary dependencies - Cleaned up Dockerfile to remove redundant installations - Ready for Hugging Face deploymentd382ddb -
Initial commit
8b5a84c -
initial commit
Verified signature995d941