Optimize connection pool and concurrency to prevent target server crashes

Increase max_workers to 200 for extreme concurrency

Increase max_workers to 50 for faster parallel crawling

Increase HTTP timeouts to prevent fallback to slow Playwright

Improve URL normalization logic and add API docs

Fix silent failure on blocked crawls

Make playwright-stealth optional and fix import errors

Migrate API to FastAPI with async job queue and progress tracking

Fix Import Error: Explicitly install playwright-stealth in Dockerfile and log actual import errors

Enable Verbose Logging: Track all WAF interactions, stealth steps, and navigation events for debugging

Update Stealth: Use playwright-stealth library and Frame-based checkbox solver for improved WAF evasion

Update Stealth Strategy: Hardcode Chrome UA, mock plugins/permissions/languages, and randomize mouse movements

Update Crawler: Add stealth args and challenge detection for Cloudflare

Enhance API stability: Add KeepAlive, increase timeout, and optimize crawler threads

Fix URL normalization: strip trailing slash from start_url

Fix Amazon crawling: Add Start URL Retry logic and more block keywords

Fix crawl evasion: Add WAF block detection to Tier 1 fetch

Fix syntax errors: Restore init, fix indentation, remove invalid arg

Ultra-fast crawler update: Caching, SSL disable, and Thread tuning

Optimize crawler speed: BFS + Reduced delays

Fix syntax error in crawler

Optimize memory usage to prevent crashes

Fix crawling for eminentcoders.com

Initial commit for deployment