Improve URL normalization logic and add API docs

Fix silent failure on blocked crawls

Make playwright-stealth optional and fix import errors

Migrate API to FastAPI with async job queue and progress tracking

Fix Import Error: Explicitly install playwright-stealth in Dockerfile and log actual import errors

Enable Verbose Logging: Track all WAF interactions, stealth steps, and navigation events for debugging

Update Stealth: Use playwright-stealth library and Frame-based checkbox solver for improved WAF evasion

Update Stealth Strategy: Hardcode Chrome UA, mock plugins/permissions/languages, and randomize mouse movements

Update Crawler: Add stealth args and challenge detection for Cloudflare

Enhance API stability: Add KeepAlive, increase timeout, and optimize crawler threads

Fix URL normalization: strip trailing slash from start_url

Fix Amazon crawling: Add Start URL Retry logic and more block keywords

Fix crawl evasion: Add WAF block detection to Tier 1 fetch

Fix syntax errors: Restore init, fix indentation, remove invalid arg

Ultra-fast crawler update: Caching, SSL disable, and Thread tuning

Optimize crawler speed: BFS + Reduced delays

Fix syntax error in crawler

Optimize memory usage to prevent crashes

Fix crawling for eminentcoders.com

Initial commit for deployment