Spaces:
Sleeping
Sleeping
|
Download apps/api/src/scraper/WebScraper/utils/ENGINE_FORCING.md from s-eto-hub/firecrawl-api: direct link, hf CLI and curl.
- Browser
- Download file 4.07 kB
-
https://huggingface.co/spaces/s-eto-hub/firecrawl-api/resolve/main/apps/api/src/scraper/WebScraper/utils/ENGINE_FORCING.md
- Command line
-
hf download hf://spaces/s-eto-hub/firecrawl-api/apps/api/src/scraper/WebScraper/utils/ENGINE_FORCING.md
-
curl -L -o ENGINE_FORCING.md https://huggingface.co/spaces/s-eto-hub/firecrawl-api/resolve/main/apps/api/src/scraper/WebScraper/utils/ENGINE_FORCING.md
4.07 kB
Engine Forcing
This feature allows you to force specific scraping engines for certain domains based on URL patterns. This is useful when you know that certain websites work better with specific engines.
Configuration
The engine forcing is configured via the FORCED_ENGINE_DOMAINS environment variable. This should be a JSON object mapping domain patterns to engines.
Environment Variable Format
{
"example.com": "playwright",
"test.com": "fetch",
"*.subdomain.com": "fire-engine;chrome-cdp",
"google.com": ["fire-engine;chrome-cdp", "playwright"]
}
Supported Patterns
- Exact domain match:
"example.com"matchesexample.comand all its subdomains (www.example.com,api.example.com, etc.) - Wildcard pattern:
"*.subdomain.com"matches only subdomains ofsubdomain.com(e.g.,api.subdomain.com,www.subdomain.com) but NOT the base domain itself - Single engine:
"playwright"forces a single engine - Multiple engines:
["fire-engine;chrome-cdp", "playwright"]provides a fallback list of engines to try in order
Available Engines
fire-engine;chrome-cdp- Advanced browser with Chrome DevTools Protocolfire-engine;tlsclient- TLS fingerprinting for anti-bot bypassfire-engine;chrome-cdp;stealth- Chrome CDP with stealth modefire-engine;tlsclient;stealth- TLS client with stealth modeplaywright- Direct Playwright integrationfetch- Simple HTTP requestspdf- PDF document parsingdocument- Office document handling
How It Works
- When a scrape request is made, the system checks if the URL matches any domain pattern in
FORCED_ENGINE_DOMAINS - If a match is found, the specified engine(s) are used instead of the default engine selection logic
- If no match is found, the normal engine selection waterfall is used
- The engine forcing only applies if
forceEngineis not already set in the internal options
Example Configuration
Example 1: Force Playwright for specific domains
export FORCED_ENGINE_DOMAINS='{"linkedin.com":"playwright","twitter.com":"playwright"}'
This forces Playwright for LinkedIn and Twitter URLs.
Example 2: Use fetch for simple sites
export FORCED_ENGINE_DOMAINS='{"example.com":"fetch","httpbin.org":"fetch"}'
This uses the simple fetch engine for example.com and httpbin.org.
Example 3: Complex configuration with wildcards
export FORCED_ENGINE_DOMAINS='{
"google.com": ["fire-engine;chrome-cdp", "playwright"],
"*.cloudflare.com": "fire-engine;tlsclient;stealth",
"wikipedia.org": "fetch"
}'
This configuration:
- Uses fire-engine with Chrome CDP for Google, falling back to Playwright if needed
- Uses fire-engine with TLS client in stealth mode for Cloudflare subdomains
- Uses simple fetch for Wikipedia
Implementation Details
The engine forcing logic is implemented in:
apps/api/src/scraper/WebScraper/utils/engine-forcing.ts- Core logicapps/api/src/scraper/scrapeURL/index.ts- Integration into scraping pipeline
The system is initialized at startup in:
apps/api/src/index.ts- Main API serverapps/api/src/services/queue-worker.ts- Queue workerapps/api/src/services/extract-worker.ts- Extract workerapps/api/src/services/worker/nuq-worker.ts- NuQ worker
Precedence
The engine forcing has the following precedence:
- If
forceEngineis already set inInternalOptions, it takes precedence (engine forcing is skipped) - If a URL matches an engine forcing pattern, that engine is used
- Otherwise, the normal engine selection waterfall is used
Testing
Unit tests are available in apps/api/src/scraper/WebScraper/utils/__tests__/engine-forcing.test.ts.
Notes
- Domain matching is case-insensitive
- The system handles invalid URLs gracefully by returning
undefined - If the JSON in
FORCED_ENGINE_DOMAINSis invalid, the system logs an error and continues with empty mappings - The feature is similar to the blocked domains logic but for engine selection instead of blocking