Spaces:
Sleeping
Sleeping
|
Download apps/api/src/scraper/scrapeURL/README.md from s-eto-hub/firecrawl-api: direct link, hf CLI and curl.
- Browser
- Download file 1.21 kB
-
https://huggingface.co/spaces/s-eto-hub/firecrawl-api/resolve/main/apps/api/src/scraper/scrapeURL/README.md
- Command line
-
hf download hf://spaces/s-eto-hub/firecrawl-api/apps/api/src/scraper/scrapeURL/README.md
-
curl -L -o README.md https://huggingface.co/spaces/s-eto-hub/firecrawl-api/resolve/main/apps/api/src/scraper/scrapeURL/README.md
1.21 kB
scrapeURL
New URL scraper for Firecrawl
Signal flow
flowchart TD;
scrapeURL-.->buildFallbackList;
buildFallbackList-.->scrapeURLWithEngine;
scrapeURLWithEngine-.->parseMarkdown;
parseMarkdown-.->wasScrapeSuccessful{{Was scrape successful?}};
wasScrapeSuccessful-."No".->areEnginesLeft{{Are there engines left to try?}};
areEnginesLeft-."Yes, try next engine".->scrapeURLWithEngine;
areEnginesLeft-."No".->NoEnginesLeftError[/NoEnginesLeftError/]
wasScrapeSuccessful-."Yes".->asd;
Differences from WebScraperDataProvider
- The job of
WebScraperDataProvider.validateInitialUrlhas been delegated to the zod layer abovescrapeUrl. WebScraperDataProvider.modehas no equivalent, onlyscrape_urlis supported.- You may no longer specify multiple URLs.
- Built on
v1definitons, instead ofv0. - PDFs are now converted straight to markdown using LlamaParse, instead of converting to just plaintext.
- DOCXs are now converted straight to HTML (and then later to markdown) using mammoth, instead of converting to just plaintext.
- Using new JSON Schema OpenAI API -- schema fails with LLM Extract will be basically non-existant.