Spaces:
Sleeping
Sleeping
|
Download apps/api/src/scraper/scrapeURL/README.md from s-eto-hub/firecrawl-api: direct link, hf CLI and curl.
- Browser
- Download file 1.21 kB
-
https://huggingface.co/spaces/s-eto-hub/firecrawl-api/resolve/main/apps/api/src/scraper/scrapeURL/README.md
- Command line
-
hf download hf://spaces/s-eto-hub/firecrawl-api/apps/api/src/scraper/scrapeURL/README.md
-
curl -L -o README.md https://huggingface.co/spaces/s-eto-hub/firecrawl-api/resolve/main/apps/api/src/scraper/scrapeURL/README.md
1.21 kB
| # `scrapeURL` | |
| New URL scraper for Firecrawl | |
| ## Signal flow | |
| ```mermaid | |
| flowchart TD; | |
| scrapeURL-.->buildFallbackList; | |
| buildFallbackList-.->scrapeURLWithEngine; | |
| scrapeURLWithEngine-.->parseMarkdown; | |
| parseMarkdown-.->wasScrapeSuccessful{{Was scrape successful?}}; | |
| wasScrapeSuccessful-."No".->areEnginesLeft{{Are there engines left to try?}}; | |
| areEnginesLeft-."Yes, try next engine".->scrapeURLWithEngine; | |
| areEnginesLeft-."No".->NoEnginesLeftError[/NoEnginesLeftError/] | |
| wasScrapeSuccessful-."Yes".->asd; | |
| ``` | |
| ## Differences from `WebScraperDataProvider` | |
| - The job of `WebScraperDataProvider.validateInitialUrl` has been delegated to the zod layer above `scrapeUrl`. | |
| - `WebScraperDataProvider.mode` has no equivalent, only `scrape_url` is supported. | |
| - You may no longer specify multiple URLs. | |
| - Built on `v1` definitons, instead of `v0`. | |
| - PDFs are now converted straight to markdown using LlamaParse, instead of converting to just plaintext. | |
| - DOCXs are now converted straight to HTML (and then later to markdown) using mammoth, instead of converting to just plaintext. | |
| - Using new JSON Schema OpenAI API -- schema fails with LLM Extract will be basically non-existant. | |