feat(proxy control): Force a proxy at request level at any given point
Karim shoair
32daccc
And merge request's meta with response's meta
feat(spiders/requests): Use a reliable method for fingerprint generation
Karim shoair
e227110
feat(spiders): Make graceful shutdown always enabled
Karim shoair
5ec9f75
fix(spiders): handle errors with manual pause during stream mode
Karim shoair
47633d8
fix(spiders): fix a bug with session stats counter
Karim shoair
98a7a0d
fix(spiders): Fix for when the pausing system has large queue
Karim shoair
59117ec
feat(spiders): Add pause/resume system for crawls
Karim shoair
1721cbb
feat(spiders): Change items hook to make it for processing items + add a stat for this
Karim shoair
01127e9
fix(spiders engine): Improve logging
Karim shoair
168160d
feat: add hook to run after collecting an item
Karim shoair
48cc7a2
feat(spiders): Add a stream mode to collect items in realtime
Karim shoair
7145885
feat(spiders): add simple export system for the results
Karim shoair
c5ebc9d
fix: make `retry_blocked_request` take the response as well
Karim shoair
e251ff4
feat(spiders): Add logic to detect blocked requests and retry them
Karim shoair
337bce0
fix(browsers): Close pages that give error
Karim shoair
8209e61
This was causing the browser and the whole code to stand still.
feat(spiders system): a prototype of the new spiders system
Karim shoair
f8f7469
- A modern spider design that uses AnyIO and asyncio, yet it's very similar to Scrapy spiders API because it's the easiest design for users, and to make it easier for new users.
- Spiders can have multiple sessions per crawl, and users decide which session to use with each request.
- A scheduler system that uses heapq logic.
- The user can set the number of concurrent requests for a spider globally or per domain.
- The user can set a download delay to control the speed of the spider more.
- There's a global function that can be overridden to handle errors for all requests. (Similar to errback in scrapy).
- There's a spider argument to set the allowed domains for the spider to stay in.
- Each spider has a very detailed crawl stats that can be accessed right away from the code after the crawl finishes. Same case with scraped items.
- The whole spider as written as any other script and you just run it. No command-line arguments, and no need to run it from the terminal through the library …
