fix(spiders): corrections and removal of dead code
Karim shoair
c923d18
style(spiders): Lower the default number of concurrent requests
Karim shoair
288b1a4
style: Fix all mypy errors and add type hints to untyped function bodies
Karim shoair
31c2447
**Resolved all 65 mypy errors across 14 files and added type annotations to all previously untyped function bodies. Final result: 0 errors with --check-untyped-defs enabled, all 454 tests pass.**
`scrapling/core/_types.py`
- Removed broken Self = object fallback — now requires typing_extensions for Python < 3.11
`scrapling/core/storage.py`
- Fixed str/bytes mismatch in _get_hash() — used separate _identifier_bytes variable instead of reassigning from str to bytes
`scrapling/core/custom_types.py`
- split() return type: Union[List, "TextHandlers"] → list[Any] (avoids LSP violation with parent list[str])
- format() kwargs: **kwargs: str → **kwargs: object (matches parent str.format signature)
- AttributesHandler.__init__: Added mapping: Any = None, **kwargs: Any and -> None
- json_string property: Added -> bytes return type
`scrapling/core/mixins.py`
- Changed self: "Selector" to self: Any on all mixin methods (mypy can't handle forward-reference self types on non-su…
feat(proxy control): Force a proxy at request level at any given point
Karim shoair
32daccc
And merge request's meta with response's meta
fix(requests): return the cached fingerprint on second call
Karim shoair
e30685e
feat(spiders/requests): Use a reliable method for fingerprint generation
Karim shoair
e227110
feat(spiders): Make graceful shutdown always enabled
Karim shoair
5ec9f75
fix(spiders): handle errors with manual pause during stream mode
Karim shoair
47633d8
fix(spiders): fix a bug with session stats counter
Karim shoair
98a7a0d
fix(spiders): Fix for when the pausing system has large queue
Karim shoair
59117ec
feat(spiders): Add pause/resume system for crawls
Karim shoair
1721cbb
style(spiders): make parse function abstract
Karim shoair
c1fb967
feat(spiders): Change items hook to make it for processing items + add a stat for this
Karim shoair
01127e9
fix(spiders engine): Improve logging
Karim shoair
168160d
feat(spiders): Add option to logging format and date format
Karim shoair
77ddca9
feat: add hook to run after collecting an item
Karim shoair
48cc7a2
feat(spiders): Add a stream mode to collect items in realtime
Karim shoair
7145885
feat(spiders): The option to start sessions lazily
Karim shoair
aa7527f
style: remove unwanted import from spiders main init file
Karim shoair
d1c2b16
feat(spiders): add simple export system for the results
Karim shoair
c5ebc9d
fix: make `retry_blocked_request` take the response as well
Karim shoair
e251ff4
feat(spiders): Add logic to detect blocked requests and retry them
Karim shoair
337bce0
style(spiders): make the usage of uvloop optional
Karim shoair
be76f99
feat: spiders use uvloop/winloop when available
Karim shoair
3068bf6
fix(browsers): Close pages that give error
Karim shoair
8209e61
This was causing the browser and the whole code to stand still.
feat(spiders system): a prototype of the new spiders system
Karim shoair
f8f7469
- A modern spider design that uses AnyIO and asyncio, yet it's very similar to Scrapy spiders API because it's the easiest design for users, and to make it easier for new users.
- Spiders can have multiple sessions per crawl, and users decide which session to use with each request.
- A scheduler system that uses heapq logic.
- The user can set the number of concurrent requests for a spider globally or per domain.
- The user can set a download delay to control the speed of the spider more.
- There's a global function that can be overridden to handle errors for all requests. (Similar to errback in scrapy).
- There's a spider argument to set the allowed domains for the spider to stay in.
- Each spider has a very detailed crawl stats that can be accessed right away from the code after the crawl finishes. Same case with scraped items.
- The whole spider as written as any other script and you just run it. No command-line arguments, and no need to run it from the terminal through the library …
