style: Fix all mypy errors and add type hints to untyped function bodies
Karim shoair
31c2447
**Resolved all 65 mypy errors across 14 files and added type annotations to all previously untyped function bodies. Final result: 0 errors with --check-untyped-defs enabled, all 454 tests pass.**
`scrapling/core/_types.py`
- Removed broken Self = object fallback — now requires typing_extensions for Python < 3.11
`scrapling/core/storage.py`
- Fixed str/bytes mismatch in _get_hash() — used separate _identifier_bytes variable instead of reassigning from str to bytes
`scrapling/core/custom_types.py`
- split() return type: Union[List, "TextHandlers"] → list[Any] (avoids LSP violation with parent list[str])
- format() kwargs: **kwargs: str → **kwargs: object (matches parent str.format signature)
- AttributesHandler.__init__: Added mapping: Any = None, **kwargs: Any and -> None
- json_string property: Added -> bytes return type
`scrapling/core/mixins.py`
- Changed self: "Selector" to self: Any on all mixin methods (mypy can't handle forward-reference self types on non-su…
feat(browsers): Add option to block requests to specific domains
Karim shoair
47dd985
feat(spiders/fetchers): Adding proxy rotation logic and change retry logic
Karim shoair
ed96cdc
- User passes a single proxy to the browser session, and it will keep using the same context. Pass a proxy manager that automatically refreshes the IP with each proxy, and you get speed but sacrifice some stealth.
- User imports the proxy rotator class and passes proxies to it and a rotation strategy (Round robin by default). Then pass the instance to the session class, which will create a context and a tab for each proxy returned by the rotator. This way, you sacrifice speed a bit, but you get maximum stealth since each context is created with the proxy that will be used.
- Normal requests use what you pass without issues, of course.
- All errors are retried now, and proxies are rotated on retries automatically.
feat(browsers): Add option to retry tabs that gives errors
Karim shoair
024cbba
fix: remove forgotten unused code
Karim shoair
a7cc20d
fix: remove old proxy logic
Karim shoair
f62e5ea
When I was caching it with the rest of the data
fix: update shell signatures for the autocompletion
Karim shoair
05b2188
Added missing fields and reordered classes as well for easier comparisons
refactor(fetchers)!: Replace Camoufox with patchright and many optimizations
Karim shoair
ee2299e
- DynamicFetcher became 20% faster
- StealthyFetcher became 99% faster
- Scrapling size decreased
- Code became ~400 lines shorter
- Most importantly, scrapling is more stable and reliable now.
- Less confusing for new users.
- More...
feat(dynamicSession): add the option to set timezone of the browser
Karim shoair
1d52f39
refactor(browser fetchers): Make all the type hints dynamic + Faster validation
Karim shoair
c908f33
Made the code shorter by an additional ~200 lines and easier to maintain in return for making the arguments autocompletion bad for shells that don't check for dynamic type hints like IPython.
fix(validators): Fix overriding session values with request ones
Karim shoair
4836205
pref(requests): Optimizing memory by specifiying slots
Karim shoair
624e46f
refactor(validation): All browser based fetchers are ~8-15% faster now in most cases
Karim shoair
8bf21c0
feat(DynamicSession): New option to add extra browser flags
Karim shoair
b6969b2
feat(Dynamic fetcher): Add `additional_args` option to customize browser as in StealthyFetcher
Karim shoair
0de8025
feat(fetchers): Add argument to control the user data directory
Karim shoair
c61a805
refactor: Making the requests logic better and cleaner
Karim shoair
906b02a
Also, solve some bugs along the way and a cleaner fix for #97
refactor: Making all the codebase acceptable by PyRight
Karim shoair
e5ecf76
Also fixes #97
perf: Improving all browser-based fetches by ~15%
Karim shoair
da46469
refactor: Make all fetchers as an optional dependency group
Karim shoair
a85d2c8
+ Removing some dead code
feat(browser fetchers): A new option to control page JS
Karim shoair
66fd35f
style: applying the new ruff rules to all files
Karim shoair
226b463
feat(DynamicFetcher): add the ability to add `init_script` to context (#56)
Karim shoair
03de577
feat(StealthyFetcher): add the ability to add `init_script` to context
Karim shoair
0ed7f13
style: A lot of type hints correction
Karim shoair
916182a
Since we are using Py3.10 as minimum version now, we remove Union when possible
style: replacing `os` with `Pathlib` and small optimizations
Karim shoair
11165c4
style: General type hints fixes and imports optimizing
Karim shoair
1d98b51
refactor: huge change, many features/class got a better naming
Karim shoair
8e67a4c
- `Adaptor` became `Selector`
- `Adaptors` became `Selectors`
- `auto_match` argument/feature became `adaptive`
- `adaptor_arguments` argument became `selector_config`
- `automatch_domain` argument became `adaptive_domain`
- `additional_arguments` argument became `additional_args`
- `storage_adaptors` file became just `storage`
refactor(StealthyFetcher): Remove virtual mode and use persistent context
Karim shoair
8400659
Solves #64 completely too
refactor: optimize imports and docstrings correction
