Commit History
-
style: remove unused import
Karim shoair10ff0d5 -
fix: update code and docstrings to remove the old google referer logic
Karim shoaira28879b -
build: pump up deps and browsers versions
Karim shoair7bd4753 -
fix: return bare Google URL in referer instead of search query
Real browsers send `https://www.google.com/` as the Referer header when clicking search results, not the full search URL with query parameters. The previous format was a fingerprinting signal that the referer was spoofed. Closes #1728b89653
-
fix(browsers): make flag concatenation type-safe
Karim shoair +1: Christian RostBetter implementation of #157 that's accepted by pyright/mypy. Thanks to @rostchri . Co-Authored-By: Christian Rost <chr@baltic-online.de>814b9a6
-
fix(stealth): improve stealth mode by removing unnecessary scripts
Karim shoairIt doesn't add anything to the table as before (it might turn out I was wrong and add them again lol)cf06b6e -
fix(solver): Solve CF faster and handle websites that show captcha twice
Karim shoair446b9d5
-
fix: improve regex precision for Cloudflare challenge detection
RinZ27979ef6f
-
fix: make browser fetchers type hints always present
Karim shoair7d2ea72 -
fix: use more accurate naming for proxy rotation strategy
Karim shoairIt can be considered the simplest form of round robin since we don't have weights or anything (as was originally planned), but let's change it to avoid confusion or useless debates. Here goes nothing61a910a
-
style: Use shorter and more accurate naming for constants
Karim shoairc2dbf4c -
style: Fix all mypy errors and add type hints to untyped function bodies
Karim shoair**Resolved all 65 mypy errors across 14 files and added type annotations to all previously untyped function bodies. Final result: 0 errors with --check-untyped-defs enabled, all 454 tests pass.** `scrapling/core/_types.py` - Removed broken Self = object fallback — now requires typing_extensions for Python < 3.11 `scrapling/core/storage.py` - Fixed str/bytes mismatch in _get_hash() — used separate _identifier_bytes variable instead of reassigning from str to bytes `scrapling/core/custom_types.py` - split() return type: Union[List, "TextHandlers"] → list[Any] (avoids LSP violation with parent list[str]) - format() kwargs: **kwargs: str → **kwargs: object (matches parent str.format signature) - AttributesHandler.__init__: Added mapping: Any = None, **kwargs: Any and -> None - json_string property: Added -> bytes return type `scrapling/core/mixins.py` - Changed self: "Selector" to self: Any on all mixin methods (mypy can't handle forward-reference self types on non-su…31c2447
-
feat(browsers): Add option to block requests to specific domains
Karim shoair47dd985 -
fix(response): Force the body of the response to always be bytes
Karim shoair1b8cc6b
-
feat(proxy control): Force a proxy at request level at any given point
Karim shoairAnd merge request's meta with response's meta32daccc
-
feat(spiders/fetchers): Adding proxy rotation logic and change retry logic
Karim shoair- User passes a single proxy to the browser session, and it will keep using the same context. Pass a proxy manager that automatically refreshes the IP with each proxy, and you get speed but sacrifice some stealth. - User imports the proxy rotator class and passes proxies to it and a rotation strategy (Round robin by default). Then pass the instance to the session class, which will create a context and a tab for each proxy returned by the rotator. This way, you sacrifice speed a bit, but you get maximum stealth since each context is created with the proxy that will be used. - Normal requests use what you pass without issues, of course. - All errors are retried now, and proxies are rotated on retries automatically.ed96cdc
-
fix(browsers): Improve stealth and speed by adjusting flags
Karim shoair88b6ab5 -
fix(browsers): disable autoplay for all browsers
Karim shoair1d04135
-
feat(parser): replacing `tldextract` with `tld` library
Karim shoairThis might break the adaptive data users have for websites BUT: 1. tld uses ~3.7x less memory during extraction operations (1.5 MB vs 5.7 MB). 2. tld uses ~56% less memory on import (5.2 MB vs 11.9 MB). 3. Zero dependencies (vs 3 for tldextract). In return, it's 30ms slower for extracting 5000 URLs, which is negligible. Also, the type hints aren't always accurate, but it's fine; I corrected them.e5733cb
-
style: update post requests type hints to be more accurate
Karim shoairc3c5ac6
-
fix(browsers): solving an issues with leaving playwright loop open when cdp connection fails
Karim shoairThis has been causing issues with tests for a long time, and now finally found the reason.8ff23b3
-
feat(browsers): Add option to retry tabs that gives errors
Karim shoair024cbba -
fix(browsers): Close pages that give error
Karim shoairThis was causing the browser and the whole code to stand still.8209e61 -
feat(response): Option to set the referer as the previous response url.
Karim shoairec44d0b -
fix: handle a bug with response follow referer flow argument
Karim shoair9b0f6b7 -
fix(browsers): solve a bug with setting referer on the request level
Karim shoairc7e573a
-
feat(spiders): Add follow function to the response
Karim shoairc0f5705
-
refactor: internal API changes
Karim shoairbbd0c34 -
fix: add missing line
Karim shoairfff357e -
refactor: internal API changes to be easily used as indicators for spiders
Karim shoair1803348
-
fix(StealthyFetcher): disable incognito mode to solve #123
Karim shoairfa0e2ca
-
docs: change docstring style for some methods
Karim shoairdaaad4e
-
fix(async stealth): wait the coroutine of the bounding box
Karim shoair1f369fd -
fix(fingerprints): do not create google referer for IPs
Karim shoairb354df4
-
build: update deps
Karim shoairand docstrings7e1bee4
-
docs: shorten docstrings for some functions
Karim shoair1812d2b
-
fix: remove forgotten unused code
Karim shoaira7cc20d -
fix: update the cookies type hint for the mcp server
Karim shoaird391239 -
fix: remove old proxy logic
Karim shoairWhen I was caching it with the rest of the dataf62e5ea -
style: removing unused code
Karim shoairf4f8cc5 -
fix: update shell signatures for the autocompletion
Karim shoairAdded missing fields and reordered classes as well for easier comparisons05b2188
-
refactor(fetchers)!: Replace Camoufox with patchright and many optimizations
Karim shoair- DynamicFetcher became 20% faster - StealthyFetcher became 99% faster - Scrapling size decreased - Code became ~400 lines shorter - Most importantly, scrapling is more stable and reliable now. - Less confusing for new users. - More...ee2299e
-
fix: improve the handling of useragent in chrome fetcher
Karim shoair123011a -
fix: Improve stealth of dynamic browser by pinning Chrome's UA version
Karim shoairf5018fb
-
docs: add docs for `timezone_id` argument
Karim shoair17e25c2 -
feat(dynamicSession): add the option to set timezone of the browser
Karim shoair1d52f39
-
refactor(fetchers): rename internal api
Karim shoairTo make it easier to use to use sessions outside `with` contexta17a010
-
fix(fetchers): better handling for infinite `network_idle`
Karim shoairc64a120
-
docs: fix docstrings for all requests methods
Karim shoair33c6b04 -
fix(fetcher): Remove non-keywords arguments
Karim shoair3f07d06