fix: update code and docstrings to remove the old google referer logic
Karim shoair
a28879b
fix(type hints): use correct import for Python < 3.12 (Fixes #163 )
Karim shoair
1c774eb
fix(mcp): make mcp use less tokens by striping useless tags
Karim shoair
130d1d8
fix: complete MCP schema validation for remaining bare array types
Karim shoair
5f93804
- Fix params (Dict|List|Tuple → Dict), urls (Tuple[str,...] → List[str]),
- Renamed _NormalizeCredentials/_ContentTranslator to snake_case.
- Also, raise a ValueError on invalid credentials instead of returning None silently.
fix: make MCP get schemas validator-safe
Robin Ede
0d7a361
Use JSON-object input types for cookies and basic-auth fields in get and bulk_get so strict MCP schema validators can register tools reliably.
Normalize auth dictionaries to the tuple format expected by fetchers to preserve runtime behavior.
style: docstrings corrections for accuracy
Karim shoair
5618e2f
style: Fix all mypy errors and add type hints to untyped function bodies
Karim shoair
31c2447
**Resolved all 65 mypy errors across 14 files and added type annotations to all previously untyped function bodies. Final result: 0 errors with --check-untyped-defs enabled, all 454 tests pass.**
`scrapling/core/_types.py`
- Removed broken Self = object fallback — now requires typing_extensions for Python < 3.11
`scrapling/core/storage.py`
- Fixed str/bytes mismatch in _get_hash() — used separate _identifier_bytes variable instead of reassigning from str to bytes
`scrapling/core/custom_types.py`
- split() return type: Union[List, "TextHandlers"] → list[Any] (avoids LSP violation with parent list[str])
- format() kwargs: **kwargs: str → **kwargs: object (matches parent str.format signature)
- AttributesHandler.__init__: Added mapping: Any = None, **kwargs: Any and -> None
- json_string property: Added -> bytes return type
`scrapling/core/mixins.py`
- Changed self: "Selector" to self: Any on all mixin methods (mypy can't handle forward-reference self types on non-su…
feat(parser)!: Make all selection return selector objects by default
Karim shoair
a5f9b38
- The strings/Texthandlers are only returned by `get`/`getall`/`extract`/`extract_first`. This makes the type checking/autocompletion experience consistent.
- Removed `css_first` and `xpath_first` since it doesn't make sense to leave them now.
- Made the type hints more accurate in multiple places.
feat(spiders/fetchers): Adding proxy rotation logic and change retry logic
Karim shoair
ed96cdc
- User passes a single proxy to the browser session, and it will keep using the same context. Pass a proxy manager that automatically refreshes the IP with each proxy, and you get speed but sacrifice some stealth.
- User imports the proxy rotator class and passes proxies to it and a rotation strategy (Round robin by default). Then pass the instance to the session class, which will create a context and a tab for each proxy returned by the rotator. This way, you sacrifice speed a bit, but you get maximum stealth since each context is created with the proxy that will be used.
- Normal requests use what you pass without issues, of course.
- All errors are retried now, and proxies are rotated on retries automatically.
feat(parser): replacing `tldextract` with `tld` library
Karim shoair
e5733cb
This might break the adaptive data users have for websites BUT:
1. tld uses ~3.7x less memory during extraction operations (1.5 MB vs 5.7 MB).
2. tld uses ~56% less memory on import (5.2 MB vs 11.9 MB).
3. Zero dependencies (vs 3 for tldextract).
In return, it's 30ms slower for extracting 5000 URLs, which is negligible. Also, the type hints aren't always accurate, but it's fine; I corrected them.
build: Add w3lib to deps
Karim shoair
da0d22b
feat: Change logger to be flexible enough for the spiders classes
Karim shoair
7003b66
fix(browsers): solve a bug with setting referer on the request level
Karim shoair
c7e573a
feat(spiders): Add follow function to the response
Karim shoair
c0f5705
docs: shorten docstrings for some functions
Karim shoair
1812d2b
fix: remove forgotten unused code
Karim shoair
a7cc20d
fix: update the cookies type hint for the mcp server
Karim shoair
d391239
fix: update shell signatures for the autocompletion
Karim shoair
05b2188
Added missing fields and reordered classes as well for easier comparisons
refactor(fetchers)!: Replace Camoufox with patchright and many optimizations
Karim shoair
ee2299e
- DynamicFetcher became 20% faster
- StealthyFetcher became 99% faster
- Scrapling size decreased
- Code became ~400 lines shorter
- Most importantly, scrapling is more stable and reliable now.
- Less confusing for new users.
- More...
fix: use the correct signature for the `stealthy_fetch` shortcut
Karim shoair
916aa58
fix(shell): use correct argument name in signatures
Karim shoair
97e2d5a
fix(shell): dynamically build the signature of shortcuts after last changes
Karim shoair
0e7f15c
fix(fetcher): Remove non-keywords arguments
Karim shoair
3f07d06
refactor(browser fetchers): Make all the type hints dynamic + Faster validation
Karim shoair
c908f33
Made the code shorter by an additional ~200 lines and easier to maintain in return for making the arguments autocompletion bad for shells that don't check for dynamic type hints like IPython.
refactor(requests): Make all the type hints dynamic
Karim shoair
a68a70b
Made the code shorter by about ~500 lines and easier to maintain in return for making the arguments autocompletion bad for shells that don't check for dynamic type hints like IPython.
feat(TextHandler): Add argument to `clean` method to remove html entities
Karim shoair
f88c43a
feat(fetcher): Make `impersonate` able to randomize fingerprint
Karim shoair
d4c3d51
refactor: Making all the codebase acceptable by PyRight
Karim shoair
e5ecf76
Also fixes #97
fix: Fixes for the type checking in the main `init` file
Karim shoair
1543315
and a bit of cleaning
style: Removing dead code/docstrings
Karim shoair
f58c872
docs: Update all pages/docstrings to reflect recent changes
Karim shoair
e357f95
refactor: better implementation for the mcp mode
Karim shoair
135c17b
feat: Make mcp able to use http transport
Karim shoair
4a670f6
perf: General code restructure to not use more than needed memory
Karim shoair
21ce5a7
fix(shell): Fixing a bug with content converting
Karim shoair
b537e49
style: Removing dead code/docstrings and correcting type hints
Karim shoair
7231786
fix(shell): Fix `view` command edge case
Karim shoair
01fef5c
fix(shell): Fix parsing curl `--data-raw`
Karim shoair
8d941c2
feat(CLI): Make the session classes available in the shell
Karim shoair
13f7ca4
fix: fixing body issue with cli view command
Karim shoair
a446c01
fix: Fixes to handle the new extra deps group
Karim shoair
e3522be
fix: Fixes for multiple encoding issues (#80 & #81 )
Karim shoair
450d5ca
fix(shell): Fix a small typo
Karim shoair
9ffc9dd
refactor: Make all fetchers as an optional dependency group
Karim shoair
a85d2c8
+ Removing some dead code
style: applying the new ruff rules to all files
Karim shoair
226b463
fix: fix for shell imported by mistake issue (#76)
Karim shoair
243d94d
style: add flags for tests coverage
Karim shoair
9b40891
- Some are already tested but the coverage report can't see it.
- Some are not necessary to test or too hard to test on GitHub's CI
fix: improve error handling
Karim shoair
0907267
perf(parser): A lot of optimizations to speed things up
