Commit History
-
Update web ui
aistoye2acf98 -
Add simple web ui demo
aistoy6759e8e -
Populate SearchResponse.warnings from snippet-quality scan
aistoy +1: Claude Opus 4.7With the snippet property (previous commit) returning None for empty or whitespace-only content, we can now flag candidates that arrive without usable text. collect_snippet_warnings() returns a single summary line per response ("N/M results have no snippet") so the warnings list stays readable; per-result attribution can come later if agents ask for it. The pipeline wires this into _build_response so every SearchResponse carries the signal. Agents gate fetch decisions on snippet, so flying blind on N candidates is actionable information. Fallback to description / meta_description (§6.2 D) is deferred — SearchResult doesn't carry those fields yet, and most engines don't populate them. When we add the fields, the fallback slots into snippet property naturally. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>5fd28d9 -
Add rerank metadata fields to SearchResult
aistoy +1: Claude Opus 4.7Agents that rerank based on content need cheap URL-derived signals and provenance data on each candidate. Previously they had to compute these themselves from parsed_url, or just do without. The plan §6.2 D calls these out explicitly. Added as properties on SearchResult (no extra storage): - normalized_url, snippet, source_engines, corroboration_count, discovery_score (aliases of existing fields; future-proof names) - is_https, netloc, path_depth, is_pdf (URL-derived) Added as a real storage field: - engine_positions: dict[str, int] — populated by ResultContainer during merge so each result records which engine placed it where. The legacy `positions: list[int]` is kept for backwards compat. snippet returns None when content is empty/whitespace, so consumers can branch on 'no usable snippet' without inspecting the raw field. This is the hook the next commit (snippet validation populating SearchResponse.warnings) will use. as_dict emits both legacy and new names. Legacy fields …f944e6a -
Add top-level status enum, query echo, and warnings to SearchResponse
aistoy +1: Claude Opus 4.7Replaces the implicit 'did anything fail' signal with an explicit SearchStatus enum (SUCCESS / PARTIAL / FAILED). A boolean `partial` would have been read as 'everything is fine' on total failure, which is exactly the wrong intuition. EMPTY counts as success-equivalent under §6.2 B. `query` is now echoed on the response so consumers don't need to thread it alongside. `warnings` is added empty — it'll be populated by snippet validation (next commit) and any other soft-issue signal. compute_status() lives next to SearchResponse so unit tests can drive it directly without spinning up the pipeline; pipeline._build_response calls it from the closure. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>bd9b9e2 -
Keep EMPTY outcomes out of the failure streak
aistoy +1: Claude Opus 4.7Previously EMPTY incremented failure_count and consecutive_failures, so a string of 3+ empty responses tripped the streak cooldown. That's the wrong default: a chronically-empty engine usually has a query with no matches or a soft rate-limit, not a hard failure. Under §6.2 B, normal empty results must not cooldown the engine. EMPTY now increments its own counters (empty_count, consecutive_empties) without touching failure_count, consecutive_failures, or cooldown_until. A warn fires once when the empty streak crosses EMPTY_STREAK_WARN_THRESHOLD (=10) so parser drift / API changes still get noticed. M2 will let profiles override the threshold via max_empty_streak; for now it's a module-level default. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>74031c1 -
Rename scoutlet/search.py to pipeline.py to avoid name shadowing
aistoy +1: Claude Opus 4.7scoutlet/__init__.py re-exports `search` (the async function), which made `scoutlet.search` resolve to the function instead of the module. This bit test code that tried to monkeypatch module attributes — the workaround was `sys.modules["scoutlet.search"]`. Renaming the module to `pipeline.py` removes the collision cleanly: - Public API unchanged (`from scoutlet import search, search_sync`) - `scoutlet.pipeline` is unambiguously the module - `pipeline` better describes the file's role (orchestration), matching the existing docstring and CLAUDE.md terminology Touched: __init__.py, cli.py, webui.py, all test imports, logger name inside the module, the CLAUDE.md module list, and the test_block_classifier_failure_does_not_break_search test (drops the sys.modules workaround). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>24cca52 -
Classify block pages as ANTI_BOT before engine parser runs
aistoy +1: Claude Opus 4.7response_classifier.detect_block_page existed but was never wired into the pipeline, so CAPTCHA'd responses fell through to engine.response() and came back as EMPTY — which under §6.2 B must not trigger cooldown. The net effect was that blocked engines kept getting redispatched. Integration runs only on 2xx with non-empty body, so: - HTTP error status (4xx/5xx) still uses raise_for_httperror path - Empty body still falls through to parser (preserves EMPTY semantics) - Classifier exceptions are swallowed to keep search resilient Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>4de0823
-
Return SearchResponse instead of bare list from search()
aistoyscoutlet's primary audience is agents, and an agent calling search() has been flying blind: it sees the result list but not what happened underneath. Which engines got CAPTCHA'd? Which were skipped because they're in cooldown? Did wave two fire? Without that signal the agent can't make informed retries — it just sees "0 results" and guesses. Introduce SearchResponse as the new return type of search() and search_sync(), carrying the minimal set an agent needs to adapt: - results: list[SearchResult] (the old return value, unchanged shape) - engines: list[EngineRunInfo] (per-engine outcome: name / status / elapsed_ms / error) - skipped: list[SkippedEngine] (name + reason, e.g. "cooldown" / "not_loaded") Deliberately omitted from this first cut: wave breakdown, coverage scores, total elapsed_ms, query echo. Each of those is easy to add once a concrete agent use case demands it; adding them now is YAGNI and bloats…64f21ed -
Dispatch engines in two waves with coverage check
aistoyscoutlet inherited SearXNG's "fire every selected engine at once" model, which is the wrong default for an embeddable library: general-category engines overlap heavily (google/bing/baidu/sogou/duckduckgo mostly return the same top URLs), so running all of them burns latency for marginal recall. Vertical engines are the opposite — each has unique coverage, skipping any loses real results. Add src/scoutlet/routing.py with plan_waves() + coverage_satisfied(): - General engines: rank by priority (weight × success_rate × 1/latency), put top GENERAL_FIRST_WAVE (4) in wave one, next GENERAL_SECOND_WAVE (3) in wave two. - Vertical engines: all in wave one, no cap. - Explicit engines=[...] from the caller: wave planning bypassed entirely. User said "run these", we run them. - Cooldown engines were already filtered upstream by health.is_available. Time budget is split 60/40 between waves so worst-case total latency stays close to the caller's `timeout`. When wave two doesn…5f3cef9 -
Layer diversity bonus and health penalty on top of base score
aistoySearXNG's base formula weight × Σ(1/position) already rewards multi- engine corroboration, and that's the right default. But two patterns it doesn't handle well: 1. An engine returning a unique URL that no other engine saw gets base score, while a corroborated URL runs away with the top of the ranking. Repetitive corroborated results dominate even when the unique result is more relevant. 2. A flaky engine (sub-30% success rate, or currently in cooldown) that did return results this round shouldn't be trusted as much as a healthy engine returning the same kind of result. Apply two multiplicative adjustments in ResultContainer.close(), after the SearXNG base score is computed: - UNIQUE_BONUS (1.2x): applied to results with exactly one contributing engine. Counteracts the corroboration bias. - HEALTH_PENALTY (0.5x): applied to single-engine results whose sole contributor is currently "bad" (in cooldown or success_rate < 0.3 with >= 5 samples…71ad770 -
Track per-engine health with in-process registry
aistoyEngines fail for different reasons — CAPTCHA, rate-limit, parser break, missing config — and the response to each is different. A CAPTCHA'd engine shouldn't be retried for several minutes; a parser-broken one can be retried immediately. Without tracking, scoutlet re-dispatches to known-dead engines on every search. Add src/scoutlet/health.py with EngineHealthRegistry: - per-engine counters (success / failure / empty / anti_bot / rate_limit / timeout) - latency EMA (alpha=0.3) so routing can prefer fast engines - cooldown_until driven by status: ANTI_BOT and RATE_LIMIT push it out 300s, three consecutive failures push it 600s; SUCCESS clears it immediately. cooldown_until is a max(), so a short cooldown never shrinks an existing long one - is_bad: True when in cooldown OR success_rate < 0.3 with >= 5 samples (gates PR-C's score penalty) Registry is thread-safe (Lock around mutations) because engines run via asyncio.to_thread. State is process-wide and intentionally not pers…2a235d3 -
Return structured EngineOutcome from _run_engine
aistoy_run_engine previously swallowed all exceptions and returned [], which discarded the failure signal downstream consumers need: the health registry wants to count CAPTCHA vs parser-broken separately, two-wave routing wants to skip dead engines, auto-heal wants to know which engines have structural failures. Flatten all that into "no results" loses the information that would drive each of those decisions. Add src/scoutlet/outcome.py with: - FailureKind enum: SUCCESS / EMPTY / ANTI_BOT / RATE_LIMIT / HTTP_ERROR / PARSER_ERROR / TIMEOUT / CONFIG_ERROR - EngineOutcome dataclass: engine / status / elapsed_ms / results / error, plus an .ok convenience property - classify_failure(exc, phase): maps exceptions to FailureKind. Order matters — Captcha is checked before TooManyRequests before generic AccessDenied so each lands in the right bucket. httpx.TimeoutException splits out from httpx.HTTPError so timeouts are distinguishable from connect errors. Phase ("request" / "fetch" / "r…a229070 -
Extract HTTP response classifier into its own module
aistoydetect_block_page and BlockDetectionResult previously lived at the top of browser.py, which made them unreachable without buying into the whole CDP stack — pychrome import, ManagedBrowser, BrowserRunner. After the CDP strip-out in the previous commit, the classifier is the only piece worth keeping, and it has legitimate non-browser uses: health checks, auto-heal, and future routing all want to tell "engine got CAPTCHA'd" apart from "engine parser broke" without launching Chrome. Move just that code into src/scoutlet/response_classifier.py: - BlockDetectionResult, detect_block_page unchanged - _ENGINE_BLOCK_PATTERNS (google/bing/ddg) unchanged - _GENERIC_BLOCK_KEYWORDS (Cloudflare/Akamai/PerimeterX/etc.) unchanged - _BLOCK_PAGE_SIZE_THRESHOLD = 2048 unchanged The module imports only `re` — no httpx, no lxml, no pychrome. The 11 test cases carry over verbatim from the deleted test_browser.py.d40c9c9 -
Remove CDP browser fallback from core search path
aistoybe541e9
-
Update search result page with engine filter
aistoy7aa4450
-
Stop engines from emitting suggestion-only result dicts
aistoy +1: Claude Opus 4.7yahoo and google_videos had a SearXNG-derived loop that appended {'suggestion': <text>} dicts alongside real result dicts. SearXNG's container has a separate suggestions channel; scoutlet doesn't, so the dicts went through SearchResult normalization with no matching field, became ghost results with empty url/title/content but position-based scores, and polluted the CLI output as: 29. [yahoo] score: 0.12 Remove the dead suggestion loop from both engines and drop the now- unused suggestion_xpath import from google_videos. Add a regression test using a live Yahoo HTML fixture that asserts every returned dict has both url and title. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>ddf0062 -
Switch no-args default to a language-aware engine set
aistoy +1: Claude Opus 4.7The previous default (all 12 general-category engines) produced a wall of CAPTCHA warnings on every plain search — google/ddg/qwant/baidu/sogou/ quark all fail out of the box without --fallback-to-browser. For a human running the CLI or WebUI without args, that noise is the wrong trade. Replace the catch-all general-category fallback with a curated set: BASE (always): google, bing, mwmbl, marginalia ZH_EXTRA: baidu, sogou, brave NONZH_EXTRA: brave, yahoo, duckduckgo Chinese-query detection uses Han-character presence filtered by kana/ hangul absence, so Japanese/Korean fall through to the international set. Explicit -e or -c still overrides the default; only the both-None path changes. Agents are expected to pass engines explicitly — this default targets human users running CLI/WebUI. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>f481492 -
Align emojipedia and wikipedia categories with SearXNG
aistoy +1: Claude Opus 4.7SearXNG declares these as categories=[], which its loader demotes to the 'other' bucket — engines that don't show up in any primary tab and must be explicitly invoked. scoutlet previously hardcoded them as ["general"] so they'd surface in default searches, but that hid their nature: both are single-site queries (emoji dictionary, Wikipedia API), not whole-web search engines. Set categories=[] to match SearXNG. Default searches now skip them; invoke explicitly with -e emojipedia or -e wikipedia. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>c89ed27 -
Expose adapter-backend flag through the CLI
aistoy +1: Claude Opus 4.7The PrimpAdapter (TLS fingerprint impersonation via primp) was fully implemented and per-engine http_client was already wired through _run_engine, but no CLI surface exposed it. Add --adapter-backend {httpx,default,fingerprint} and thread search_adapter_backend through search()/search_sync(). Per-engine http_client still overrides the global flag. Required for engines like pixabay that now sit behind Cloudflare and need browser-grade TLS to reach their JSON bootstrap endpoint. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>76956dc -
Raise SearchEngineAccessDenied on pixabay Cloudflare blocks
aistoy +1: Claude Opus 4.7Pixabay's bootstrap JSON endpoint now sits behind Cloudflare. The engine called resp.json() unconditionally, so the 403 HTML challenge produced a cryptic "Expecting value: line 1 column 1" error. Set raise_for_httperror on the request so the orchestrator raises AccessDenied before parsing, matching the pattern used by github_code, loc, dailymotion, and qwant. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>3c1b7f1 -
Fix Enter key not submitting search in the web UI
aistoy +1: Claude Opus 4.7The search form contains two text inputs (the query box and the engine filter `.ms-search` inside the dropdown panel). Per the HTML spec's implicit-submission rule, Enter only fires submit when a form has at most one text-like input — with two, the keypress was silently dropped. Add an explicit keydown handler that routes Enter to form.requestSubmit(), with an IME guard so CJK composition still uses Enter to confirm candidates. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>bb5fbb4 -
Default search to the general category when engines/categories unset
aistoy +1: Claude Opus 4.7Loading every engine by default pulls in ones that need out-of-band config (spotify, youtube_api, tubearchivist, MRS) or only make sense for specific query types (images, videos, music), producing a wall of setup warnings and parser noise for a plain query. Defaulting to the "general" category matches the CLI's documented behavior and keeps the first-run experience clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>252fcd0 -
Expose CDP browser fallback via CLI and harden managed browser state
aistoy +1: Claude Opus 4.7The CDP fallback path existed in browser.py but was unreachable from the CLI, and several cached-state bugs meant that even when wired in, headful mode silently reused a headless Chrome, endpoint changes were ignored, and BrowserRunner.close() would close every tab in the user's real Chrome session. 1. cli.py adds --fallback-to-browser, --auto-launch-browser, --cdp-endpoint, and --headful; these thread through search() and search_sync() as search_* overrides that OR-merge with per-engine config. --auto-launch-browser implies --fallback-to-browser. 2. browser.py scope-fixes: - ManagedBrowser user_data_dir is now port-scoped (scoutlet-chrome-profile-{port}) so launching on a second port doesn't collide with a profile locked by the first. - resolve_cdp_endpoint_for_launch() probes /json/version; when the user asked for headful but the endpoint is already occupied by HeadlessChrome, it picks a free port instead of silently reusing headless. - ge…f5578c3 -
Cascade category selection into the engine multiselect
aistoy +1: Claude Opus 4.7Exposes engines_by_category from /api/engines (still peek-only — no setup() calls). When one or more categories are checked, the engine list narrows to their union; unchecking all restores the full list. Previously-checked engines that fall outside the visible set are pruned. Backend search semantics are unchanged (engines still takes precedence over categories when both are supplied). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>eb46596 -
Add local web UI launched via scoutlet --ui
aistoy +1: Claude Opus 4.7Runs on Python's built-in http.server (no extra deps) and auto-opens the default browser. Single page with home + results views; Category and Engine filters are multi-select, Language and Time Range stay single. The /api/engines endpoint peeks categories without triggering setup(), so UI startup is silent. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>6622a40
-
Reduce search-time noise: filter by category before setup, drop tracebacks
aistoy +1: Claude Opus 4.7Three coupled fixes for the noisy/traceback-laden UX when running a category search (e.g., `scoutlet -c news "SpaceX IPO"`): 1. duckduckgo_extra now has a setup() that rejects the empty default ddg_category, instead of letting request() KeyError later. Was the source of the only remaining traceback. 2. search() peeks each engine's declared categories via a lightweight module import (no setup()), filters to the requested categories, then loads only matches — so unrelated engines (spotify, youtube_api, …) no longer fire their setup() errors during a category search. 3. _run_engine() now logs "Engine 'X' failed: <reason>" instead of dumping the full traceback via exc_info=True. Loader also stops re-announcing setup() failures (the engine's own logger already did). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>c6c201c -
Replace engine-load tracebacks with friendly error messages
aistoy +1: Claude Opus 4.7load_engine() now records a short reason per failure into _failed_engines and emits a single log.error() line, instead of log.exception() which dumped the full traceback. The CLI's --by-category output gains a "failed to load" section listing engines that could not be set up (typically missing API keys or base_url). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>adaf9b2 -
Fill empty engine categories and add --by-category flag
aistoy +1: Claude Opus 4.7Six engines (baidu, brave, duckduckgo_extra, emojipedia, quark, qwant) had categories=[], which excluded them from all category buckets. The new --list-engines --by-category flag groups engines by their declared categories; the default flat list is unchanged for script compatibility. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>2507a77 -
Add frinkiac and emojipedia engines
aistoy +1: Claude Opus 4.7Both verified live-healthy before migration: - frinkiac: 36 Simpsons screenshot results for 'heart' (922ms), JSON API - emojipedia: 20 emoji reference results for 'heart' (627ms), HTML xpath Engine notes: - frinkiac hits /api/search?q=... and builds caption URLs from the Episode/Timestamp fields in each record. - emojipedia keeps an empty categories list (matches SearXNG upstream); the engine is registered globally and must be invoked by name via engines=['emojipedia'], it does not appear in category-based searches. Engine count: 106 -> 108. Tests: 463 -> 473. README/README_cn synced. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>0108e1f -
Add 53 SearXNG engines: science, images, videos, music, movies/social/news
aistoy +1: Claude Opus 4.7Migrates A-tier engines from SearXNG (~/Src/searXNG) into scoutlet, bringing the built-in engine count from 44 to 106. Each engine is adapted to scoutlet's import scheme and tested against a saved fixture. Engines added (53): - science (9): arxiv, pypi, crossref, openalex, semantic_scholar, pubmed, pdbe, astrophysics_data_system (api_key), scanr_structures - images (17): artic, artstation, deviantart, findthatmeme, flickr (api_key), flickr_noapi, ipernity, loc, openclipart, openverse, pexels, pinterest, pixabay, pixiv, public_domain_image_archive, sogou_images, www1x - videos (18): 360search_videos, acfun, bitchute, ccc_media, dailymotion, digbt, ina, iqiyi, mediathekviewweb, niconico, odysee, peertube, rumble, sepiasearch, sogou_videos, tubearchivist (base_url+token), youtube_api (api_key), youtube_noapi - music (5): mixcloud, radio_browser, soundcloud, spotify (credentials), yandex_music - movies/social/news (13): imdb, moviepilot, rottentomatoes, senscritique, 9gag,…8f54a13 -
Initial commit: scoutlet lightweight search aggregation library
aistoy500336a