Spaces:
Running on Zero
Running on Zero
Upload 4 files
Browse files
README.md
CHANGED
|
@@ -1,14 +1,94 @@
|
|
| 1 |
---
|
| 2 |
title: Google Search Scraper
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
|
|
|
| 6 |
sdk: gradio
|
| 7 |
-
sdk_version: 6.
|
| 8 |
-
python_version: '3.12'
|
| 9 |
app_file: app.py
|
| 10 |
pinned: false
|
| 11 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 12 |
---
|
| 13 |
|
| 14 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
title: Google Search Scraper
|
| 3 |
+
short_description: Any Google results page as JSON, plus a rank checker
|
| 4 |
+
emoji: 🔎
|
| 5 |
+
colorFrom: blue
|
| 6 |
+
colorTo: green
|
| 7 |
sdk: gradio
|
| 8 |
+
sdk_version: 6.0.2
|
|
|
|
| 9 |
app_file: app.py
|
| 10 |
pinned: false
|
| 11 |
+
tags:
|
| 12 |
+
- google
|
| 13 |
+
- google-search
|
| 14 |
+
- serp
|
| 15 |
+
- serp-api
|
| 16 |
+
- google-scraper
|
| 17 |
+
- rank-tracker
|
| 18 |
+
- seo
|
| 19 |
+
- ai-overview
|
| 20 |
+
- people-also-ask
|
| 21 |
+
- web-scraping
|
| 22 |
+
- scraper
|
| 23 |
+
- data-extraction
|
| 24 |
+
- smolagents
|
| 25 |
+
- tool
|
| 26 |
+
- scrape.do
|
| 27 |
---
|
| 28 |
|
| 29 |
+
# Google Search Scraper — Powered by Scrape.do
|
| 30 |
+
|
| 31 |
+
Run a Google search from any country and get the whole results page back as
|
| 32 |
+
structured data: organic rankings, ads, People Also Ask, related searches, the
|
| 33 |
+
AI Overview, knowledge panel, local pack, top stories and videos.
|
| 34 |
+
|
| 35 |
+
## 🔑 Get your free API token
|
| 36 |
+
|
| 37 |
+
👉 **[Sign up free on Scrape.do](https://dashboard.scrape.do/sign-up?utm_source=huggingface&utm_medium=google-search-scraper)** — 1,000 free credits per month, no credit card required. One results page costs 10 credits, so that's about 100 searches.
|
| 38 |
+
|
| 39 |
+
## Two tabs
|
| 40 |
+
|
| 41 |
+
| Tab | Input | Output |
|
| 42 |
+
| --- | --- | --- |
|
| 43 |
+
| **Search results** | Any query | A live preview of the results page, a table of organic results, CSV download and the raw JSON |
|
| 44 |
+
| **Rank checker** | Keyword + your domain | Your position, the page it's on, and every site ranking above you — highlighted on the results page |
|
| 45 |
+
|
| 46 |
+
Both tabs share the same settings: 14 countries (each with its own Google domain
|
| 47 |
+
and language), desktop or mobile results, 1–5 pages deep, a time filter, and
|
| 48 |
+
optional city-level location.
|
| 49 |
+
|
| 50 |
+
## Why an API instead of parsing Google's HTML
|
| 51 |
+
|
| 52 |
+
Google renames the code behind its results page without warning, so a scraper
|
| 53 |
+
built on page selectors breaks silently. This Space uses the
|
| 54 |
+
[Scrape.do Google Search API](https://scrape.do/documentation/google-scraper-api/search/),
|
| 55 |
+
which returns the same field names whatever Google changes, and handles
|
| 56 |
+
proxies and CAPTCHAs.
|
| 57 |
+
|
| 58 |
+
Two details are handled for you:
|
| 59 |
+
|
| 60 |
+
- **Multi-page searches stay consistent.** Every page request carries a start
|
| 61 |
+
offset from the first page on, so all pages come from the same browsing
|
| 62 |
+
session instead of reshuffling or repeating results.
|
| 63 |
+
- **Late AI Overviews are collected.** When Google is still writing its AI
|
| 64 |
+
Overview as the page returns, the tool fetches the finished version with one
|
| 65 |
+
follow-up call (5 credits), immediately — the key for it expires in 90 seconds.
|
| 66 |
+
|
| 67 |
+
## Use as a smolagents Tool
|
| 68 |
+
|
| 69 |
+
```python
|
| 70 |
+
from smolagents import CodeAgent, InferenceClientModel
|
| 71 |
+
from tool import ScrapeDoGoogleSearchTool
|
| 72 |
+
|
| 73 |
+
agent = CodeAgent(
|
| 74 |
+
tools=[ScrapeDoGoogleSearchTool(api_token="YOUR_SCRAPEDO_TOKEN")],
|
| 75 |
+
model=InferenceClientModel(),
|
| 76 |
+
)
|
| 77 |
+
agent.run("Who ranks in the top 5 for 'best web scraping api' in Germany, and do any run ads?")
|
| 78 |
+
```
|
| 79 |
+
|
| 80 |
+
You can also set `SCRAPE_DO_API_TOKEN` in your environment and instantiate
|
| 81 |
+
`ScrapeDoGoogleSearchTool()` with no arguments.
|
| 82 |
+
|
| 83 |
+
## Inputs
|
| 84 |
+
|
| 85 |
+
- `query` *(required)* — The search, as you'd type it into Google.
|
| 86 |
+
- `country` *(optional)* — Two-letter code (`us`, `gb`, `de`, `tr`…). Sets ranking country, language and Google domain. Defaults to `us`.
|
| 87 |
+
- `pages` *(optional)* — 1–5 result pages. 10 credits each. Defaults to `1`.
|
| 88 |
+
- `device` *(optional)* — `desktop` or `mobile`.
|
| 89 |
+
- `time_period` *(optional)* — `last_hour`, `last_day`, `last_week`, `last_month`, `last_year`.
|
| 90 |
+
- `location` *(optional)* — City in Google's format, e.g. `Austin,Texas,United States`.
|
| 91 |
+
|
| 92 |
+
---
|
| 93 |
+
|
| 94 |
+
Learn more at [scrape.do](https://scrape.do/) · [Documentation](https://scrape.do/documentation/) · [Google search scraping guide](https://scrape.do/blog/scraping-google-search-results/)
|
app.py
ADDED
|
@@ -0,0 +1,559 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from __future__ import annotations
|
| 2 |
+
|
| 3 |
+
import csv
|
| 4 |
+
import html
|
| 5 |
+
import tempfile
|
| 6 |
+
from typing import Any, Dict, List, Optional, Tuple
|
| 7 |
+
from urllib.parse import urlparse
|
| 8 |
+
|
| 9 |
+
import gradio as gr
|
| 10 |
+
|
| 11 |
+
from tool import COUNTRIES, SCRAPE_DO_SIGNUP_URL, TIME_PERIODS, ScrapeDoGoogleSearchTool
|
| 12 |
+
|
| 13 |
+
# ZeroGPU hardware refuses to keep a Space alive unless it finds at least one
|
| 14 |
+
# @spaces.GPU function at startup. This app is pure HTTP and never touches a GPU,
|
| 15 |
+
# so the probe below exists only to satisfy that check. It is never called.
|
| 16 |
+
try:
|
| 17 |
+
import spaces
|
| 18 |
+
|
| 19 |
+
@spaces.GPU(duration=1)
|
| 20 |
+
def _zerogpu_probe() -> str:
|
| 21 |
+
return "ok"
|
| 22 |
+
|
| 23 |
+
except ImportError:
|
| 24 |
+
pass
|
| 25 |
+
|
| 26 |
+
|
| 27 |
+
COUNTRY_CHOICES = [(label, gl) for label, gl, _, _ in COUNTRIES]
|
| 28 |
+
TIME_CHOICES = [("Any time", "")] + [
|
| 29 |
+
(value.replace("_", " ").capitalize(), value) for value in TIME_PERIODS
|
| 30 |
+
]
|
| 31 |
+
|
| 32 |
+
NO_TOKEN_MD = (
|
| 33 |
+
"### Paste your Scrape.do API token above\n\n"
|
| 34 |
+
f"No token yet? [Create a free account]({SCRAPE_DO_SIGNUP_URL}) — "
|
| 35 |
+
"1,000 free credits a month, no credit card. One search costs 10 credits."
|
| 36 |
+
)
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
# --------------------------------------------------------------------- helpers
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
def _esc(value: Any) -> str:
|
| 43 |
+
return html.escape(str(value or ""), quote=True)
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
def _safe_link(url: Any) -> str:
|
| 47 |
+
url = str(url or "")
|
| 48 |
+
return url if url.startswith(("http://", "https://")) else "#"
|
| 49 |
+
|
| 50 |
+
|
| 51 |
+
def _domain(url: Any) -> str:
|
| 52 |
+
try:
|
| 53 |
+
host = urlparse(str(url or "")).netloc.lower()
|
| 54 |
+
except ValueError:
|
| 55 |
+
return ""
|
| 56 |
+
return host[4:] if host.startswith("www.") else host
|
| 57 |
+
|
| 58 |
+
|
| 59 |
+
def _tool_or_none(token: str) -> Optional[ScrapeDoGoogleSearchTool]:
|
| 60 |
+
if not token or not token.strip():
|
| 61 |
+
return None
|
| 62 |
+
return ScrapeDoGoogleSearchTool(api_token=token.strip())
|
| 63 |
+
|
| 64 |
+
|
| 65 |
+
def _write_csv(rows: List[Dict[str, Any]], prefix: str) -> Optional[str]:
|
| 66 |
+
if not rows:
|
| 67 |
+
return None
|
| 68 |
+
fieldnames: List[str] = []
|
| 69 |
+
for row in rows:
|
| 70 |
+
for key in row:
|
| 71 |
+
if key not in fieldnames:
|
| 72 |
+
fieldnames.append(key)
|
| 73 |
+
handle = tempfile.NamedTemporaryFile(
|
| 74 |
+
mode="w", suffix=".csv", prefix=f"{prefix}-", delete=False,
|
| 75 |
+
newline="", encoding="utf-8-sig",
|
| 76 |
+
)
|
| 77 |
+
writer = csv.DictWriter(handle, fieldnames=fieldnames)
|
| 78 |
+
writer.writeheader()
|
| 79 |
+
for row in rows:
|
| 80 |
+
writer.writerow({key: row.get(key, "") for key in fieldnames})
|
| 81 |
+
handle.close()
|
| 82 |
+
return handle.name
|
| 83 |
+
|
| 84 |
+
|
| 85 |
+
def _error_md(data: Dict[str, Any]) -> str:
|
| 86 |
+
out = f"**That search didn't go through.**\n\n{data.get('errorMessage') or 'Unknown error'}"
|
| 87 |
+
if data.get("hint"):
|
| 88 |
+
out += f"\n\n{data['hint']}"
|
| 89 |
+
return out
|
| 90 |
+
|
| 91 |
+
|
| 92 |
+
# ------------------------------------------------------------ SERP rendering
|
| 93 |
+
|
| 94 |
+
|
| 95 |
+
def _render_ai_blocks(blocks: List[Dict[str, Any]]) -> str:
|
| 96 |
+
parts: List[str] = []
|
| 97 |
+
|
| 98 |
+
def render_list(items: List[Dict[str, Any]]) -> str:
|
| 99 |
+
inner = ""
|
| 100 |
+
for item in items or []:
|
| 101 |
+
inner += f"<li>{_esc(item.get('snippet'))}"
|
| 102 |
+
if item.get("list"):
|
| 103 |
+
inner += render_list(item["list"])
|
| 104 |
+
inner += "</li>"
|
| 105 |
+
return f"<ul>{inner}</ul>"
|
| 106 |
+
|
| 107 |
+
for block in blocks or []:
|
| 108 |
+
kind = block.get("type")
|
| 109 |
+
if kind == "heading":
|
| 110 |
+
parts.append(f"<h4>{_esc(block.get('snippet'))}</h4>")
|
| 111 |
+
elif kind == "list":
|
| 112 |
+
if block.get("snippet"):
|
| 113 |
+
parts.append(f"<p>{_esc(block['snippet'])}</p>")
|
| 114 |
+
parts.append(render_list(block.get("list") or []))
|
| 115 |
+
elif kind == "table":
|
| 116 |
+
head = "".join(f"<th>{_esc(h)}</th>" for h in block.get("table_headers") or [])
|
| 117 |
+
rows = "".join(
|
| 118 |
+
"<tr>" + "".join(f"<td>{_esc(c)}</td>" for c in row) + "</tr>"
|
| 119 |
+
for row in block.get("table_rows") or []
|
| 120 |
+
)
|
| 121 |
+
parts.append(f"<table><thead><tr>{head}</tr></thead><tbody>{rows}</tbody></table>")
|
| 122 |
+
else:
|
| 123 |
+
parts.append(f"<p>{_esc(block.get('snippet'))}</p>")
|
| 124 |
+
return "".join(parts)
|
| 125 |
+
|
| 126 |
+
|
| 127 |
+
def _render_ad(ad: Dict[str, Any]) -> str:
|
| 128 |
+
link = _safe_link(ad.get("url") or ad.get("link"))
|
| 129 |
+
return (
|
| 130 |
+
'<div class="gs-result">'
|
| 131 |
+
f'<div class="gs-cite"><span class="gs-sponsored">Sponsored</span> {_esc(ad.get("displayed_url") or _domain(link))}</div>'
|
| 132 |
+
f'<a class="gs-title" href="{_esc(link)}" target="_blank" rel="noopener noreferrer">{_esc(ad.get("title"))}</a>'
|
| 133 |
+
f'<div class="gs-snippet">{_esc(ad.get("description"))}</div>'
|
| 134 |
+
"</div>"
|
| 135 |
+
)
|
| 136 |
+
|
| 137 |
+
|
| 138 |
+
def render_serp(data: Dict[str, Any], highlight_domain: str = "") -> str:
|
| 139 |
+
info = data.get("search_information") or {}
|
| 140 |
+
out: List[str] = ['<div class="gs-serp">']
|
| 141 |
+
|
| 142 |
+
stats = []
|
| 143 |
+
if info.get("total_results"):
|
| 144 |
+
stats.append(f"About {int(info['total_results']):,} results")
|
| 145 |
+
if info.get("time_taken_displayed"):
|
| 146 |
+
stats.append(f"({info['time_taken_displayed']} seconds)")
|
| 147 |
+
if stats:
|
| 148 |
+
out.append(f'<div class="gs-stats">{_esc(" ".join(stats))}</div>')
|
| 149 |
+
if info.get("results_for"):
|
| 150 |
+
out.append(f'<div class="gs-stats">Showing results for <b>{_esc(info["results_for"])}</b></div>')
|
| 151 |
+
|
| 152 |
+
ai = data.get("ai_overview")
|
| 153 |
+
if isinstance(ai, dict) and ai.get("state") == "complete" and ai.get("text_blocks"):
|
| 154 |
+
refs = "".join(
|
| 155 |
+
f'<li><a href="{_esc(_safe_link(ref.get("link")))}" target="_blank" rel="noopener noreferrer">'
|
| 156 |
+
f'{_esc(ref.get("title") or ref.get("source"))}</a> <span>{_esc(ref.get("source"))}</span></li>'
|
| 157 |
+
for ref in (ai.get("references") or [])[:8]
|
| 158 |
+
)
|
| 159 |
+
out.append(
|
| 160 |
+
'<div class="gs-ai"><div class="gs-ai-label">✦ AI Overview</div>'
|
| 161 |
+
f"{_render_ai_blocks(ai['text_blocks'])}"
|
| 162 |
+
+ (f'<div class="gs-ai-refs"><b>Sources</b><ol>{refs}</ol></div>' if refs else "")
|
| 163 |
+
+ "</div>"
|
| 164 |
+
)
|
| 165 |
+
|
| 166 |
+
for ad in data.get("top_ads") or []:
|
| 167 |
+
out.append(_render_ad(ad))
|
| 168 |
+
|
| 169 |
+
kg = data.get("knowledge_graph")
|
| 170 |
+
if isinstance(kg, dict) and kg.get("title"):
|
| 171 |
+
facts = "".join(
|
| 172 |
+
f"<div><b>{_esc(f.get('label'))}:</b> {_esc(f.get('value'))}</div>"
|
| 173 |
+
for f in (kg.get("facts") or [])[:6]
|
| 174 |
+
)
|
| 175 |
+
out.append(
|
| 176 |
+
'<div class="gs-kg">'
|
| 177 |
+
f"<div class=\"gs-kg-title\">{_esc(kg.get('title'))}</div>"
|
| 178 |
+
f"<div class=\"gs-kg-type\">{_esc(kg.get('type'))}</div>"
|
| 179 |
+
f"<p>{_esc(kg.get('description'))}</p>{facts}</div>"
|
| 180 |
+
)
|
| 181 |
+
|
| 182 |
+
local = data.get("local_results")
|
| 183 |
+
if isinstance(local, dict) and local.get("places"):
|
| 184 |
+
places = "".join(
|
| 185 |
+
f"<li><b>{_esc(p.get('title'))}</b> · {_esc(p.get('rating') or '—')}★ {_esc(p.get('price'))}</li>"
|
| 186 |
+
for p in local["places"][:5]
|
| 187 |
+
)
|
| 188 |
+
out.append(f'<div class="gs-box"><div class="gs-box-title">Places</div><ul>{places}</ul></div>')
|
| 189 |
+
|
| 190 |
+
paa = data.get("related_questions") or []
|
| 191 |
+
organic = data.get("organic_results") or []
|
| 192 |
+
highlight = highlight_domain.lower().removeprefix("www.")
|
| 193 |
+
|
| 194 |
+
for index, result in enumerate(organic):
|
| 195 |
+
link = _safe_link(result.get("link"))
|
| 196 |
+
is_hit = bool(highlight) and (
|
| 197 |
+
_domain(link) == highlight or _domain(link).endswith("." + highlight)
|
| 198 |
+
)
|
| 199 |
+
cls = "gs-result gs-hit" if is_hit else "gs-result"
|
| 200 |
+
page_badge = (
|
| 201 |
+
f'<span class="gs-pos">#{result.get("overall_position", index + 1)}</span>'
|
| 202 |
+
)
|
| 203 |
+
sitelinks = ""
|
| 204 |
+
sl = result.get("sitelinks") or {}
|
| 205 |
+
sl_items = (sl.get("inline") or []) + (sl.get("expanded") or [])
|
| 206 |
+
if sl_items:
|
| 207 |
+
sitelinks = '<div class="gs-sitelinks">' + " · ".join(
|
| 208 |
+
f'<a href="{_esc(_safe_link(s.get("link")))}" target="_blank" rel="noopener noreferrer">{_esc(s.get("title"))}</a>'
|
| 209 |
+
for s in sl_items[:4]
|
| 210 |
+
) + "</div>"
|
| 211 |
+
date = f'<span class="gs-date">{_esc(result.get("date"))} — </span>' if result.get("date") else ""
|
| 212 |
+
out.append(
|
| 213 |
+
f'<div class="{cls}">'
|
| 214 |
+
f'<div class="gs-cite">{page_badge}<span class="gs-source">{_esc(result.get("source") or _domain(link))}</span>'
|
| 215 |
+
f'<span class="gs-url">{_esc(result.get("displayed_link") or link)}</span></div>'
|
| 216 |
+
f'<a class="gs-title" href="{_esc(link)}" target="_blank" rel="noopener noreferrer">{_esc(result.get("title"))}</a>'
|
| 217 |
+
f'<div class="gs-snippet">{date}{_esc(result.get("snippet"))}</div>{sitelinks}</div>'
|
| 218 |
+
)
|
| 219 |
+
# Google puts People Also Ask after the first few results.
|
| 220 |
+
if index == 2 and paa:
|
| 221 |
+
qs = "".join(f"<li>{_esc(q.get('question'))}</li>" for q in paa[:6])
|
| 222 |
+
out.append(f'<div class="gs-box"><div class="gs-box-title">People also ask</div><ul class="gs-paa">{qs}</ul></div>')
|
| 223 |
+
paa = []
|
| 224 |
+
|
| 225 |
+
if paa:
|
| 226 |
+
qs = "".join(f"<li>{_esc(q.get('question'))}</li>" for q in paa[:6])
|
| 227 |
+
out.append(f'<div class="gs-box"><div class="gs-box-title">People also ask</div><ul class="gs-paa">{qs}</ul></div>')
|
| 228 |
+
|
| 229 |
+
for ad in data.get("bottom_ads") or []:
|
| 230 |
+
out.append(_render_ad(ad))
|
| 231 |
+
|
| 232 |
+
related = data.get("related_searches") or []
|
| 233 |
+
if related:
|
| 234 |
+
chips = "".join(f'<span class="gs-chip">🔍 {_esc(r.get("query"))}</span>' for r in related[:10])
|
| 235 |
+
out.append(f'<div class="gs-box"><div class="gs-box-title">People also search for</div><div class="gs-chips">{chips}</div></div>')
|
| 236 |
+
|
| 237 |
+
if not organic:
|
| 238 |
+
out.append('<div class="gs-stats">No organic results came back for this query.</div>')
|
| 239 |
+
|
| 240 |
+
out.append("</div>")
|
| 241 |
+
return "".join(out)
|
| 242 |
+
|
| 243 |
+
|
| 244 |
+
def _organic_rows(data: Dict[str, Any]) -> List[Dict[str, Any]]:
|
| 245 |
+
return [
|
| 246 |
+
{
|
| 247 |
+
"position": r.get("overall_position"),
|
| 248 |
+
"page": r.get("page"),
|
| 249 |
+
"title": r.get("title"),
|
| 250 |
+
"link": r.get("link"),
|
| 251 |
+
"domain": _domain(r.get("link")),
|
| 252 |
+
"snippet": r.get("snippet"),
|
| 253 |
+
"date": r.get("date"),
|
| 254 |
+
"source": r.get("source"),
|
| 255 |
+
}
|
| 256 |
+
for r in data.get("organic_results") or []
|
| 257 |
+
]
|
| 258 |
+
|
| 259 |
+
|
| 260 |
+
# ---------------------------------------------------------------- handlers
|
| 261 |
+
|
| 262 |
+
|
| 263 |
+
TABLE_HEADERS = ["#", "Domain", "Title", "Link"]
|
| 264 |
+
|
| 265 |
+
|
| 266 |
+
def run_search(
|
| 267 |
+
token: str, query: str, country: str, pages: int, device: str, time_period: str, location: str
|
| 268 |
+
) -> Tuple[str, str, List[List[Any]], Dict[str, Any], Optional[str]]:
|
| 269 |
+
tool = _tool_or_none(token)
|
| 270 |
+
if tool is None:
|
| 271 |
+
return NO_TOKEN_MD, "", [], {}, None
|
| 272 |
+
if not query or not query.strip():
|
| 273 |
+
return "Type a search query to begin.", "", [], {}, None
|
| 274 |
+
|
| 275 |
+
data = tool.search(query, country, int(pages or 1), device, time_period or None, location)
|
| 276 |
+
if data.get("status") != "success":
|
| 277 |
+
return _error_md(data), "", [], data, None
|
| 278 |
+
|
| 279 |
+
rows = _organic_rows(data)
|
| 280 |
+
counts = [
|
| 281 |
+
f"**{len(rows)}** organic results",
|
| 282 |
+
f"**{len(data.get('top_ads') or []) + len(data.get('bottom_ads') or [])}** ads",
|
| 283 |
+
f"**{len(data.get('related_questions') or [])}** People Also Ask",
|
| 284 |
+
f"**{len(data.get('related_searches') or [])}** related searches",
|
| 285 |
+
]
|
| 286 |
+
ai = data.get("ai_overview")
|
| 287 |
+
counts.append("AI Overview **yes**" if isinstance(ai, dict) and ai.get("state") == "complete" else "AI Overview **no**")
|
| 288 |
+
summary = (
|
| 289 |
+
" · ".join(counts)
|
| 290 |
+
+ f"\n\n{data.get('pages_fetched')} page(s) fetched · {data.get('credits_used')} credits used"
|
| 291 |
+
)
|
| 292 |
+
if data.get("warnings"):
|
| 293 |
+
summary += "\n\n" + "\n".join(f"- {w}" for w in data["warnings"])
|
| 294 |
+
|
| 295 |
+
table = [[r["position"], r["domain"], (r["title"] or "")[:90], r["link"]] for r in rows]
|
| 296 |
+
csv_path = _write_csv(rows, "google-serp")
|
| 297 |
+
return summary, render_serp(data), table, data, csv_path
|
| 298 |
+
|
| 299 |
+
|
| 300 |
+
def run_rank_check(
|
| 301 |
+
token: str, query: str, target_domain: str, country: str, pages: int,
|
| 302 |
+
device: str, time_period: str, location: str,
|
| 303 |
+
) -> Tuple[str, str, Dict[str, Any]]:
|
| 304 |
+
tool = _tool_or_none(token)
|
| 305 |
+
if tool is None:
|
| 306 |
+
return NO_TOKEN_MD, "", {}
|
| 307 |
+
if not query or not query.strip():
|
| 308 |
+
return "Type the keyword you want to check.", "", {}
|
| 309 |
+
domain = _domain(target_domain if "://" in (target_domain or "") else f"https://{(target_domain or '').strip()}")
|
| 310 |
+
if not domain:
|
| 311 |
+
return "Enter the domain to look for, e.g. `scrape.do`.", "", {}
|
| 312 |
+
|
| 313 |
+
data = tool.search(query, country, int(pages or 1), device, time_period or None, location)
|
| 314 |
+
if data.get("status") != "success":
|
| 315 |
+
return _error_md(data), "", data
|
| 316 |
+
|
| 317 |
+
organic = data.get("organic_results") or []
|
| 318 |
+
hits = [
|
| 319 |
+
r for r in organic
|
| 320 |
+
if _domain(r.get("link")) == domain or _domain(r.get("link")).endswith("." + domain)
|
| 321 |
+
]
|
| 322 |
+
looked = len(organic)
|
| 323 |
+
country_label = dict((gl, label) for label, gl in COUNTRY_CHOICES).get(country, country)
|
| 324 |
+
|
| 325 |
+
if hits:
|
| 326 |
+
best = hits[0]
|
| 327 |
+
lines = [
|
| 328 |
+
f"# #{best.get('overall_position')}",
|
| 329 |
+
f"**{domain}** ranks **#{best.get('overall_position')}** for “{query.strip()}” "
|
| 330 |
+
f"in {country_label} ({device}), page {best.get('page')}.",
|
| 331 |
+
]
|
| 332 |
+
if len(hits) > 1:
|
| 333 |
+
others = ", ".join(f"#{h.get('overall_position')}" for h in hits[1:])
|
| 334 |
+
lines.append(f"It also appears at {others}.")
|
| 335 |
+
above = [r for r in organic if r.get("overall_position", 0) < best.get("overall_position", 0)]
|
| 336 |
+
if above:
|
| 337 |
+
lines += ["", "**Ranking above you:**"]
|
| 338 |
+
lines += [f"{r.get('overall_position')}. {_domain(r.get('link'))}" for r in above[:10]]
|
| 339 |
+
else:
|
| 340 |
+
lines = [
|
| 341 |
+
"# Not found",
|
| 342 |
+
f"**{domain}** is not in the top {looked} organic results for “{query.strip()}” "
|
| 343 |
+
f"in {country_label}. Fetch more pages to look deeper.",
|
| 344 |
+
]
|
| 345 |
+
lines.append(f"\n{data.get('credits_used')} credits used.")
|
| 346 |
+
return "\n".join(lines), render_serp(data, highlight_domain=domain), data
|
| 347 |
+
|
| 348 |
+
|
| 349 |
+
# ----------------------------------------------------------------------- UI
|
| 350 |
+
|
| 351 |
+
|
| 352 |
+
STYLE = """
|
| 353 |
+
<style>
|
| 354 |
+
@import url('https://fonts.googleapis.com/css2?family=Roboto:wght@400;500;700&display=swap');
|
| 355 |
+
|
| 356 |
+
.gradio-container, .gradio-container button, .gradio-container input,
|
| 357 |
+
.gradio-container textarea, .gradio-container select {
|
| 358 |
+
font-family: 'Roboto', Arial, sans-serif;
|
| 359 |
+
}
|
| 360 |
+
.gradio-container { color: #202124; }
|
| 361 |
+
|
| 362 |
+
.gs-hero {
|
| 363 |
+
background: #ffffff; border: 1px solid #dadce0; border-radius: 16px;
|
| 364 |
+
padding: 28px 32px 24px 32px; margin-bottom: 16px;
|
| 365 |
+
}
|
| 366 |
+
.gs-hero h1 { color: #202124 !important; font-size: 2rem !important; font-weight: 500 !important;
|
| 367 |
+
margin: 0 0 10px 0 !important; letter-spacing: -0.01em; }
|
| 368 |
+
.gs-dots { display: flex; gap: 6px; margin-bottom: 14px; }
|
| 369 |
+
.gs-dots span { width: 12px; height: 12px; border-radius: 50%; display: inline-block; }
|
| 370 |
+
.gs-hero p { color: #4d5156 !important; margin: 0 0 8px 0 !important; font-size: 1rem !important; line-height: 1.55 !important; }
|
| 371 |
+
.gs-hero a { color: #1a0dab !important; }
|
| 372 |
+
.gs-cta {
|
| 373 |
+
display: inline-block; margin-top: 10px; padding: 11px 24px;
|
| 374 |
+
background: #1a73e8 !important; color: #ffffff !important;
|
| 375 |
+
border-radius: 999px; font-weight: 500 !important; text-decoration: none !important;
|
| 376 |
+
}
|
| 377 |
+
.gs-note { border-left: 3px solid #fbbc05; background: #fef7e0; padding: 12px 16px;
|
| 378 |
+
border-radius: 0 8px 8px 0; font-size: 0.9rem; margin-bottom: 12px; color: #202124; }
|
| 379 |
+
|
| 380 |
+
.gradio-container .primary, .gradio-container button.primary,
|
| 381 |
+
.gradio-container button[variant="primary"] {
|
| 382 |
+
background: #1a73e8 !important; background-image: none !important;
|
| 383 |
+
border-color: #1a73e8 !important; color: #ffffff !important;
|
| 384 |
+
border-radius: 999px !important; font-weight: 500 !important;
|
| 385 |
+
}
|
| 386 |
+
.gradio-container .primary:hover, .gradio-container button.primary:hover,
|
| 387 |
+
.gradio-container button[variant="primary"]:hover { background: #1765cc !important; }
|
| 388 |
+
.gradio-container .tab-nav button.selected { color: #1a73e8 !important; border-bottom-color: #1a73e8 !important; }
|
| 389 |
+
|
| 390 |
+
/* results page preview */
|
| 391 |
+
.gs-serp { background: #ffffff; border: 1px solid #dadce0; border-radius: 12px;
|
| 392 |
+
padding: 20px 24px; max-width: 760px; color: #202124; }
|
| 393 |
+
.gs-stats { color: #70757a; font-size: 0.85rem; margin-bottom: 14px; }
|
| 394 |
+
.gs-result { margin: 0 0 24px 0; padding: 2px 0; }
|
| 395 |
+
.gs-hit { background: #e8f0fe; border-radius: 8px; padding: 10px 12px; margin-left: -12px; }
|
| 396 |
+
.gs-cite { font-size: 0.85rem; color: #202124; display: flex; gap: 8px; align-items: baseline; flex-wrap: wrap; }
|
| 397 |
+
.gs-source { font-weight: 500; }
|
| 398 |
+
.gs-url { color: #4d5156; overflow-wrap: anywhere; }
|
| 399 |
+
.gs-pos { background: #f1f3f4; color: #5f6368; border-radius: 999px; padding: 1px 8px; font-size: 0.75rem; }
|
| 400 |
+
.gs-sponsored { font-weight: 700; }
|
| 401 |
+
.gs-title { display: block; color: #1a0dab !important; font-size: 1.25rem; line-height: 1.3;
|
| 402 |
+
margin: 4px 0 3px 0; text-decoration: none !important; }
|
| 403 |
+
.gs-title:hover { text-decoration: underline !important; }
|
| 404 |
+
.gs-snippet { color: #4d5156; font-size: 0.9rem; line-height: 1.58; }
|
| 405 |
+
.gs-date { color: #70757a; }
|
| 406 |
+
.gs-sitelinks { margin-top: 6px; font-size: 0.85rem; }
|
| 407 |
+
.gs-sitelinks a { color: #1a0dab !important; }
|
| 408 |
+
.gs-ai { border: 1px solid #dadce0; border-radius: 12px; padding: 16px 18px; margin-bottom: 24px;
|
| 409 |
+
background: linear-gradient(180deg, #f8faff 0%, #ffffff 100%); }
|
| 410 |
+
.gs-ai-label { font-weight: 500; color: #1a73e8; margin-bottom: 8px; }
|
| 411 |
+
.gs-ai p, .gs-ai li { color: #202124; font-size: 0.92rem; line-height: 1.6; }
|
| 412 |
+
.gs-ai table { border-collapse: collapse; margin: 8px 0; font-size: 0.85rem; }
|
| 413 |
+
.gs-ai th, .gs-ai td { border: 1px solid #dadce0; padding: 4px 8px; }
|
| 414 |
+
.gs-ai-refs { margin-top: 10px; font-size: 0.82rem; }
|
| 415 |
+
.gs-ai-refs a { color: #1a0dab !important; }
|
| 416 |
+
.gs-ai-refs span { color: #70757a; }
|
| 417 |
+
.gs-box { border-top: 1px solid #ebebeb; border-bottom: 1px solid #ebebeb; padding: 12px 0; margin-bottom: 24px; }
|
| 418 |
+
.gs-box-title { font-size: 1.1rem; margin-bottom: 8px; }
|
| 419 |
+
.gs-box ul { margin: 0; padding-left: 18px; }
|
| 420 |
+
.gs-paa { list-style: none; padding-left: 0 !important; }
|
| 421 |
+
.gs-paa li { border-bottom: 1px solid #ebebeb; padding: 10px 0; }
|
| 422 |
+
.gs-chips { display: flex; flex-wrap: wrap; gap: 8px; }
|
| 423 |
+
.gs-chip { background: #f1f3f4; border-radius: 999px; padding: 6px 12px; font-size: 0.88rem; }
|
| 424 |
+
.gs-kg { border: 1px solid #dadce0; border-radius: 12px; padding: 14px 16px; margin-bottom: 24px; }
|
| 425 |
+
.gs-kg-title { font-size: 1.4rem; }
|
| 426 |
+
.gs-kg-type { color: #70757a; margin-bottom: 6px; }
|
| 427 |
+
footer { display: none !important; }
|
| 428 |
+
</style>
|
| 429 |
+
"""
|
| 430 |
+
|
| 431 |
+
HERO = f"""
|
| 432 |
+
<div class="gs-hero">
|
| 433 |
+
<div class="gs-dots"><span style="background:#4285F4"></span><span style="background:#EA4335"></span><span style="background:#FBBC05"></span><span style="background:#34A853"></span></div>
|
| 434 |
+
<h1>Google Search Scraper</h1>
|
| 435 |
+
<p><strong>Any Google results page as structured data</strong> — organic rankings, ads, People Also Ask,
|
| 436 |
+
related searches, the AI Overview, knowledge panel and local pack, from any country and in any language.</p>
|
| 437 |
+
<p>No selectors to maintain: Google changes its page code constantly, so this uses the
|
| 438 |
+
<a href="https://scrape.do/documentation/google-scraper-api/search/" target="_blank" rel="noopener noreferrer">Scrape.do Google Search API</a>,
|
| 439 |
+
which returns the same clean fields whatever Google ships. Check where a site ranks, watch competitors' ads,
|
| 440 |
+
or feed live results to an AI agent.</p>
|
| 441 |
+
<a class="gs-cta" href="{SCRAPE_DO_SIGNUP_URL}" target="_blank" rel="noopener noreferrer">Get a free API token →</a>
|
| 442 |
+
</div>
|
| 443 |
+
"""
|
| 444 |
+
|
| 445 |
+
CREDIT_NOTE = """
|
| 446 |
+
<div class="gs-note">
|
| 447 |
+
Each results page costs 10 credits, so the free 1,000 credits cover about 100 searches.
|
| 448 |
+
If Google is still writing its AI Overview when the page comes back, it's fetched with one
|
| 449 |
+
follow-up call for 5 more credits.
|
| 450 |
+
</div>
|
| 451 |
+
"""
|
| 452 |
+
|
| 453 |
+
FOOTER_MD = f"""
|
| 454 |
+
---
|
| 455 |
+
|
| 456 |
+
### Use it as a smolagents Tool
|
| 457 |
+
|
| 458 |
+
```python
|
| 459 |
+
from smolagents import CodeAgent, InferenceClientModel
|
| 460 |
+
from tool import ScrapeDoGoogleSearchTool
|
| 461 |
+
|
| 462 |
+
agent = CodeAgent(
|
| 463 |
+
tools=[ScrapeDoGoogleSearchTool(api_token="YOUR_SCRAPEDO_TOKEN")],
|
| 464 |
+
model=InferenceClientModel(),
|
| 465 |
+
)
|
| 466 |
+
agent.run("Who ranks in the top 5 for 'best web scraping api' in Germany, and do any run ads?")
|
| 467 |
+
```
|
| 468 |
+
|
| 469 |
+
Or set `SCRAPE_DO_API_TOKEN` in the environment and instantiate with no arguments.
|
| 470 |
+
|
| 471 |
+
Want to parse the raw HTML yourself instead? The
|
| 472 |
+
[Google search scraping guide](https://scrape.do/blog/scraping-google-search-results/) covers both approaches.
|
| 473 |
+
|
| 474 |
+
[Scrape.do](https://scrape.do/) · [Google Search API docs](https://scrape.do/documentation/google-scraper-api/search/) · [Sign up free]({SCRAPE_DO_SIGNUP_URL})
|
| 475 |
+
"""
|
| 476 |
+
|
| 477 |
+
|
| 478 |
+
def build_demo() -> gr.Blocks:
|
| 479 |
+
with gr.Blocks(title="Google Search Scraper — Scrape.do") as demo:
|
| 480 |
+
gr.HTML(STYLE)
|
| 481 |
+
gr.HTML(HERO)
|
| 482 |
+
|
| 483 |
+
api_token = gr.Textbox(
|
| 484 |
+
label="Scrape.do API token",
|
| 485 |
+
placeholder="Paste your token here",
|
| 486 |
+
type="password",
|
| 487 |
+
info="Free tier: 1,000 credits a month, about 100 searches.",
|
| 488 |
+
)
|
| 489 |
+
|
| 490 |
+
with gr.Row():
|
| 491 |
+
country = gr.Dropdown(label="Search from", choices=COUNTRY_CHOICES, value="us")
|
| 492 |
+
device = gr.Radio(label="Device", choices=["desktop", "mobile"], value="desktop")
|
| 493 |
+
pages = gr.Slider(label="Result pages", minimum=1, maximum=5, step=1, value=1,
|
| 494 |
+
info="10 credits per page.")
|
| 495 |
+
|
| 496 |
+
with gr.Accordion("More filters", open=False):
|
| 497 |
+
with gr.Row():
|
| 498 |
+
time_period = gr.Dropdown(label="Time", choices=TIME_CHOICES, value="")
|
| 499 |
+
location = gr.Textbox(
|
| 500 |
+
label="City (optional)",
|
| 501 |
+
placeholder="Austin,Texas,United States",
|
| 502 |
+
info="Google's own place name. Must be in the country above.",
|
| 503 |
+
)
|
| 504 |
+
|
| 505 |
+
gr.HTML(CREDIT_NOTE)
|
| 506 |
+
|
| 507 |
+
with gr.Tabs():
|
| 508 |
+
with gr.Tab("Search results"):
|
| 509 |
+
query = gr.Textbox(label="Search Google", placeholder="best web scraping api")
|
| 510 |
+
search_btn = gr.Button("Google Search", variant="primary")
|
| 511 |
+
search_summary = gr.Markdown()
|
| 512 |
+
serp_view = gr.HTML()
|
| 513 |
+
serp_table = gr.Dataframe(headers=TABLE_HEADERS, wrap=True, label="Organic results")
|
| 514 |
+
serp_csv = gr.File(label="Download CSV")
|
| 515 |
+
with gr.Accordion("Raw JSON response", open=False):
|
| 516 |
+
serp_json = gr.JSON()
|
| 517 |
+
|
| 518 |
+
search_btn.click(
|
| 519 |
+
fn=run_search,
|
| 520 |
+
inputs=[api_token, query, country, pages, device, time_period, location],
|
| 521 |
+
outputs=[search_summary, serp_view, serp_table, serp_json, serp_csv],
|
| 522 |
+
)
|
| 523 |
+
query.submit(
|
| 524 |
+
fn=run_search,
|
| 525 |
+
inputs=[api_token, query, country, pages, device, time_period, location],
|
| 526 |
+
outputs=[search_summary, serp_view, serp_table, serp_json, serp_csv],
|
| 527 |
+
)
|
| 528 |
+
|
| 529 |
+
with gr.Tab("Rank checker"):
|
| 530 |
+
gr.Markdown(
|
| 531 |
+
"Where does a website rank for a keyword? Enter both, and the matching "
|
| 532 |
+
"result is highlighted on the page with everyone ranking above it."
|
| 533 |
+
)
|
| 534 |
+
with gr.Row():
|
| 535 |
+
rank_query = gr.Textbox(label="Keyword", placeholder="web scraping api", scale=3)
|
| 536 |
+
rank_domain = gr.Textbox(label="Your domain", placeholder="scrape.do", scale=2)
|
| 537 |
+
rank_btn = gr.Button("Check ranking", variant="primary")
|
| 538 |
+
rank_summary = gr.Markdown()
|
| 539 |
+
rank_view = gr.HTML()
|
| 540 |
+
with gr.Accordion("Raw JSON response", open=False):
|
| 541 |
+
rank_json = gr.JSON()
|
| 542 |
+
|
| 543 |
+
rank_btn.click(
|
| 544 |
+
fn=run_rank_check,
|
| 545 |
+
inputs=[api_token, rank_query, rank_domain, country, pages, device, time_period, location],
|
| 546 |
+
outputs=[rank_summary, rank_view, rank_json],
|
| 547 |
+
)
|
| 548 |
+
|
| 549 |
+
gr.Markdown(FOOTER_MD)
|
| 550 |
+
|
| 551 |
+
return demo
|
| 552 |
+
|
| 553 |
+
|
| 554 |
+
# Hugging Face Spaces launches the module-level `demo` object rather than
|
| 555 |
+
# calling launch() itself, so build it at import time.
|
| 556 |
+
demo = build_demo()
|
| 557 |
+
|
| 558 |
+
if __name__ == "__main__":
|
| 559 |
+
demo.launch()
|
requirements.txt
ADDED
|
@@ -0,0 +1,4 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
gradio
|
| 2 |
+
requests
|
| 3 |
+
smolagents
|
| 4 |
+
spaces
|
tool.py
ADDED
|
@@ -0,0 +1,248 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from __future__ import annotations
|
| 2 |
+
|
| 3 |
+
import json
|
| 4 |
+
import os
|
| 5 |
+
import urllib.parse
|
| 6 |
+
from typing import Any, Dict, List, Optional
|
| 7 |
+
|
| 8 |
+
import requests
|
| 9 |
+
from smolagents.tools import Tool
|
| 10 |
+
|
| 11 |
+
|
| 12 |
+
SCRAPE_DO_SIGNUP_URL = (
|
| 13 |
+
"https://dashboard.scrape.do/sign-up?utm_source=huggingface&utm_medium=google-search-scraper"
|
| 14 |
+
)
|
| 15 |
+
SEARCH_ENDPOINT = "https://api.scrape.do/plugin/google/search"
|
| 16 |
+
AI_OVERVIEW_ENDPOINT = "https://api.scrape.do/plugin/google/search/ai-overview"
|
| 17 |
+
|
| 18 |
+
# Credit costs from the Scrape.do docs.
|
| 19 |
+
CREDITS_PER_SEARCH = 10
|
| 20 |
+
CREDITS_PER_AI_OVERVIEW = 5
|
| 21 |
+
|
| 22 |
+
# (label, gl, hl, google_domain)
|
| 23 |
+
COUNTRIES: List[tuple] = [
|
| 24 |
+
("United States", "us", "en", "google.com"),
|
| 25 |
+
("United Kingdom", "gb", "en", "google.co.uk"),
|
| 26 |
+
("Germany", "de", "de", "google.de"),
|
| 27 |
+
("France", "fr", "fr", "google.fr"),
|
| 28 |
+
("Spain", "es", "es", "google.es"),
|
| 29 |
+
("Italy", "it", "it", "google.it"),
|
| 30 |
+
("Netherlands", "nl", "nl", "google.nl"),
|
| 31 |
+
("Turkey", "tr", "tr", "google.com.tr"),
|
| 32 |
+
("Canada", "ca", "en", "google.ca"),
|
| 33 |
+
("Australia", "au", "en", "google.com.au"),
|
| 34 |
+
("India", "in", "en", "google.co.in"),
|
| 35 |
+
("Japan", "jp", "ja", "google.co.jp"),
|
| 36 |
+
("Brazil", "br", "pt", "google.com.br"),
|
| 37 |
+
("Mexico", "mx", "es", "google.com.mx"),
|
| 38 |
+
]
|
| 39 |
+
COUNTRY_BY_GL = {gl: (label, gl, hl, domain) for label, gl, hl, domain in COUNTRIES}
|
| 40 |
+
|
| 41 |
+
TIME_PERIODS = ["last_hour", "last_day", "last_week", "last_month", "last_year"]
|
| 42 |
+
|
| 43 |
+
|
| 44 |
+
class ScrapeDoGoogleSearchTool(Tool):
|
| 45 |
+
name = "google_search_scraper"
|
| 46 |
+
description = """
|
| 47 |
+
Run a Google search and get the whole results page back as structured JSON:
|
| 48 |
+
organic results with positions, top and bottom ads, People Also Ask
|
| 49 |
+
questions, related searches, the AI Overview, knowledge panel, local map
|
| 50 |
+
pack, top stories, videos and forum discussions. Supports any country and
|
| 51 |
+
language, desktop or mobile layouts, time filters and pagination.
|
| 52 |
+
Powered by Scrape.do's Google Search API, which handles proxies, CAPTCHAs
|
| 53 |
+
and Google's changing HTML. Use this for rank tracking, SEO research,
|
| 54 |
+
ad monitoring, or grounding an answer in live search results.
|
| 55 |
+
"""
|
| 56 |
+
output_type = "string"
|
| 57 |
+
|
| 58 |
+
def __init__(self, api_token: Optional[str] = None) -> None:
|
| 59 |
+
self._api_token = api_token
|
| 60 |
+
self.inputs = {
|
| 61 |
+
"query": {
|
| 62 |
+
"type": "string",
|
| 63 |
+
"description": "The search query, exactly as you would type it into Google.",
|
| 64 |
+
},
|
| 65 |
+
"country": {
|
| 66 |
+
"type": "string",
|
| 67 |
+
"description": (
|
| 68 |
+
"Two-letter country code to search from, e.g. 'us', 'gb', 'de', 'tr'. "
|
| 69 |
+
"Sets the ranking country, interface language and Google domain. Defaults to 'us'."
|
| 70 |
+
),
|
| 71 |
+
"nullable": True,
|
| 72 |
+
},
|
| 73 |
+
"pages": {
|
| 74 |
+
"type": "integer",
|
| 75 |
+
"description": "How many result pages to fetch (1-5). Each page costs 10 credits. Defaults to 1.",
|
| 76 |
+
"nullable": True,
|
| 77 |
+
},
|
| 78 |
+
"device": {
|
| 79 |
+
"type": "string",
|
| 80 |
+
"description": "'desktop' or 'mobile' results layout. Defaults to 'desktop'.",
|
| 81 |
+
"nullable": True,
|
| 82 |
+
},
|
| 83 |
+
"time_period": {
|
| 84 |
+
"type": "string",
|
| 85 |
+
"description": (
|
| 86 |
+
"Only return results from a recent window: last_hour, last_day, "
|
| 87 |
+
"last_week, last_month or last_year."
|
| 88 |
+
),
|
| 89 |
+
"nullable": True,
|
| 90 |
+
},
|
| 91 |
+
"location": {
|
| 92 |
+
"type": "string",
|
| 93 |
+
"description": (
|
| 94 |
+
"City-level location in Google's format, e.g. 'Austin,Texas,United States' "
|
| 95 |
+
"or 'Istanbul,Istanbul,Turkey'. Must match the country code."
|
| 96 |
+
),
|
| 97 |
+
"nullable": True,
|
| 98 |
+
},
|
| 99 |
+
}
|
| 100 |
+
super().__init__()
|
| 101 |
+
|
| 102 |
+
# ------------------------------------------------------------------ token
|
| 103 |
+
|
| 104 |
+
def _resolve_token(self) -> Optional[str]:
|
| 105 |
+
if self._api_token:
|
| 106 |
+
return self._api_token.strip()
|
| 107 |
+
env_token = os.getenv("SCRAPE_DO_API_TOKEN")
|
| 108 |
+
return env_token.strip() if env_token else None
|
| 109 |
+
|
| 110 |
+
# ------------------------------------------------------------------ calls
|
| 111 |
+
|
| 112 |
+
def _get(self, url: str, params: Dict[str, Any]) -> Dict[str, Any]:
|
| 113 |
+
full_url = url + "?" + urllib.parse.urlencode(params, quote_via=urllib.parse.quote_plus)
|
| 114 |
+
try:
|
| 115 |
+
response = requests.get(full_url, timeout=120)
|
| 116 |
+
except requests.exceptions.RequestException as exc:
|
| 117 |
+
return {"_error": f"Request failed: {exc}"}
|
| 118 |
+
|
| 119 |
+
try:
|
| 120 |
+
data = response.json()
|
| 121 |
+
except ValueError:
|
| 122 |
+
return {"_error": f"Unexpected non-JSON response (HTTP {response.status_code})."}
|
| 123 |
+
|
| 124 |
+
if response.status_code == 401:
|
| 125 |
+
return {"_error": "Scrape.do rejected the token (HTTP 401).", "_auth": True}
|
| 126 |
+
if not response.ok:
|
| 127 |
+
message = data.get("message") or data.get("error") or f"HTTP {response.status_code}"
|
| 128 |
+
return {"_error": f"Scrape.do returned an error: {message}", "_status": response.status_code}
|
| 129 |
+
return data
|
| 130 |
+
|
| 131 |
+
def search(
|
| 132 |
+
self,
|
| 133 |
+
query: str,
|
| 134 |
+
country: str = "us",
|
| 135 |
+
pages: int = 1,
|
| 136 |
+
device: str = "desktop",
|
| 137 |
+
time_period: Optional[str] = None,
|
| 138 |
+
location: Optional[str] = None,
|
| 139 |
+
fetch_deferred_ai: bool = True,
|
| 140 |
+
) -> Dict[str, Any]:
|
| 141 |
+
token = self._resolve_token()
|
| 142 |
+
if not token:
|
| 143 |
+
return {
|
| 144 |
+
"status": "error",
|
| 145 |
+
"errorMessage": "Missing Scrape.do API token.",
|
| 146 |
+
"hint": f"Get a free token at {SCRAPE_DO_SIGNUP_URL}",
|
| 147 |
+
}
|
| 148 |
+
if not query or not query.strip():
|
| 149 |
+
return {"status": "error", "errorMessage": "Enter a search query."}
|
| 150 |
+
|
| 151 |
+
gl = (country or "us").strip().lower()
|
| 152 |
+
_, gl, hl, domain = COUNTRY_BY_GL.get(gl, ("", gl, "en", "google.com"))
|
| 153 |
+
pages = max(1, min(int(pages or 1), 5))
|
| 154 |
+
|
| 155 |
+
base: Dict[str, Any] = {
|
| 156 |
+
"token": token,
|
| 157 |
+
"q": query.strip(),
|
| 158 |
+
"gl": gl,
|
| 159 |
+
"hl": hl,
|
| 160 |
+
"google_domain": domain,
|
| 161 |
+
"device": device if device in ("desktop", "mobile") else "desktop",
|
| 162 |
+
"resolveGoto": "true",
|
| 163 |
+
}
|
| 164 |
+
if time_period in TIME_PERIODS:
|
| 165 |
+
base["time_period"] = time_period
|
| 166 |
+
if location and location.strip():
|
| 167 |
+
base["location"] = location.strip()
|
| 168 |
+
|
| 169 |
+
combined: Dict[str, Any] = {}
|
| 170 |
+
organic: List[Dict[str, Any]] = []
|
| 171 |
+
credits = 0
|
| 172 |
+
pages_fetched = 0
|
| 173 |
+
warnings: List[str] = []
|
| 174 |
+
|
| 175 |
+
for page_index in range(pages):
|
| 176 |
+
# The docs say to send `start` from the very first request of a
|
| 177 |
+
# multi-page sweep so all pages come from one coherent session.
|
| 178 |
+
params = dict(base, start=page_index * 10)
|
| 179 |
+
data = self._get(SEARCH_ENDPOINT, params)
|
| 180 |
+
|
| 181 |
+
if "_error" in data:
|
| 182 |
+
if page_index == 0:
|
| 183 |
+
result = {"status": "error", "errorMessage": data["_error"]}
|
| 184 |
+
if data.get("_auth"):
|
| 185 |
+
result["hint"] = f"Check your token or create one at {SCRAPE_DO_SIGNUP_URL}"
|
| 186 |
+
return result
|
| 187 |
+
warnings.append(f"Page {page_index + 1}: {data['_error']}")
|
| 188 |
+
break
|
| 189 |
+
|
| 190 |
+
credits += CREDITS_PER_SEARCH
|
| 191 |
+
pages_fetched += 1
|
| 192 |
+
|
| 193 |
+
for item in data.get("organic_results") or []:
|
| 194 |
+
item = dict(item)
|
| 195 |
+
item["page"] = page_index + 1
|
| 196 |
+
item["overall_position"] = len(organic) + 1
|
| 197 |
+
organic.append(item)
|
| 198 |
+
|
| 199 |
+
if page_index == 0:
|
| 200 |
+
combined = data
|
| 201 |
+
# The AI Overview can come back "deferred". Its session key
|
| 202 |
+
# expires after 90 seconds, so fetch it before any more pages.
|
| 203 |
+
ai = data.get("ai_overview")
|
| 204 |
+
if (
|
| 205 |
+
fetch_deferred_ai
|
| 206 |
+
and isinstance(ai, dict)
|
| 207 |
+
and ai.get("state") == "deferred"
|
| 208 |
+
and ai.get("session_key")
|
| 209 |
+
):
|
| 210 |
+
follow = self._get(
|
| 211 |
+
AI_OVERVIEW_ENDPOINT, {"token": token, "session_key": ai["session_key"]}
|
| 212 |
+
)
|
| 213 |
+
if "_error" in follow:
|
| 214 |
+
warnings.append(f"AI Overview follow-up: {follow['_error']}")
|
| 215 |
+
else:
|
| 216 |
+
credits += CREDITS_PER_AI_OVERVIEW
|
| 217 |
+
combined["ai_overview"] = follow
|
| 218 |
+
if not (data.get("pagination") or {}).get("next"):
|
| 219 |
+
break
|
| 220 |
+
|
| 221 |
+
combined["organic_results"] = organic
|
| 222 |
+
combined["status"] = "success"
|
| 223 |
+
combined["pages_fetched"] = pages_fetched
|
| 224 |
+
combined["credits_used"] = credits
|
| 225 |
+
if warnings:
|
| 226 |
+
combined["warnings"] = warnings
|
| 227 |
+
return combined
|
| 228 |
+
|
| 229 |
+
# ---------------------------------------------------------------- forward
|
| 230 |
+
|
| 231 |
+
def forward(
|
| 232 |
+
self,
|
| 233 |
+
query: str,
|
| 234 |
+
country: Optional[str] = None,
|
| 235 |
+
pages: Optional[int] = None,
|
| 236 |
+
device: Optional[str] = None,
|
| 237 |
+
time_period: Optional[str] = None,
|
| 238 |
+
location: Optional[str] = None,
|
| 239 |
+
) -> str:
|
| 240 |
+
result = self.search(
|
| 241 |
+
query=query,
|
| 242 |
+
country=country or "us",
|
| 243 |
+
pages=pages or 1,
|
| 244 |
+
device=device or "desktop",
|
| 245 |
+
time_period=time_period,
|
| 246 |
+
location=location,
|
| 247 |
+
)
|
| 248 |
+
return json.dumps(result, ensure_ascii=False, indent=2)
|