bugrahans commited on
Commit
4aeb13f
·
verified ·
1 Parent(s): 9da0572

Upload 4 files

Browse files
Files changed (4) hide show
  1. README.md +87 -7
  2. app.py +559 -0
  3. requirements.txt +4 -0
  4. tool.py +248 -0
README.md CHANGED
@@ -1,14 +1,94 @@
1
  ---
2
  title: Google Search Scraper
3
- emoji: 🏃
4
- colorFrom: gray
5
- colorTo: yellow
 
6
  sdk: gradio
7
- sdk_version: 6.28.0
8
- python_version: '3.12'
9
  app_file: app.py
10
  pinned: false
11
- license: mit
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
12
  ---
13
 
14
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  title: Google Search Scraper
3
+ short_description: Any Google results page as JSON, plus a rank checker
4
+ emoji: 🔎
5
+ colorFrom: blue
6
+ colorTo: green
7
  sdk: gradio
8
+ sdk_version: 6.0.2
 
9
  app_file: app.py
10
  pinned: false
11
+ tags:
12
+ - google
13
+ - google-search
14
+ - serp
15
+ - serp-api
16
+ - google-scraper
17
+ - rank-tracker
18
+ - seo
19
+ - ai-overview
20
+ - people-also-ask
21
+ - web-scraping
22
+ - scraper
23
+ - data-extraction
24
+ - smolagents
25
+ - tool
26
+ - scrape.do
27
  ---
28
 
29
+ # Google Search Scraper — Powered by Scrape.do
30
+
31
+ Run a Google search from any country and get the whole results page back as
32
+ structured data: organic rankings, ads, People Also Ask, related searches, the
33
+ AI Overview, knowledge panel, local pack, top stories and videos.
34
+
35
+ ## 🔑 Get your free API token
36
+
37
+ 👉 **[Sign up free on Scrape.do](https://dashboard.scrape.do/sign-up?utm_source=huggingface&utm_medium=google-search-scraper)** — 1,000 free credits per month, no credit card required. One results page costs 10 credits, so that's about 100 searches.
38
+
39
+ ## Two tabs
40
+
41
+ | Tab | Input | Output |
42
+ | --- | --- | --- |
43
+ | **Search results** | Any query | A live preview of the results page, a table of organic results, CSV download and the raw JSON |
44
+ | **Rank checker** | Keyword + your domain | Your position, the page it's on, and every site ranking above you — highlighted on the results page |
45
+
46
+ Both tabs share the same settings: 14 countries (each with its own Google domain
47
+ and language), desktop or mobile results, 1–5 pages deep, a time filter, and
48
+ optional city-level location.
49
+
50
+ ## Why an API instead of parsing Google's HTML
51
+
52
+ Google renames the code behind its results page without warning, so a scraper
53
+ built on page selectors breaks silently. This Space uses the
54
+ [Scrape.do Google Search API](https://scrape.do/documentation/google-scraper-api/search/),
55
+ which returns the same field names whatever Google changes, and handles
56
+ proxies and CAPTCHAs.
57
+
58
+ Two details are handled for you:
59
+
60
+ - **Multi-page searches stay consistent.** Every page request carries a start
61
+ offset from the first page on, so all pages come from the same browsing
62
+ session instead of reshuffling or repeating results.
63
+ - **Late AI Overviews are collected.** When Google is still writing its AI
64
+ Overview as the page returns, the tool fetches the finished version with one
65
+ follow-up call (5 credits), immediately — the key for it expires in 90 seconds.
66
+
67
+ ## Use as a smolagents Tool
68
+
69
+ ```python
70
+ from smolagents import CodeAgent, InferenceClientModel
71
+ from tool import ScrapeDoGoogleSearchTool
72
+
73
+ agent = CodeAgent(
74
+ tools=[ScrapeDoGoogleSearchTool(api_token="YOUR_SCRAPEDO_TOKEN")],
75
+ model=InferenceClientModel(),
76
+ )
77
+ agent.run("Who ranks in the top 5 for 'best web scraping api' in Germany, and do any run ads?")
78
+ ```
79
+
80
+ You can also set `SCRAPE_DO_API_TOKEN` in your environment and instantiate
81
+ `ScrapeDoGoogleSearchTool()` with no arguments.
82
+
83
+ ## Inputs
84
+
85
+ - `query` *(required)* — The search, as you'd type it into Google.
86
+ - `country` *(optional)* — Two-letter code (`us`, `gb`, `de`, `tr`…). Sets ranking country, language and Google domain. Defaults to `us`.
87
+ - `pages` *(optional)* — 1–5 result pages. 10 credits each. Defaults to `1`.
88
+ - `device` *(optional)* — `desktop` or `mobile`.
89
+ - `time_period` *(optional)* — `last_hour`, `last_day`, `last_week`, `last_month`, `last_year`.
90
+ - `location` *(optional)* — City in Google's format, e.g. `Austin,Texas,United States`.
91
+
92
+ ---
93
+
94
+ Learn more at [scrape.do](https://scrape.do/) · [Documentation](https://scrape.do/documentation/) · [Google search scraping guide](https://scrape.do/blog/scraping-google-search-results/)
app.py ADDED
@@ -0,0 +1,559 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from __future__ import annotations
2
+
3
+ import csv
4
+ import html
5
+ import tempfile
6
+ from typing import Any, Dict, List, Optional, Tuple
7
+ from urllib.parse import urlparse
8
+
9
+ import gradio as gr
10
+
11
+ from tool import COUNTRIES, SCRAPE_DO_SIGNUP_URL, TIME_PERIODS, ScrapeDoGoogleSearchTool
12
+
13
+ # ZeroGPU hardware refuses to keep a Space alive unless it finds at least one
14
+ # @spaces.GPU function at startup. This app is pure HTTP and never touches a GPU,
15
+ # so the probe below exists only to satisfy that check. It is never called.
16
+ try:
17
+ import spaces
18
+
19
+ @spaces.GPU(duration=1)
20
+ def _zerogpu_probe() -> str:
21
+ return "ok"
22
+
23
+ except ImportError:
24
+ pass
25
+
26
+
27
+ COUNTRY_CHOICES = [(label, gl) for label, gl, _, _ in COUNTRIES]
28
+ TIME_CHOICES = [("Any time", "")] + [
29
+ (value.replace("_", " ").capitalize(), value) for value in TIME_PERIODS
30
+ ]
31
+
32
+ NO_TOKEN_MD = (
33
+ "### Paste your Scrape.do API token above\n\n"
34
+ f"No token yet? [Create a free account]({SCRAPE_DO_SIGNUP_URL}) — "
35
+ "1,000 free credits a month, no credit card. One search costs 10 credits."
36
+ )
37
+
38
+
39
+ # --------------------------------------------------------------------- helpers
40
+
41
+
42
+ def _esc(value: Any) -> str:
43
+ return html.escape(str(value or ""), quote=True)
44
+
45
+
46
+ def _safe_link(url: Any) -> str:
47
+ url = str(url or "")
48
+ return url if url.startswith(("http://", "https://")) else "#"
49
+
50
+
51
+ def _domain(url: Any) -> str:
52
+ try:
53
+ host = urlparse(str(url or "")).netloc.lower()
54
+ except ValueError:
55
+ return ""
56
+ return host[4:] if host.startswith("www.") else host
57
+
58
+
59
+ def _tool_or_none(token: str) -> Optional[ScrapeDoGoogleSearchTool]:
60
+ if not token or not token.strip():
61
+ return None
62
+ return ScrapeDoGoogleSearchTool(api_token=token.strip())
63
+
64
+
65
+ def _write_csv(rows: List[Dict[str, Any]], prefix: str) -> Optional[str]:
66
+ if not rows:
67
+ return None
68
+ fieldnames: List[str] = []
69
+ for row in rows:
70
+ for key in row:
71
+ if key not in fieldnames:
72
+ fieldnames.append(key)
73
+ handle = tempfile.NamedTemporaryFile(
74
+ mode="w", suffix=".csv", prefix=f"{prefix}-", delete=False,
75
+ newline="", encoding="utf-8-sig",
76
+ )
77
+ writer = csv.DictWriter(handle, fieldnames=fieldnames)
78
+ writer.writeheader()
79
+ for row in rows:
80
+ writer.writerow({key: row.get(key, "") for key in fieldnames})
81
+ handle.close()
82
+ return handle.name
83
+
84
+
85
+ def _error_md(data: Dict[str, Any]) -> str:
86
+ out = f"**That search didn't go through.**\n\n{data.get('errorMessage') or 'Unknown error'}"
87
+ if data.get("hint"):
88
+ out += f"\n\n{data['hint']}"
89
+ return out
90
+
91
+
92
+ # ------------------------------------------------------------ SERP rendering
93
+
94
+
95
+ def _render_ai_blocks(blocks: List[Dict[str, Any]]) -> str:
96
+ parts: List[str] = []
97
+
98
+ def render_list(items: List[Dict[str, Any]]) -> str:
99
+ inner = ""
100
+ for item in items or []:
101
+ inner += f"<li>{_esc(item.get('snippet'))}"
102
+ if item.get("list"):
103
+ inner += render_list(item["list"])
104
+ inner += "</li>"
105
+ return f"<ul>{inner}</ul>"
106
+
107
+ for block in blocks or []:
108
+ kind = block.get("type")
109
+ if kind == "heading":
110
+ parts.append(f"<h4>{_esc(block.get('snippet'))}</h4>")
111
+ elif kind == "list":
112
+ if block.get("snippet"):
113
+ parts.append(f"<p>{_esc(block['snippet'])}</p>")
114
+ parts.append(render_list(block.get("list") or []))
115
+ elif kind == "table":
116
+ head = "".join(f"<th>{_esc(h)}</th>" for h in block.get("table_headers") or [])
117
+ rows = "".join(
118
+ "<tr>" + "".join(f"<td>{_esc(c)}</td>" for c in row) + "</tr>"
119
+ for row in block.get("table_rows") or []
120
+ )
121
+ parts.append(f"<table><thead><tr>{head}</tr></thead><tbody>{rows}</tbody></table>")
122
+ else:
123
+ parts.append(f"<p>{_esc(block.get('snippet'))}</p>")
124
+ return "".join(parts)
125
+
126
+
127
+ def _render_ad(ad: Dict[str, Any]) -> str:
128
+ link = _safe_link(ad.get("url") or ad.get("link"))
129
+ return (
130
+ '<div class="gs-result">'
131
+ f'<div class="gs-cite"><span class="gs-sponsored">Sponsored</span> {_esc(ad.get("displayed_url") or _domain(link))}</div>'
132
+ f'<a class="gs-title" href="{_esc(link)}" target="_blank" rel="noopener noreferrer">{_esc(ad.get("title"))}</a>'
133
+ f'<div class="gs-snippet">{_esc(ad.get("description"))}</div>'
134
+ "</div>"
135
+ )
136
+
137
+
138
+ def render_serp(data: Dict[str, Any], highlight_domain: str = "") -> str:
139
+ info = data.get("search_information") or {}
140
+ out: List[str] = ['<div class="gs-serp">']
141
+
142
+ stats = []
143
+ if info.get("total_results"):
144
+ stats.append(f"About {int(info['total_results']):,} results")
145
+ if info.get("time_taken_displayed"):
146
+ stats.append(f"({info['time_taken_displayed']} seconds)")
147
+ if stats:
148
+ out.append(f'<div class="gs-stats">{_esc(" ".join(stats))}</div>')
149
+ if info.get("results_for"):
150
+ out.append(f'<div class="gs-stats">Showing results for <b>{_esc(info["results_for"])}</b></div>')
151
+
152
+ ai = data.get("ai_overview")
153
+ if isinstance(ai, dict) and ai.get("state") == "complete" and ai.get("text_blocks"):
154
+ refs = "".join(
155
+ f'<li><a href="{_esc(_safe_link(ref.get("link")))}" target="_blank" rel="noopener noreferrer">'
156
+ f'{_esc(ref.get("title") or ref.get("source"))}</a> <span>{_esc(ref.get("source"))}</span></li>'
157
+ for ref in (ai.get("references") or [])[:8]
158
+ )
159
+ out.append(
160
+ '<div class="gs-ai"><div class="gs-ai-label">✦ AI Overview</div>'
161
+ f"{_render_ai_blocks(ai['text_blocks'])}"
162
+ + (f'<div class="gs-ai-refs"><b>Sources</b><ol>{refs}</ol></div>' if refs else "")
163
+ + "</div>"
164
+ )
165
+
166
+ for ad in data.get("top_ads") or []:
167
+ out.append(_render_ad(ad))
168
+
169
+ kg = data.get("knowledge_graph")
170
+ if isinstance(kg, dict) and kg.get("title"):
171
+ facts = "".join(
172
+ f"<div><b>{_esc(f.get('label'))}:</b> {_esc(f.get('value'))}</div>"
173
+ for f in (kg.get("facts") or [])[:6]
174
+ )
175
+ out.append(
176
+ '<div class="gs-kg">'
177
+ f"<div class=\"gs-kg-title\">{_esc(kg.get('title'))}</div>"
178
+ f"<div class=\"gs-kg-type\">{_esc(kg.get('type'))}</div>"
179
+ f"<p>{_esc(kg.get('description'))}</p>{facts}</div>"
180
+ )
181
+
182
+ local = data.get("local_results")
183
+ if isinstance(local, dict) and local.get("places"):
184
+ places = "".join(
185
+ f"<li><b>{_esc(p.get('title'))}</b> · {_esc(p.get('rating') or '—')}★ {_esc(p.get('price'))}</li>"
186
+ for p in local["places"][:5]
187
+ )
188
+ out.append(f'<div class="gs-box"><div class="gs-box-title">Places</div><ul>{places}</ul></div>')
189
+
190
+ paa = data.get("related_questions") or []
191
+ organic = data.get("organic_results") or []
192
+ highlight = highlight_domain.lower().removeprefix("www.")
193
+
194
+ for index, result in enumerate(organic):
195
+ link = _safe_link(result.get("link"))
196
+ is_hit = bool(highlight) and (
197
+ _domain(link) == highlight or _domain(link).endswith("." + highlight)
198
+ )
199
+ cls = "gs-result gs-hit" if is_hit else "gs-result"
200
+ page_badge = (
201
+ f'<span class="gs-pos">#{result.get("overall_position", index + 1)}</span>'
202
+ )
203
+ sitelinks = ""
204
+ sl = result.get("sitelinks") or {}
205
+ sl_items = (sl.get("inline") or []) + (sl.get("expanded") or [])
206
+ if sl_items:
207
+ sitelinks = '<div class="gs-sitelinks">' + " · ".join(
208
+ f'<a href="{_esc(_safe_link(s.get("link")))}" target="_blank" rel="noopener noreferrer">{_esc(s.get("title"))}</a>'
209
+ for s in sl_items[:4]
210
+ ) + "</div>"
211
+ date = f'<span class="gs-date">{_esc(result.get("date"))} — </span>' if result.get("date") else ""
212
+ out.append(
213
+ f'<div class="{cls}">'
214
+ f'<div class="gs-cite">{page_badge}<span class="gs-source">{_esc(result.get("source") or _domain(link))}</span>'
215
+ f'<span class="gs-url">{_esc(result.get("displayed_link") or link)}</span></div>'
216
+ f'<a class="gs-title" href="{_esc(link)}" target="_blank" rel="noopener noreferrer">{_esc(result.get("title"))}</a>'
217
+ f'<div class="gs-snippet">{date}{_esc(result.get("snippet"))}</div>{sitelinks}</div>'
218
+ )
219
+ # Google puts People Also Ask after the first few results.
220
+ if index == 2 and paa:
221
+ qs = "".join(f"<li>{_esc(q.get('question'))}</li>" for q in paa[:6])
222
+ out.append(f'<div class="gs-box"><div class="gs-box-title">People also ask</div><ul class="gs-paa">{qs}</ul></div>')
223
+ paa = []
224
+
225
+ if paa:
226
+ qs = "".join(f"<li>{_esc(q.get('question'))}</li>" for q in paa[:6])
227
+ out.append(f'<div class="gs-box"><div class="gs-box-title">People also ask</div><ul class="gs-paa">{qs}</ul></div>')
228
+
229
+ for ad in data.get("bottom_ads") or []:
230
+ out.append(_render_ad(ad))
231
+
232
+ related = data.get("related_searches") or []
233
+ if related:
234
+ chips = "".join(f'<span class="gs-chip">🔍 {_esc(r.get("query"))}</span>' for r in related[:10])
235
+ out.append(f'<div class="gs-box"><div class="gs-box-title">People also search for</div><div class="gs-chips">{chips}</div></div>')
236
+
237
+ if not organic:
238
+ out.append('<div class="gs-stats">No organic results came back for this query.</div>')
239
+
240
+ out.append("</div>")
241
+ return "".join(out)
242
+
243
+
244
+ def _organic_rows(data: Dict[str, Any]) -> List[Dict[str, Any]]:
245
+ return [
246
+ {
247
+ "position": r.get("overall_position"),
248
+ "page": r.get("page"),
249
+ "title": r.get("title"),
250
+ "link": r.get("link"),
251
+ "domain": _domain(r.get("link")),
252
+ "snippet": r.get("snippet"),
253
+ "date": r.get("date"),
254
+ "source": r.get("source"),
255
+ }
256
+ for r in data.get("organic_results") or []
257
+ ]
258
+
259
+
260
+ # ---------------------------------------------------------------- handlers
261
+
262
+
263
+ TABLE_HEADERS = ["#", "Domain", "Title", "Link"]
264
+
265
+
266
+ def run_search(
267
+ token: str, query: str, country: str, pages: int, device: str, time_period: str, location: str
268
+ ) -> Tuple[str, str, List[List[Any]], Dict[str, Any], Optional[str]]:
269
+ tool = _tool_or_none(token)
270
+ if tool is None:
271
+ return NO_TOKEN_MD, "", [], {}, None
272
+ if not query or not query.strip():
273
+ return "Type a search query to begin.", "", [], {}, None
274
+
275
+ data = tool.search(query, country, int(pages or 1), device, time_period or None, location)
276
+ if data.get("status") != "success":
277
+ return _error_md(data), "", [], data, None
278
+
279
+ rows = _organic_rows(data)
280
+ counts = [
281
+ f"**{len(rows)}** organic results",
282
+ f"**{len(data.get('top_ads') or []) + len(data.get('bottom_ads') or [])}** ads",
283
+ f"**{len(data.get('related_questions') or [])}** People Also Ask",
284
+ f"**{len(data.get('related_searches') or [])}** related searches",
285
+ ]
286
+ ai = data.get("ai_overview")
287
+ counts.append("AI Overview **yes**" if isinstance(ai, dict) and ai.get("state") == "complete" else "AI Overview **no**")
288
+ summary = (
289
+ " · ".join(counts)
290
+ + f"\n\n{data.get('pages_fetched')} page(s) fetched · {data.get('credits_used')} credits used"
291
+ )
292
+ if data.get("warnings"):
293
+ summary += "\n\n" + "\n".join(f"- {w}" for w in data["warnings"])
294
+
295
+ table = [[r["position"], r["domain"], (r["title"] or "")[:90], r["link"]] for r in rows]
296
+ csv_path = _write_csv(rows, "google-serp")
297
+ return summary, render_serp(data), table, data, csv_path
298
+
299
+
300
+ def run_rank_check(
301
+ token: str, query: str, target_domain: str, country: str, pages: int,
302
+ device: str, time_period: str, location: str,
303
+ ) -> Tuple[str, str, Dict[str, Any]]:
304
+ tool = _tool_or_none(token)
305
+ if tool is None:
306
+ return NO_TOKEN_MD, "", {}
307
+ if not query or not query.strip():
308
+ return "Type the keyword you want to check.", "", {}
309
+ domain = _domain(target_domain if "://" in (target_domain or "") else f"https://{(target_domain or '').strip()}")
310
+ if not domain:
311
+ return "Enter the domain to look for, e.g. `scrape.do`.", "", {}
312
+
313
+ data = tool.search(query, country, int(pages or 1), device, time_period or None, location)
314
+ if data.get("status") != "success":
315
+ return _error_md(data), "", data
316
+
317
+ organic = data.get("organic_results") or []
318
+ hits = [
319
+ r for r in organic
320
+ if _domain(r.get("link")) == domain or _domain(r.get("link")).endswith("." + domain)
321
+ ]
322
+ looked = len(organic)
323
+ country_label = dict((gl, label) for label, gl in COUNTRY_CHOICES).get(country, country)
324
+
325
+ if hits:
326
+ best = hits[0]
327
+ lines = [
328
+ f"# #{best.get('overall_position')}",
329
+ f"**{domain}** ranks **#{best.get('overall_position')}** for “{query.strip()}” "
330
+ f"in {country_label} ({device}), page {best.get('page')}.",
331
+ ]
332
+ if len(hits) > 1:
333
+ others = ", ".join(f"#{h.get('overall_position')}" for h in hits[1:])
334
+ lines.append(f"It also appears at {others}.")
335
+ above = [r for r in organic if r.get("overall_position", 0) < best.get("overall_position", 0)]
336
+ if above:
337
+ lines += ["", "**Ranking above you:**"]
338
+ lines += [f"{r.get('overall_position')}. {_domain(r.get('link'))}" for r in above[:10]]
339
+ else:
340
+ lines = [
341
+ "# Not found",
342
+ f"**{domain}** is not in the top {looked} organic results for “{query.strip()}” "
343
+ f"in {country_label}. Fetch more pages to look deeper.",
344
+ ]
345
+ lines.append(f"\n{data.get('credits_used')} credits used.")
346
+ return "\n".join(lines), render_serp(data, highlight_domain=domain), data
347
+
348
+
349
+ # ----------------------------------------------------------------------- UI
350
+
351
+
352
+ STYLE = """
353
+ <style>
354
+ @import url('https://fonts.googleapis.com/css2?family=Roboto:wght@400;500;700&display=swap');
355
+
356
+ .gradio-container, .gradio-container button, .gradio-container input,
357
+ .gradio-container textarea, .gradio-container select {
358
+ font-family: 'Roboto', Arial, sans-serif;
359
+ }
360
+ .gradio-container { color: #202124; }
361
+
362
+ .gs-hero {
363
+ background: #ffffff; border: 1px solid #dadce0; border-radius: 16px;
364
+ padding: 28px 32px 24px 32px; margin-bottom: 16px;
365
+ }
366
+ .gs-hero h1 { color: #202124 !important; font-size: 2rem !important; font-weight: 500 !important;
367
+ margin: 0 0 10px 0 !important; letter-spacing: -0.01em; }
368
+ .gs-dots { display: flex; gap: 6px; margin-bottom: 14px; }
369
+ .gs-dots span { width: 12px; height: 12px; border-radius: 50%; display: inline-block; }
370
+ .gs-hero p { color: #4d5156 !important; margin: 0 0 8px 0 !important; font-size: 1rem !important; line-height: 1.55 !important; }
371
+ .gs-hero a { color: #1a0dab !important; }
372
+ .gs-cta {
373
+ display: inline-block; margin-top: 10px; padding: 11px 24px;
374
+ background: #1a73e8 !important; color: #ffffff !important;
375
+ border-radius: 999px; font-weight: 500 !important; text-decoration: none !important;
376
+ }
377
+ .gs-note { border-left: 3px solid #fbbc05; background: #fef7e0; padding: 12px 16px;
378
+ border-radius: 0 8px 8px 0; font-size: 0.9rem; margin-bottom: 12px; color: #202124; }
379
+
380
+ .gradio-container .primary, .gradio-container button.primary,
381
+ .gradio-container button[variant="primary"] {
382
+ background: #1a73e8 !important; background-image: none !important;
383
+ border-color: #1a73e8 !important; color: #ffffff !important;
384
+ border-radius: 999px !important; font-weight: 500 !important;
385
+ }
386
+ .gradio-container .primary:hover, .gradio-container button.primary:hover,
387
+ .gradio-container button[variant="primary"]:hover { background: #1765cc !important; }
388
+ .gradio-container .tab-nav button.selected { color: #1a73e8 !important; border-bottom-color: #1a73e8 !important; }
389
+
390
+ /* results page preview */
391
+ .gs-serp { background: #ffffff; border: 1px solid #dadce0; border-radius: 12px;
392
+ padding: 20px 24px; max-width: 760px; color: #202124; }
393
+ .gs-stats { color: #70757a; font-size: 0.85rem; margin-bottom: 14px; }
394
+ .gs-result { margin: 0 0 24px 0; padding: 2px 0; }
395
+ .gs-hit { background: #e8f0fe; border-radius: 8px; padding: 10px 12px; margin-left: -12px; }
396
+ .gs-cite { font-size: 0.85rem; color: #202124; display: flex; gap: 8px; align-items: baseline; flex-wrap: wrap; }
397
+ .gs-source { font-weight: 500; }
398
+ .gs-url { color: #4d5156; overflow-wrap: anywhere; }
399
+ .gs-pos { background: #f1f3f4; color: #5f6368; border-radius: 999px; padding: 1px 8px; font-size: 0.75rem; }
400
+ .gs-sponsored { font-weight: 700; }
401
+ .gs-title { display: block; color: #1a0dab !important; font-size: 1.25rem; line-height: 1.3;
402
+ margin: 4px 0 3px 0; text-decoration: none !important; }
403
+ .gs-title:hover { text-decoration: underline !important; }
404
+ .gs-snippet { color: #4d5156; font-size: 0.9rem; line-height: 1.58; }
405
+ .gs-date { color: #70757a; }
406
+ .gs-sitelinks { margin-top: 6px; font-size: 0.85rem; }
407
+ .gs-sitelinks a { color: #1a0dab !important; }
408
+ .gs-ai { border: 1px solid #dadce0; border-radius: 12px; padding: 16px 18px; margin-bottom: 24px;
409
+ background: linear-gradient(180deg, #f8faff 0%, #ffffff 100%); }
410
+ .gs-ai-label { font-weight: 500; color: #1a73e8; margin-bottom: 8px; }
411
+ .gs-ai p, .gs-ai li { color: #202124; font-size: 0.92rem; line-height: 1.6; }
412
+ .gs-ai table { border-collapse: collapse; margin: 8px 0; font-size: 0.85rem; }
413
+ .gs-ai th, .gs-ai td { border: 1px solid #dadce0; padding: 4px 8px; }
414
+ .gs-ai-refs { margin-top: 10px; font-size: 0.82rem; }
415
+ .gs-ai-refs a { color: #1a0dab !important; }
416
+ .gs-ai-refs span { color: #70757a; }
417
+ .gs-box { border-top: 1px solid #ebebeb; border-bottom: 1px solid #ebebeb; padding: 12px 0; margin-bottom: 24px; }
418
+ .gs-box-title { font-size: 1.1rem; margin-bottom: 8px; }
419
+ .gs-box ul { margin: 0; padding-left: 18px; }
420
+ .gs-paa { list-style: none; padding-left: 0 !important; }
421
+ .gs-paa li { border-bottom: 1px solid #ebebeb; padding: 10px 0; }
422
+ .gs-chips { display: flex; flex-wrap: wrap; gap: 8px; }
423
+ .gs-chip { background: #f1f3f4; border-radius: 999px; padding: 6px 12px; font-size: 0.88rem; }
424
+ .gs-kg { border: 1px solid #dadce0; border-radius: 12px; padding: 14px 16px; margin-bottom: 24px; }
425
+ .gs-kg-title { font-size: 1.4rem; }
426
+ .gs-kg-type { color: #70757a; margin-bottom: 6px; }
427
+ footer { display: none !important; }
428
+ </style>
429
+ """
430
+
431
+ HERO = f"""
432
+ <div class="gs-hero">
433
+ <div class="gs-dots"><span style="background:#4285F4"></span><span style="background:#EA4335"></span><span style="background:#FBBC05"></span><span style="background:#34A853"></span></div>
434
+ <h1>Google Search Scraper</h1>
435
+ <p><strong>Any Google results page as structured data</strong> — organic rankings, ads, People Also Ask,
436
+ related searches, the AI Overview, knowledge panel and local pack, from any country and in any language.</p>
437
+ <p>No selectors to maintain: Google changes its page code constantly, so this uses the
438
+ <a href="https://scrape.do/documentation/google-scraper-api/search/" target="_blank" rel="noopener noreferrer">Scrape.do Google Search API</a>,
439
+ which returns the same clean fields whatever Google ships. Check where a site ranks, watch competitors' ads,
440
+ or feed live results to an AI agent.</p>
441
+ <a class="gs-cta" href="{SCRAPE_DO_SIGNUP_URL}" target="_blank" rel="noopener noreferrer">Get a free API token →</a>
442
+ </div>
443
+ """
444
+
445
+ CREDIT_NOTE = """
446
+ <div class="gs-note">
447
+ Each results page costs 10 credits, so the free 1,000 credits cover about 100 searches.
448
+ If Google is still writing its AI Overview when the page comes back, it's fetched with one
449
+ follow-up call for 5 more credits.
450
+ </div>
451
+ """
452
+
453
+ FOOTER_MD = f"""
454
+ ---
455
+
456
+ ### Use it as a smolagents Tool
457
+
458
+ ```python
459
+ from smolagents import CodeAgent, InferenceClientModel
460
+ from tool import ScrapeDoGoogleSearchTool
461
+
462
+ agent = CodeAgent(
463
+ tools=[ScrapeDoGoogleSearchTool(api_token="YOUR_SCRAPEDO_TOKEN")],
464
+ model=InferenceClientModel(),
465
+ )
466
+ agent.run("Who ranks in the top 5 for 'best web scraping api' in Germany, and do any run ads?")
467
+ ```
468
+
469
+ Or set `SCRAPE_DO_API_TOKEN` in the environment and instantiate with no arguments.
470
+
471
+ Want to parse the raw HTML yourself instead? The
472
+ [Google search scraping guide](https://scrape.do/blog/scraping-google-search-results/) covers both approaches.
473
+
474
+ [Scrape.do](https://scrape.do/) · [Google Search API docs](https://scrape.do/documentation/google-scraper-api/search/) · [Sign up free]({SCRAPE_DO_SIGNUP_URL})
475
+ """
476
+
477
+
478
+ def build_demo() -> gr.Blocks:
479
+ with gr.Blocks(title="Google Search Scraper — Scrape.do") as demo:
480
+ gr.HTML(STYLE)
481
+ gr.HTML(HERO)
482
+
483
+ api_token = gr.Textbox(
484
+ label="Scrape.do API token",
485
+ placeholder="Paste your token here",
486
+ type="password",
487
+ info="Free tier: 1,000 credits a month, about 100 searches.",
488
+ )
489
+
490
+ with gr.Row():
491
+ country = gr.Dropdown(label="Search from", choices=COUNTRY_CHOICES, value="us")
492
+ device = gr.Radio(label="Device", choices=["desktop", "mobile"], value="desktop")
493
+ pages = gr.Slider(label="Result pages", minimum=1, maximum=5, step=1, value=1,
494
+ info="10 credits per page.")
495
+
496
+ with gr.Accordion("More filters", open=False):
497
+ with gr.Row():
498
+ time_period = gr.Dropdown(label="Time", choices=TIME_CHOICES, value="")
499
+ location = gr.Textbox(
500
+ label="City (optional)",
501
+ placeholder="Austin,Texas,United States",
502
+ info="Google's own place name. Must be in the country above.",
503
+ )
504
+
505
+ gr.HTML(CREDIT_NOTE)
506
+
507
+ with gr.Tabs():
508
+ with gr.Tab("Search results"):
509
+ query = gr.Textbox(label="Search Google", placeholder="best web scraping api")
510
+ search_btn = gr.Button("Google Search", variant="primary")
511
+ search_summary = gr.Markdown()
512
+ serp_view = gr.HTML()
513
+ serp_table = gr.Dataframe(headers=TABLE_HEADERS, wrap=True, label="Organic results")
514
+ serp_csv = gr.File(label="Download CSV")
515
+ with gr.Accordion("Raw JSON response", open=False):
516
+ serp_json = gr.JSON()
517
+
518
+ search_btn.click(
519
+ fn=run_search,
520
+ inputs=[api_token, query, country, pages, device, time_period, location],
521
+ outputs=[search_summary, serp_view, serp_table, serp_json, serp_csv],
522
+ )
523
+ query.submit(
524
+ fn=run_search,
525
+ inputs=[api_token, query, country, pages, device, time_period, location],
526
+ outputs=[search_summary, serp_view, serp_table, serp_json, serp_csv],
527
+ )
528
+
529
+ with gr.Tab("Rank checker"):
530
+ gr.Markdown(
531
+ "Where does a website rank for a keyword? Enter both, and the matching "
532
+ "result is highlighted on the page with everyone ranking above it."
533
+ )
534
+ with gr.Row():
535
+ rank_query = gr.Textbox(label="Keyword", placeholder="web scraping api", scale=3)
536
+ rank_domain = gr.Textbox(label="Your domain", placeholder="scrape.do", scale=2)
537
+ rank_btn = gr.Button("Check ranking", variant="primary")
538
+ rank_summary = gr.Markdown()
539
+ rank_view = gr.HTML()
540
+ with gr.Accordion("Raw JSON response", open=False):
541
+ rank_json = gr.JSON()
542
+
543
+ rank_btn.click(
544
+ fn=run_rank_check,
545
+ inputs=[api_token, rank_query, rank_domain, country, pages, device, time_period, location],
546
+ outputs=[rank_summary, rank_view, rank_json],
547
+ )
548
+
549
+ gr.Markdown(FOOTER_MD)
550
+
551
+ return demo
552
+
553
+
554
+ # Hugging Face Spaces launches the module-level `demo` object rather than
555
+ # calling launch() itself, so build it at import time.
556
+ demo = build_demo()
557
+
558
+ if __name__ == "__main__":
559
+ demo.launch()
requirements.txt ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ gradio
2
+ requests
3
+ smolagents
4
+ spaces
tool.py ADDED
@@ -0,0 +1,248 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from __future__ import annotations
2
+
3
+ import json
4
+ import os
5
+ import urllib.parse
6
+ from typing import Any, Dict, List, Optional
7
+
8
+ import requests
9
+ from smolagents.tools import Tool
10
+
11
+
12
+ SCRAPE_DO_SIGNUP_URL = (
13
+ "https://dashboard.scrape.do/sign-up?utm_source=huggingface&utm_medium=google-search-scraper"
14
+ )
15
+ SEARCH_ENDPOINT = "https://api.scrape.do/plugin/google/search"
16
+ AI_OVERVIEW_ENDPOINT = "https://api.scrape.do/plugin/google/search/ai-overview"
17
+
18
+ # Credit costs from the Scrape.do docs.
19
+ CREDITS_PER_SEARCH = 10
20
+ CREDITS_PER_AI_OVERVIEW = 5
21
+
22
+ # (label, gl, hl, google_domain)
23
+ COUNTRIES: List[tuple] = [
24
+ ("United States", "us", "en", "google.com"),
25
+ ("United Kingdom", "gb", "en", "google.co.uk"),
26
+ ("Germany", "de", "de", "google.de"),
27
+ ("France", "fr", "fr", "google.fr"),
28
+ ("Spain", "es", "es", "google.es"),
29
+ ("Italy", "it", "it", "google.it"),
30
+ ("Netherlands", "nl", "nl", "google.nl"),
31
+ ("Turkey", "tr", "tr", "google.com.tr"),
32
+ ("Canada", "ca", "en", "google.ca"),
33
+ ("Australia", "au", "en", "google.com.au"),
34
+ ("India", "in", "en", "google.co.in"),
35
+ ("Japan", "jp", "ja", "google.co.jp"),
36
+ ("Brazil", "br", "pt", "google.com.br"),
37
+ ("Mexico", "mx", "es", "google.com.mx"),
38
+ ]
39
+ COUNTRY_BY_GL = {gl: (label, gl, hl, domain) for label, gl, hl, domain in COUNTRIES}
40
+
41
+ TIME_PERIODS = ["last_hour", "last_day", "last_week", "last_month", "last_year"]
42
+
43
+
44
+ class ScrapeDoGoogleSearchTool(Tool):
45
+ name = "google_search_scraper"
46
+ description = """
47
+ Run a Google search and get the whole results page back as structured JSON:
48
+ organic results with positions, top and bottom ads, People Also Ask
49
+ questions, related searches, the AI Overview, knowledge panel, local map
50
+ pack, top stories, videos and forum discussions. Supports any country and
51
+ language, desktop or mobile layouts, time filters and pagination.
52
+ Powered by Scrape.do's Google Search API, which handles proxies, CAPTCHAs
53
+ and Google's changing HTML. Use this for rank tracking, SEO research,
54
+ ad monitoring, or grounding an answer in live search results.
55
+ """
56
+ output_type = "string"
57
+
58
+ def __init__(self, api_token: Optional[str] = None) -> None:
59
+ self._api_token = api_token
60
+ self.inputs = {
61
+ "query": {
62
+ "type": "string",
63
+ "description": "The search query, exactly as you would type it into Google.",
64
+ },
65
+ "country": {
66
+ "type": "string",
67
+ "description": (
68
+ "Two-letter country code to search from, e.g. 'us', 'gb', 'de', 'tr'. "
69
+ "Sets the ranking country, interface language and Google domain. Defaults to 'us'."
70
+ ),
71
+ "nullable": True,
72
+ },
73
+ "pages": {
74
+ "type": "integer",
75
+ "description": "How many result pages to fetch (1-5). Each page costs 10 credits. Defaults to 1.",
76
+ "nullable": True,
77
+ },
78
+ "device": {
79
+ "type": "string",
80
+ "description": "'desktop' or 'mobile' results layout. Defaults to 'desktop'.",
81
+ "nullable": True,
82
+ },
83
+ "time_period": {
84
+ "type": "string",
85
+ "description": (
86
+ "Only return results from a recent window: last_hour, last_day, "
87
+ "last_week, last_month or last_year."
88
+ ),
89
+ "nullable": True,
90
+ },
91
+ "location": {
92
+ "type": "string",
93
+ "description": (
94
+ "City-level location in Google's format, e.g. 'Austin,Texas,United States' "
95
+ "or 'Istanbul,Istanbul,Turkey'. Must match the country code."
96
+ ),
97
+ "nullable": True,
98
+ },
99
+ }
100
+ super().__init__()
101
+
102
+ # ------------------------------------------------------------------ token
103
+
104
+ def _resolve_token(self) -> Optional[str]:
105
+ if self._api_token:
106
+ return self._api_token.strip()
107
+ env_token = os.getenv("SCRAPE_DO_API_TOKEN")
108
+ return env_token.strip() if env_token else None
109
+
110
+ # ------------------------------------------------------------------ calls
111
+
112
+ def _get(self, url: str, params: Dict[str, Any]) -> Dict[str, Any]:
113
+ full_url = url + "?" + urllib.parse.urlencode(params, quote_via=urllib.parse.quote_plus)
114
+ try:
115
+ response = requests.get(full_url, timeout=120)
116
+ except requests.exceptions.RequestException as exc:
117
+ return {"_error": f"Request failed: {exc}"}
118
+
119
+ try:
120
+ data = response.json()
121
+ except ValueError:
122
+ return {"_error": f"Unexpected non-JSON response (HTTP {response.status_code})."}
123
+
124
+ if response.status_code == 401:
125
+ return {"_error": "Scrape.do rejected the token (HTTP 401).", "_auth": True}
126
+ if not response.ok:
127
+ message = data.get("message") or data.get("error") or f"HTTP {response.status_code}"
128
+ return {"_error": f"Scrape.do returned an error: {message}", "_status": response.status_code}
129
+ return data
130
+
131
+ def search(
132
+ self,
133
+ query: str,
134
+ country: str = "us",
135
+ pages: int = 1,
136
+ device: str = "desktop",
137
+ time_period: Optional[str] = None,
138
+ location: Optional[str] = None,
139
+ fetch_deferred_ai: bool = True,
140
+ ) -> Dict[str, Any]:
141
+ token = self._resolve_token()
142
+ if not token:
143
+ return {
144
+ "status": "error",
145
+ "errorMessage": "Missing Scrape.do API token.",
146
+ "hint": f"Get a free token at {SCRAPE_DO_SIGNUP_URL}",
147
+ }
148
+ if not query or not query.strip():
149
+ return {"status": "error", "errorMessage": "Enter a search query."}
150
+
151
+ gl = (country or "us").strip().lower()
152
+ _, gl, hl, domain = COUNTRY_BY_GL.get(gl, ("", gl, "en", "google.com"))
153
+ pages = max(1, min(int(pages or 1), 5))
154
+
155
+ base: Dict[str, Any] = {
156
+ "token": token,
157
+ "q": query.strip(),
158
+ "gl": gl,
159
+ "hl": hl,
160
+ "google_domain": domain,
161
+ "device": device if device in ("desktop", "mobile") else "desktop",
162
+ "resolveGoto": "true",
163
+ }
164
+ if time_period in TIME_PERIODS:
165
+ base["time_period"] = time_period
166
+ if location and location.strip():
167
+ base["location"] = location.strip()
168
+
169
+ combined: Dict[str, Any] = {}
170
+ organic: List[Dict[str, Any]] = []
171
+ credits = 0
172
+ pages_fetched = 0
173
+ warnings: List[str] = []
174
+
175
+ for page_index in range(pages):
176
+ # The docs say to send `start` from the very first request of a
177
+ # multi-page sweep so all pages come from one coherent session.
178
+ params = dict(base, start=page_index * 10)
179
+ data = self._get(SEARCH_ENDPOINT, params)
180
+
181
+ if "_error" in data:
182
+ if page_index == 0:
183
+ result = {"status": "error", "errorMessage": data["_error"]}
184
+ if data.get("_auth"):
185
+ result["hint"] = f"Check your token or create one at {SCRAPE_DO_SIGNUP_URL}"
186
+ return result
187
+ warnings.append(f"Page {page_index + 1}: {data['_error']}")
188
+ break
189
+
190
+ credits += CREDITS_PER_SEARCH
191
+ pages_fetched += 1
192
+
193
+ for item in data.get("organic_results") or []:
194
+ item = dict(item)
195
+ item["page"] = page_index + 1
196
+ item["overall_position"] = len(organic) + 1
197
+ organic.append(item)
198
+
199
+ if page_index == 0:
200
+ combined = data
201
+ # The AI Overview can come back "deferred". Its session key
202
+ # expires after 90 seconds, so fetch it before any more pages.
203
+ ai = data.get("ai_overview")
204
+ if (
205
+ fetch_deferred_ai
206
+ and isinstance(ai, dict)
207
+ and ai.get("state") == "deferred"
208
+ and ai.get("session_key")
209
+ ):
210
+ follow = self._get(
211
+ AI_OVERVIEW_ENDPOINT, {"token": token, "session_key": ai["session_key"]}
212
+ )
213
+ if "_error" in follow:
214
+ warnings.append(f"AI Overview follow-up: {follow['_error']}")
215
+ else:
216
+ credits += CREDITS_PER_AI_OVERVIEW
217
+ combined["ai_overview"] = follow
218
+ if not (data.get("pagination") or {}).get("next"):
219
+ break
220
+
221
+ combined["organic_results"] = organic
222
+ combined["status"] = "success"
223
+ combined["pages_fetched"] = pages_fetched
224
+ combined["credits_used"] = credits
225
+ if warnings:
226
+ combined["warnings"] = warnings
227
+ return combined
228
+
229
+ # ---------------------------------------------------------------- forward
230
+
231
+ def forward(
232
+ self,
233
+ query: str,
234
+ country: Optional[str] = None,
235
+ pages: Optional[int] = None,
236
+ device: Optional[str] = None,
237
+ time_period: Optional[str] = None,
238
+ location: Optional[str] = None,
239
+ ) -> str:
240
+ result = self.search(
241
+ query=query,
242
+ country=country or "us",
243
+ pages=pages or 1,
244
+ device=device or "desktop",
245
+ time_period=time_period,
246
+ location=location,
247
+ )
248
+ return json.dumps(result, ensure_ascii=False, indent=2)