Deploy bbcollect 8fcea0a244395b5057e019e22276481549cb1a4b

README.md

@@ -10,86 +10,109 @@ app_file: app.py
 pinned: false
 ---
 
-# bbcollect / bbconnectapi
+# bbcollect · bbconnectapi
 
-A Gradio UI and JSON API for public bug bounty program listings and scope.
+Collect public bug bounty programs, published reward ranges, scopes, exclusions, and program rules from **HackerOne, Bugcrowd, Intigriti, and YesWeHack**.
 
-- Space: https://huggingface.co/spaces/wuhp/bbconnectapi
-- Source: https://github.com/Wuhpondiscord/bbcollect
-- HackerOne: live public GraphQL requests initialized from https://hackerone.com/opportunities/all, with explicit dataset fallback.
-- Bugcrowd, Intigriti, YesWeHack directories: public snapshots from [arkadiyt/bounty-targets-data](https://github.com/arkadiyt/bounty-targets-data).
-- All four platforms: live program details, reward ranges/tables, scopes, exclusions and full program rules.
+**No API key. No account. No Authorization header.** The hosted API is public and supports browser requests with CORS.
 
-This collects the public open-program directory, not every promotion or feature of the Opportunities page. Private programs and authenticated account data are not collected. No platform credentials are required.
+- [Interactive API documentation](https://wuhp-bbconnectapi.hf.space/api/docs) — try requests in your browser
+- [API reference](https://github.com/Wuhpondiscord/bbcollect/blob/main/docs/API.md) — endpoints, parameters, responses, errors, limits
+- [Usage recipes](https://github.com/Wuhpondiscord/bbcollect/blob/main/docs/EXAMPLES.md) — Python, JavaScript, curl, pagination, and exports
+- [Gradio app](https://huggingface.co/spaces/wuhp/bbconnectapi) · [OpenAPI JSON](https://wuhp-bbconnectapi.hf.space/api/openapi.json)
 
-## API usage
+## Quick start
 
-Install `gradio_client`, then:
+Base URL: **`https://wuhp-bbconnectapi.hf.space/api/v1`**
 
-```python
-from gradio_client import Client
-client = Client("wuhp/bbconnectapi")
-page = client.predict(
-    platforms=["hackerone", "bugcrowd", "intigriti", "yeswehack"],
-    search="", bounty_only=False, limit=10, offset=0,
-    api_name="/collect",
-)
-print(page["programs"])
-# For the next page use page["next_offset"] as offset, unless it is None.
-scope = client.predict(platform="hackerone", handle="security", api_name="/scope")
-print(scope["targets"]["in_scope"])
-print(scope["targets"]["out_of_scope"])
-print(client.predict(api_name="/sources"))
-details = client.predict(platform="bugcrowd", handle="CCData-mbb-og", api_name="/details")
-print(details["min_bounty"], details["max_bounty"], details["rules"])
+```bash
+curl 'https://wuhp-bbconnectapi.hf.space/api/v1/programs?platforms=hackerone,bugcrowd&limit=10'
 ```
 
-Raw HTTP uses Gradio's two-step event API, not REST GET routes:
-
-```bash
-curl -X POST https://wuhp-bbconnectapi.hf.space/gradio_api/call/collect \
-  -H 'Content-Type: application/json' \
-  -d '{"data":[["hackerone"],"",false,10,0]}'
-# Use the returned event_id:
-curl -N https://wuhp-bbconnectapi.hf.space/gradio_api/call/collect/EVENT_ID
+```python
+import requests
+
+base = "https://wuhp-bbconnectapi.hf.space/api/v1"
+response = requests.get(f"{base}/programs", params={
+    "platforms": "hackerone,bugcrowd",
+    "search": "", "bounty_only": True, "limit": 10,
+}, timeout=180)
+response.raise_for_status()
+page = response.json()
+for program in page["programs"]:
+    print(program["name"], program["max_bounty"], program["currency"])
+
+# One program's complete live document: rewards, scopes and policies.
+response = requests.get(f"{base}/programs/hackerone/security", timeout=180)
+response.raise_for_status()
+detail = response.json()
+print(detail["targets"]["in_scope"])
+print(detail["targets"]["out_of_scope"])
+print(detail["rules"])
 ```
 
-`/collect` accepts exactly five inputs in this order: `platforms`, `search`, `bounty_only`, `limit`, `offset`. It returns `ok`, `partial`, `total` (after filtering), `next_offset`, `sources`, and `programs`. It always includes live details: each program has `targets`, `rules`, `reward_table`, `field_status`, and `details_source`.
+Install the Python example dependency with `pip install requests`. REST responses are ordinary JSON; no event IDs or streaming client are required.
 
-Use `/collect_options` for the same five inputs plus a sixth `include_details` boolean. Set it to `False` for fast summary pages. Clients using the six-value payload from the previous version should change their endpoint to `/collect_options`; original five-value `/collect` clients require no changes. Python default arguments do not make Gradio component inputs optional in raw HTTP requests.
+## What can I do?
 
-Full-detail pages support 1–100 programs; summary pages support 1–1000. `offset` is 0–100000. Four workers fetch details with a shared 15-minute cache, so large first requests take longer.
+| Endpoint | Purpose |
+| --- | --- |
+| `GET /programs` | Search multiple platforms, filter by bounty/currency, sort, and paginate |
+| `GET /programs/{platform}/{handle}` | Live bounty tables, scopes, exclusions, and full program rules |
+| `GET /programs/{platform}/{handle}/assets` | Common asset identifiers/types, with native records and all exclusions retained |
+| `POST /programs/batch` | Up to 20 detail lookups in one request |
+| `GET /stats` | Directory counts by platform without crawling individual policies |
+| `GET /sources` | Supported sources, capabilities, and cache settings |
+| `GET /health` | Cheap application readiness check |
 
-Failures are reported per source and program. A successful empty search differs from an upstream failure. Sorting is by platform, name, then ID. Search matches name, handle, URL or website. The bounty filter excludes unknown values.
+Paths above are relative to `/api/v1`. **REST listings return summaries by default.** Add `include_details=true` to include live scopes and rules for every program on the page. Start with `limit=5` or `10`: full policies and large scopes can take much longer than summaries.
 
-Program fields: `id`, `platform`, `handle`, `name`, `url`, `website`, `offers_bounties`, `min_bounty`, `max_bounty`, `currency`, `submission_state`. Bounty amounts are numeric with a separate currency when available. Unknown values are null; an unpublished minimum is never inferred from the maximum. Reward tables preserve severity/asset-tier ranges. Bugcrowd ranges use all published in-scope severity tiers; HackerOne ranges use its published bounty table. These are advertised rewards, not historical payouts or guarantees.
+## Understand the data
 
-Use the exact platform and handle returned by `/collect` for `/details` or `/scope` (an alias returning the same full details). Intigriti uses its program UUID because URL slugs can repeat across companies. `targets` contains `in_scope` and `out_of_scope` arrays with native fields, group instructions, and exclusions. `rules` preserves full policy text/HTML, disclosure terms, safe-harbor information, testing requirements, FAQs, or qualifying vulnerability types where published. Intigriti uses the latest version of every section. HTML is returned as JSON strings, not executed.
+| Platform | Directory | Program details |
+| --- | --- | --- |
+| HackerOne | Public live GraphQL, with dataset fallback | Live public policy, reward table, and paginated scope |
+| Bugcrowd | Public dataset | Live public engagement brief, reward tiers, target groups, rules, safe harbor |
+| Intigriti | Public dataset | Live current versions of assets, reward tables, rules, FAQs, and testing requirements |
+| YesWeHack | Public dataset | Live public rules, reward grids, scopes, exclusions, and vulnerability requirements |
 
-If live details fail, available dataset scopes/rewards are retained, `partial` is true, and `field_status` explicitly marks missing rules. Missing policies are never represented as an empty set of permissions. Rules linked from program text remain links; the collector does not recursively crawl external documents.
+Directory snapshots and fallbacks come from [arkadiyt/bounty-targets-data](https://github.com/arkadiyt/bounty-targets-data). Only public metadata is collected; no target scanning or authenticated/private program data is involved. HackerOne covers the public open-program directory rather than every promotion on its Opportunities page.
 
-Each source reports `mode` (`live` or `public_dataset`), `source_url`, `fetched_at`, `source_updated_at` (null if unavailable), and `cached`. Retrieval time is not the time the platform last changed its data. Live pages are fetched sequentially, not atomically; pagination may shift after the cache expires.
+- Check `ok`, `partial`, `sources`, `details_source`, and `field_status`. A partial response may retain dataset scopes while live rules are unavailable.
+- `null` means unknown/unpublished/unavailable, not zero. Advertised rewards are not payout guarantees. Currency is separate; amount filters require an explicit currency.
+- Most directory reward/currency fields outside HackerOne come from snapshots. Reward filters run on directory metadata **before** live detail enrichment, so values may differ from a refreshed detail document.
+- Source results are cached in memory for 15 minutes; failures for 60 seconds. `fetched_at` is retrieval time, not the platform's last edit. `source_updated_at` may be unavailable.
+- Always check the linked program's current rules. Asset filters retain exclusions; program rules and group instructions still apply.
 
-## Deployment
+## Existing Gradio clients
 
-The GitHub Actions workflow tests pushes/PRs and deploys successful `main` builds to the existing `wuhp/bbconnectapi` Space. Manual workflow dispatch is supported. The repository Actions secret **HF_TOKEN** must have write access to that Space. The token is used only by the deployment job; the public app never receives it.
+The Gradio API remains available without a token:
 
-Only `app.py`, `collector.py`, `details.py`, `requirements.txt`, and `README.md` are uploaded. Existing unrelated Space files are retained. Tokens are never printed or placed in Git remote URLs.
+```python
+from gradio_client import Client
+client = Client("wuhp/bbconnectapi")
+page = client.predict(["hackerone"], "", False, 10, 0, api_name="/collect")
+detail = client.predict("hackerone", "security", api_name="/details")
+```
+
+`/collect` keeps exactly **five** inputs and always includes full details. `/collect_options` adds a sixth `include_details` boolean. `/scope` remains an alias for `/details`; `/sources` takes no inputs. See the [compatibility reference](https://github.com/Wuhpondiscord/bbcollect/blob/main/docs/API.md#gradio-compatibility) for raw HTTP payloads.
 
-## Local development
+## Run your own instance
 
 ```bash
+git clone https://github.com/Wuhpondiscord/bbcollect.git
+cd bbcollect
 python -m venv .venv
-# Activate the environment for your shell, then:
+# Activate .venv for your shell, then:
 pip install -r requirements.txt
-python -m unittest discover -s tests -v
 python app.py
 ```
 
-Open http://localhost:7860. Public Spaces expose their Gradio API publicly; use a private Space and an authenticated client for restricted access. Free Spaces may sleep and lose their cache. Collection runs on demand with a shared 15-minute memory cache, or 60 seconds for failures. There is no scheduled crawler or persistent database.
+Open `http://localhost:7860`, `http://localhost:7860/api/docs`, or `http://localhost:7860/api/v1/health`. `PORT` overrides port 7860. No runtime secrets are required. [Deployment guide](https://github.com/Wuhpondiscord/bbcollect/blob/main/docs/DEPLOYMENT.md).
+
+The hosted Space may sleep or restart; there is no uptime guarantee or persistent database. Busy REST collection requests receive `429` and a `Retry-After` header. Prefer summary pages and batch only the details you need. Cache expiration can shift offset pagination; deduplicate program IDs during long crawls.
 
-Check current program rules and scope at the linked platform before testing. Snapshots can lag behind changes and do not grant permission to test assets.
+## Contribute
 
-## Data attribution
+Run `python -m unittest discover -s tests -v` before submitting changes. Tests cover the REST contract, filters, normalization, partial failures, and the original five-input Gradio request. New providers should preserve scopes, exclusions, native policies, and source provenance. See [CONTRIBUTING.md](https://github.com/Wuhpondiscord/bbcollect/blob/main/CONTRIBUTING.md) and [CHANGELOG.md](https://github.com/Wuhpondiscord/bbcollect/blob/main/CHANGELOG.md).
 
-Additional feeds and the HackerOne fallback use public data maintained by [arkadiyt/bounty-targets-data](https://github.com/arkadiyt/bounty-targets-data). The upstream Ruby collector is not bundled.

api.py

@@ -0,0 +1,371 @@
+"""Public, unauthenticated REST v1 interface alongside the Gradio API."""
+from concurrent.futures import ThreadPoolExecutor
+from contextlib import contextmanager
+from enum import Enum
+import threading
+from typing import Any, Literal
+
+from fastapi import FastAPI, HTTPException, Path, Query, Response
+from fastapi.exceptions import RequestValidationError
+from fastapi.middleware.cors import CORSMiddleware
+from fastapi.responses import JSONResponse
+from pydantic import BaseModel, ConfigDict, Field
+from starlette.exceptions import HTTPException as StarletteHTTPException
+
+from collector import CACHE_SECONDS, FAILURE_SECONDS, PLATFORMS, collector, utcnow
+
+VERSION = "1.0.0"
+_work_slots = threading.BoundedSemaphore(2)
+
+
+class Platform(str, Enum):
+    hackerone = "hackerone"
+    bugcrowd = "bugcrowd"
+    intigriti = "intigriti"
+    yeswehack = "yeswehack"
+
+
+class ProgramRef(BaseModel):
+    model_config = ConfigDict(extra="forbid")
+    platform: Platform
+    handle: str = Field(min_length=1, max_length=200, pattern=r"^[A-Za-z0-9_.-]+$",
+                        description="Use the exact handle returned by the programs endpoint.")
+
+
+class BatchRequest(BaseModel):
+    model_config = ConfigDict(extra="forbid")
+    programs: list[ProgramRef] = Field(min_length=1, max_length=20)
+
+
+class Payload(BaseModel):
+    """Shared envelope; platform-native policy and reward structures remain JSON."""
+    model_config = ConfigDict(extra="allow")
+    ok: bool
+    partial: bool = False
+    api_version: str = VERSION
+
+
+class ErrorInfo(BaseModel):
+    code: str
+    message: str
+    details: list[dict[str, Any]] | None = None
+
+
+class ErrorPayload(BaseModel):
+    ok: Literal[False] = False
+    api_version: str = VERSION
+    error: ErrorInfo
+
+
+class ProgramResult(BaseModel):
+    model_config = ConfigDict(extra="allow")
+    id: str
+    platform: Platform
+    handle: str
+    name: str
+    url: str
+    offers_bounties: bool | None = None
+    min_bounty: float | None = None
+    max_bounty: float | None = None
+    currency: str | None = None
+    targets: dict[str, Any] | None = None
+    rules: dict[str, Any] | None = None
+    reward_table: Any = None
+    details_source: dict[str, Any] | None = None
+    field_status: dict[str, str] | None = None
+
+
+class ProgramPage(Payload):
+    total: int
+    limit: int
+    offset: int
+    next_offset: int | None
+    include_details: bool
+    sources: dict[str, dict[str, Any]]
+    programs: list[ProgramResult]
+    filters: dict[str, Any]
+
+
+class ProgramDetails(Payload):
+    platform: str
+    handle: str
+    targets: dict[str, Any] | None = None
+    rules: dict[str, Any] | None = None
+    reward_table: Any = None
+    min_bounty: float | None = None
+    max_bounty: float | None = None
+    currency: str | None = None
+    field_status: dict[str, str] | None = None
+    mode: str | None = None
+    source_url: str | None = None
+    fetched_at: str | None = None
+    source_updated_at: str | None = None
+    warning: str | None = None
+    cached: bool = False
+
+
+class BatchResult(Payload):
+    count: int
+    succeeded: int
+    programs: list[ProgramDetails]
+
+
+class Asset(BaseModel):
+    identifier: str
+    type: str
+    native_type: str | int | None
+    instructions: str | None
+    eligible_for_bounty: bool | None
+    raw: Any
+
+
+class AssetPage(Payload):
+    platform: str
+    handle: str
+    total: int
+    limit: int
+    offset: int
+    next_offset: int | None
+    in_scope: list[Asset]
+    out_of_scope: list[Asset]
+    rules: dict[str, Any] | None = None
+    source: dict[str, Any]
+    note: str
+
+
+ERRORS = {code: {"model": ErrorPayload, "description": label} for code, label in (
+    (400, "Invalid filter combination"), (422, "Invalid query or request body"),
+    (429, "Collection capacity busy; retry after the Retry-After header"),
+    (503, "Required upstream data is unavailable"))}
+
+
+def envelope(result):
+    return {"api_version": VERSION, **result}
+
+
+def bad_request(message):
+    raise HTTPException(400, {"code": "invalid_filters", "message": message})
+
+
+def parse_platforms(value):
+    selected = list(dict.fromkeys(p.strip().lower() for p in value.split(",") if p.strip()))
+    if not selected or any(p not in PLATFORMS for p in selected):
+        bad_request("platforms must contain comma-separated values from: " + ", ".join(PLATFORMS))
+    return selected
+
+
+@contextmanager
+def work_slot():
+    if not _work_slots.acquire(blocking=False):
+        raise HTTPException(429, {"code": "busy", "message": "Collection capacity is busy; retry in a few seconds."},
+                            headers={"Retry-After": "5"})
+    try:
+        yield
+    finally:
+        _work_slots.release()
+
+
+def require_data(result):
+    if not result.get("ok"):
+        raise HTTPException(503, {"code": "source_unavailable",
+                                 "message": "Public program data was not found or is temporarily unavailable."})
+    return envelope(result)
+
+
+def normalize_asset(asset):
+    """Common identifiers/types, retaining the complete native asset record."""
+    if isinstance(asset, str):
+        return {"identifier": asset, "type": "other", "native_type": None,
+                "instructions": None, "eligible_for_bounty": None, "raw": asset}
+    identifier = next((asset[k] for k in ("asset_identifier", "scope", "endpoint", "target", "uri", "name", "ipAddress")
+                       if asset.get(k)), "")
+    native = asset.get("asset_type", asset.get("scope_type", asset.get("category", asset.get("type", asset.get("typeId")))))
+    types = {1: "url", 2: "android", 3: "ios", 4: "ip_range", 5: "device", 6: "other", 7: "wildcard",
+             "website": "url", "web-application": "url", "url": "url", "wildcard": "wildcard",
+             "api": "api", "cidr": "ip_range", "iprange": "ip_range", "ip_range": "ip_range",
+             "ip_address": "ip", "ip": "ip", "android": "android", "ios": "ios",
+             "apple_store_app_id": "ios", "google_play_app_id": "android", "domain": "domain"}
+    key = native.lower() if isinstance(native, str) else native
+    kind = types.get(key, "other")
+    return {"identifier": str(identifier), "type": kind, "native_type": native,
+            "instructions": asset.get("instruction", asset.get("description")),
+            "eligible_for_bounty": asset.get("eligible_for_bounty"), "raw": asset}
+
+
+def create_api():
+    app = FastAPI(title="bbconnectapi", version=VERSION,
+        description="Public bug bounty program metadata, rewards, scopes and policies. "
+                    "No API key, login or Authorization header required. "
+                    "Directory summaries are fast; live details can take longer. "
+                    "Check partial/source status before using results.",
+        docs_url="/api/docs", redoc_url="/api/redoc", openapi_url="/api/openapi.json",
+        servers=[{"url": "/", "description": "Current host"}],
+        openapi_tags=[{"name": "Programs", "description": "Search public programs and retrieve details."},
+                      {"name": "Assets", "description": "Normalized assets with native records and exclusions."},
+                      {"name": "Service", "description": "Readiness, supported sources and directory statistics."}])
+    app.add_middleware(CORSMiddleware, allow_origins=["*"], allow_methods=["GET", "POST", "OPTIONS"],
+                       allow_headers=["Content-Type"], allow_credentials=False,
+                       expose_headers=["Retry-After", "X-API-Version"])
+
+    @app.middleware("http")
+    async def version_header(request, call_next):
+        response = await call_next(request)
+        if request.url.path.startswith("/api/"):
+            response.headers["X-API-Version"] = VERSION
+            response.headers["Cache-Control"] = "no-store"
+        return response
+
+    @app.exception_handler(StarletteHTTPException)
+    async def http_error(request, exc):
+        detail = exc.detail if isinstance(exc.detail, dict) else {"code": "http_error", "message": str(exc.detail)}
+        return JSONResponse(status_code=exc.status_code, headers=exc.headers,
+                            content={"ok": False, "api_version": VERSION, "error": detail})
+
+    @app.exception_handler(RequestValidationError)
+    async def validation_error(request, exc):
+        details = [{"location": list(e["loc"]), "message": e["msg"], "type": e["type"]} for e in exc.errors()]
+        return JSONResponse(status_code=422, content={"ok": False, "api_version": VERSION,
+            "error": {"code": "validation_error", "message": "Check the request parameters.", "details": details}})
+
+    @app.get("/api/v1/health", tags=["Service"], response_model=Payload,
+             summary="Check application readiness without contacting platforms")
+    def health():
+        return {"ok": True, "api_version": VERSION, "authentication_required": False,
+                "generated_at": utcnow(), "upstream_checked": False,
+                "docs_url": "/api/docs", "openapi_url": "/api/openapi.json"}
+
+    @app.get("/api/v1/sources", tags=["Service"], response_model=Payload,
+             summary="Describe data sources, capabilities and cache lifetimes")
+    def sources():
+        return envelope({"ok": True, "authentication_required": False,
+            "cache_seconds": CACHE_SECONDS, "failure_cache_seconds": FAILURE_SECONDS,
+            "platforms": [{"platform": p, "directory": "live_with_dataset_fallback" if p == "hackerone" else "public_dataset",
+                           "details": "live_with_dataset_fallback", "rules": "live_only"} for p in PLATFORMS],
+            "limits": {"full_detail_page": 100, "summary_page": 1000, "batch_programs": 20},
+            "availability": "Public programs only. Fields may be unpublished; fallbacks can be partial."})
+
+    @app.get("/api/v1/programs", tags=["Programs"], response_model=ProgramPage, responses=ERRORS,
+             summary="Search and paginate programs; summaries by default")
+    def programs(
+        platforms: str = Query("hackerone", description="Comma-separated platform names; use /sources for supported values."),
+        search: str = Query("", max_length=300, description="Case-insensitive substring in name, handle, URL or website."),
+        bounty_only: bool = False,
+        limit: int = Query(25, ge=1, le=1000), offset: int = Query(0, ge=0, le=100000),
+        include_details: bool = Query(False, description="Live rewards, targets and rules for this page; maximum limit 100."),
+        currency: str | None = Query(None, pattern=r"^[A-Za-z]{3}$", description="ISO currency code; excludes unknown currencies."),
+        reward_at_least: float | None = Query(None, ge=0, allow_inf_nan=False,
+            description="Require advertised max_bounty >= this amount. Requires currency; applied to directory metadata."),
+        sort_by: Literal["name", "max_bounty", "min_bounty"] = "name",
+        descending: bool = False,
+    ):
+        selected = parse_platforms(platforms)
+        if (reward_at_least is not None or sort_by != "name") and currency is None:
+            bad_request("Set currency when filtering or sorting by bounty amounts; currencies are never mixed.")
+        if include_details and limit > 100:
+            bad_request("include_details=true supports limit <= 100; use summary pages for larger limits.")
+        with work_slot():
+            # Fetch the complete small directory before filtering/sorting/pagination.
+            result = collector.collect(selected, search, bounty_only, 1000, 0, False)
+            require_data(result)
+            if result.get("next_offset") is not None:
+                # A platform may grow beyond one summary page.
+                items = list(result["programs"])
+                cursor = result["next_offset"]
+                while cursor is not None:
+                    extra = collector.collect(selected, search, bounty_only, 1000, cursor, False)
+                    require_data(extra)
+                    items.extend(extra["programs"])
+                    cursor = extra["next_offset"]
+                result["programs"] = items
+            rows = result["programs"]
+            if currency:
+                rows = [p for p in rows if (p.get("currency") or "").upper() == currency.upper()]
+            if reward_at_least is not None:
+                rows = [p for p in rows if isinstance(p.get("max_bounty"), (int, float)) and p["max_bounty"] >= reward_at_least]
+            if sort_by == "name":
+                rows.sort(key=lambda p: (p["name"].casefold(), p["id"]), reverse=descending)
+            else:
+                known = [p for p in rows if isinstance(p.get(sort_by), (int, float))]
+                unknown = [p for p in rows if not isinstance(p.get(sort_by), (int, float))]
+                known.sort(key=lambda p: (p[sort_by], p["id"]), reverse=descending)
+                rows = known + sorted(unknown, key=lambda p: p["id"])
+            total = len(rows)
+            page = rows[offset:offset + limit]
+            if include_details and page:
+                def enrich(p):
+                    detail = collector.details(p["platform"], p["handle"])
+                    fields = ("targets", "rules", "reward_table", "field_status", "currency", "min_bounty", "max_bounty")
+                    return {**p, **{k: detail[k] for k in fields if k in detail},
+                            "details_source": {k: v for k, v in detail.items() if k not in fields}}
+                with ThreadPoolExecutor(max_workers=4) as pool:
+                    page = list(pool.map(enrich, page))
+            partial = result.get("partial", False) or any(
+                p.get("details_source", {}).get("partial") or p.get("details_source", {}).get("ok") is False for p in page)
+        return envelope({**result, "programs": page, "total": total, "limit": limit, "offset": offset,
+            "next_offset": offset + limit if offset + limit < total else None,
+            "include_details": include_details, "partial": partial,
+            "filters": {"currency": currency.upper() if currency else None, "reward_at_least": reward_at_least,
+                        "reward_filter_source": "directory_metadata", "sort_by": sort_by, "descending": descending}})
+
+    @app.post("/api/v1/programs/batch", tags=["Programs"], response_model=BatchResult,
+              responses={**ERRORS, 503: {"model": BatchResult, "description": "All requested program lookups failed; inspect each item."}},
+              summary="Retrieve details for up to 20 programs; preserve input order")
+    def batch(body: BatchRequest, response: Response):
+        with work_slot():
+            def lookup(ref):
+                return collector.details(ref.platform.value, ref.handle)
+            with ThreadPoolExecutor(max_workers=4) as pool:
+                results = list(pool.map(lookup, body.programs))
+        successes = sum(bool(r.get("ok")) for r in results)
+        if not successes:
+            response.status_code = 503
+        return envelope({"ok": successes > 0, "partial": any(not r.get("ok") or r.get("partial") for r in results),
+                         "count": len(results), "succeeded": successes, "programs": results})
+
+    @app.get("/api/v1/programs/{platform}/{handle}", tags=["Programs"], response_model=ProgramDetails, responses=ERRORS,
+             summary="Retrieve public program rewards, rules, scopes and exclusions")
+    def program(platform: Platform, handle: str = Path(pattern=r"^[A-Za-z0-9_.-]{1,200}$")):
+        with work_slot():
+            return require_data(collector.details(platform.value, handle))
+
+    @app.get("/api/v1/programs/{platform}/{handle}/assets", tags=["Assets"], response_model=AssetPage, responses=ERRORS,
+             summary="Get normalized assets; all exclusions remain visible")
+    def assets(platform: Platform, handle: str = Path(pattern=r"^[A-Za-z0-9_.-]{1,200}$"),
+               asset_type: Literal["url", "api", "domain", "wildcard", "ip", "ip_range", "android", "ios", "device", "other"] | None = None,
+               search: str = Query("", max_length=300), limit: int = Query(100, ge=1, le=1000),
+               offset: int = Query(0, ge=0, le=100000)):
+        with work_slot():
+            detail = require_data(collector.details(platform.value, handle))
+        targets = detail.get("targets")
+        if not isinstance(targets, dict):
+            raise HTTPException(503, {"code": "scope_unavailable", "message": "No scope data is available for this program."})
+        rows = [normalize_asset(a) for a in targets.get("in_scope", [])]
+        rows = [a for a in rows if (asset_type is None or a["type"] == asset_type) and search.casefold() in a["identifier"].casefold()]
+        exclusions = [normalize_asset(a) for a in targets.get("out_of_scope", [])]
+        return envelope({"ok": True, "partial": detail.get("partial", False), "platform": platform.value,
+                         "handle": handle, "total": len(rows), "limit": limit, "offset": offset,
+                         "next_offset": offset + limit if offset + limit < len(rows) else None,
+                         "in_scope": rows[offset:offset + limit], "out_of_scope": exclusions,
+                         "rules": detail.get("rules"), "field_status": detail.get("field_status"),
+                         "source": {k: detail.get(k) for k in ("mode", "source_url", "fetched_at", "source_updated_at", "cached", "warning")},
+                         "note": "Filters/pagination apply only to in_scope. All exclusions and program rules are retained."})
+
+    @app.get("/api/v1/stats", tags=["Service"], response_model=Payload, responses=ERRORS,
+             summary="Count directory programs without fetching live detail documents")
+    def stats(platforms: str = "hackerone,bugcrowd,intigriti,yeswehack"):
+        selected = parse_platforms(platforms)
+        sources, counts = {}, {}
+        with work_slot():
+            for p in selected:
+                result = collector.programs(p)
+                rows = list({r["id"]: r for r in result.get("programs", [])}.values())
+                counts[p] = {"total": len(rows),
+                             "offers_bounties": sum(r.get("offers_bounties") is True for r in rows),
+                             "unknown_bounty_status": sum(r.get("offers_bounties") is None for r in rows)}
+                sources[p] = {k: v for k, v in result.items() if k != "programs"}
+        ok = any(s.get("ok") for s in sources.values())
+        result = {"ok": ok, "partial": any(not s.get("ok") for s in sources.values()), "generated_at": utcnow(),
+                  "total": sum(c["total"] for c in counts.values()), "platforms": counts, "sources": sources}
+        return require_data(result)
+
+    return app

app.py

@@ -5,6 +5,7 @@ os.environ.setdefault("GRADIO_ANALYTICS_ENABLED", "False")
 
 import gradio as gr
 from collector import PLATFORMS, collector
+from api import create_api
 
 
 def collect(platforms, search, bounty_only, limit, offset):
@@ -38,7 +39,9 @@ def sources():
 
 
 with gr.Blocks(title="bbconnectapi") as demo:
-    gr.Markdown("# bbconnectapi\nPublic bug bounty programs and scope, available as a JSON API. "
+    gr.Markdown("# bbconnectapi\nPublic bug bounty programs and scope. **No API key or login required.** "
+                "[Interactive REST docs](/api/docs) · [OpenAPI schema](/api/openapi.json) · "
+                "[GitHub guide](https://github.com/Wuhpondiscord/bbcollect)\n\n"
                 "HackerOne uses live public data with an explicit dataset fallback. "
                 "All four platforms have live program details: reward tables, scopes and rules. "
                 "Directory feeds use public snapshots outside HackerOne. Results are cached for 15 minutes.")
@@ -68,7 +71,28 @@ with gr.Blocks(title="bbconnectapi") as demo:
         details_button = gr.Button("Get full details via /details")
         details_button.click(details, [platform, handle], scope_json, api_name="details", concurrency_limit=2)
     with gr.Tab("API"):
-        gr.Markdown('''Use **View API** in the footer for generated client examples.
+        gr.Markdown('''### REST API — direct JSON, no key
+
+Base URL: `https://wuhp-bbconnectapi.hf.space/api/v1`
+
+```python
+import requests
+base = "https://wuhp-bbconnectapi.hf.space/api/v1"
+page = requests.get(f"{base}/programs", params={
+    "platforms": "hackerone,bugcrowd", "search": "", "limit": 10
+}, timeout=120).json()
+detail = requests.get(f"{base}/programs/hackerone/security", timeout=120).json()
+```
+
+Summaries are the REST default. Add `include_details=true` for full policies and scopes.
+Use `/programs/{platform}/{handle}/assets` for normalized assets and exclusions,
+`POST /programs/batch` for up to 20 detail lookups, and `/stats`, `/sources`, `/health` for metadata.
+
+[Try requests in the interactive docs](/api/docs) · [Full API reference on GitHub](https://github.com/Wuhpondiscord/bbcollect/blob/main/docs/API.md)
+
+### Existing Gradio API
+
+Use **View API** in the footer for generated client examples. `/collect` keeps its five-input contract.
 
 ```python
 from gradio_client import Client
@@ -86,6 +110,9 @@ Scope is a snapshot: check the linked program's current rules and exclusions bef
 
 demo.queue(max_size=32, default_concurrency_limit=2)
 
+# Register REST routes before the root Gradio mount. Existing Gradio URLs stay unchanged.
+app = gr.mount_gradio_app(create_api(), demo, path="/", show_error=False, ssr_mode=False)
+
 if __name__ == "__main__":
-    demo.launch(server_name="0.0.0.0", server_port=int(os.getenv("PORT", "7860")),
-                show_error=False)
+    import uvicorn
+    uvicorn.run(app, host="0.0.0.0", port=int(os.getenv("PORT", "7860")))

docs/API.md

@@ -0,0 +1,184 @@
+# API reference · v1
+
+Base: `https://wuhp-bbconnectapi.hf.space/api/v1`
+
+**All endpoints are public. Do not send an API key, HF token, login cookie, or Authorization header.** The deployment secret is unrelated to calling this API.
+
+[Interactive docs](https://wuhp-bbconnectapi.hf.space/api/docs) · [OpenAPI JSON](https://wuhp-bbconnectapi.hf.space/api/openapi.json) · [Examples](EXAMPLES.md)
+
+## List programs
+
+`GET /programs`
+
+| Query parameter | Default | Meaning |
+| --- | --- | --- |
+| `platforms` | `hackerone` | Comma-separated `hackerone`, `bugcrowd`, `intigriti`, `yeswehack`; case-insensitive |
+| `search` | empty | Case-insensitive substring in name, handle, program URL, or website; not policy/asset text |
+| `bounty_only` | `false` | Include only programs known to offer bounties; unknown status is excluded |
+| `include_details` | `false` | Retrieve live rewards, scopes, rules, and tables for the selected page |
+| `limit` | `25` | 1–1000 for summaries; 1–100 with full details |
+| `offset` | `0` | 0–100000; use the returned `next_offset` |
+| `currency` | unset | Three-letter currency code, case-insensitive; excludes unknown currencies |
+| `reward_at_least` | unset | Require advertised directory `max_bounty` >= amount; requires `currency` |
+| `sort_by` | `name` | `name`, `max_bounty`, or `min_bounty`; bounty sorting requires `currency` |
+| `descending` | `false` | Reverse the selected sort; unknown bounty values always come last |
+
+```bash
+curl 'https://wuhp-bbconnectapi.hf.space/api/v1/programs?platforms=hackerone&currency=USD&reward_at_least=1000&sort_by=max_bounty&descending=true&limit=10'
+```
+
+Filtering/sorting uses directory metadata before pagination and live enrichment. It does not fetch every program's reward table to decide eligibility. Missing snapshot currencies/reward values can exclude otherwise matching programs; use individual details for current values. No currency conversion is performed.
+
+Response shape (illustrative values):
+
+```json
+{
+  "ok": true,
+  "partial": false,
+  "api_version": "1.0.0",
+  "total": 200,
+  "limit": 10,
+  "offset": 0,
+  "next_offset": 10,
+  "include_details": false,
+  "programs": [{
+    "id": "hackerone:security",
+    "platform": "hackerone",
+    "handle": "security",
+    "name": "HackerOne",
+    "url": "https://hackerone.com/security",
+    "offers_bounties": true,
+    "min_bounty": 100,
+    "max_bounty": 15000,
+    "currency": "usd"
+  }],
+  "sources": {},
+  "filters": {
+    "currency": null,
+    "reward_at_least": null,
+    "reward_filter_source": "directory_metadata",
+    "sort_by": "name",
+    "descending": false
+  }
+}
+```
+
+The example is abbreviated; generated response models also allow native/source metadata. Optional detail fields are null in summary responses. `total` counts filtered directory programs, not only the returned page. `next_offset: null` means the crawl is finished. IDs include the platform; Intigriti uses its program UUID as `handle` because company URL slugs can repeat.
+
+With `include_details=true`, each program also has `targets`, `rules`, `reward_table`, `field_status`, and `details_source`. Any failed/partial detail lookup makes the page `partial: true`; inspect each program rather than discarding the whole page.
+
+## Get program details
+
+`GET /programs/{platform}/{handle}`
+
+```bash
+curl 'https://wuhp-bbconnectapi.hf.space/api/v1/programs/hackerone/security'
+```
+
+Use handles from `/programs`. Successful responses include:
+
+| Field | Description |
+| --- | --- |
+| `ok`, `partial`, `api_version` | Result status and REST contract version |
+| `platform`, `handle` | Requested program identity |
+| `min_bounty`, `max_bounty`, `currency` | Published numeric range and currency; null when unavailable |
+| `reward_table` | Native severity/asset-tier table or grid, preserving its descriptions |
+| `targets.in_scope`, `targets.out_of_scope` | Native platform records, kept separate |
+| `rules` | Native policy sections; fields vary by platform, described below |
+| `field_status` | Availability for scopes, rules, and ranges |
+| `mode` | `live` or `public_dataset` |
+| `source_url`, `fetched_at`, `source_updated_at`, `cached`, `warning` | Provenance, retrieval time, last edit if known, fallback/cache state |
+
+HackerOne rules include policy text, scope description, structured policy, disclosure/safe-harbor declarations, and exclusions. Bugcrowd includes brief HTML, targets overview, target-group instructions, safe harbor, resources, and VRT rules. Intigriti includes the latest in/out-of-scope text, FAQs, severity assessment, and rules of engagement/testing requirements. YesWeHack includes policy text/HTML, qualifying/non-qualifying vulnerabilities, account requirements, and relevant program rule grids.
+
+Policy HTML is data, not trusted executable markup. Referenced external documents remain links; the API does not recursively crawl them. Absence of a rule never implies permission to test.
+
+Fallback responses have `partial: true`: dataset scopes and known rewards may remain available, while `rules` is null and its `field_status` is `unavailable`. Live fields may be marked `not_published` rather than being invented. `0` is a published zero; `null` is not zero.
+
+A missing or unreachable program returns `503 source_unavailable` when no usable public data exists. The API cannot always distinguish an unknown program from upstream denial/outage, so it does not falsely claim a definitive program 404.
+
+## Normalized program assets
+
+`GET /programs/{platform}/{handle}/assets`
+
+Optional queries: `asset_type`, `search`, `limit` (1–1000, default 100), `offset` (default 0).
+
+Common asset types: `url`, `api`, `domain`, `wildcard`, `ip`, `ip_range`, `android`, `ios`, `device`, `other`. Unsupported native categories become `other`, with their native type retained.
+
+```bash
+curl 'https://wuhp-bbconnectapi.hf.space/api/v1/programs/hackerone/security/assets?asset_type=wildcard&limit=20'
+```
+
+Each normalized record has `identifier`, `type`, `native_type`, `instructions`, `eligible_for_bounty`, and `raw` (the complete original record). Out-of-scope prose can be a native string, retained in `raw`.
+
+Response fields: `in_scope`, `out_of_scope`, `rules`, `source`, `field_status`, `total`, `limit`, `offset`, `next_offset`, `ok`, `partial`, `api_version`, `note`.
+
+**Asset filtering and pagination apply only to `in_scope`. Every exclusion and the full program rules are retained.** `total` is the filtered in-scope count. A wildcard never overrides an exclusion or a restriction in the rules. Group instructions remain in the native `raw` record.
+
+## Batch program details
+
+`POST /programs/batch`
+
+```bash
+curl -X POST 'https://wuhp-bbconnectapi.hf.space/api/v1/programs/batch' \
+  -H 'Content-Type: application/json' \
+  -d '{"programs":[{"platform":"hackerone","handle":"security"},{"platform":"yeswehack","handle":"outscale"}]}'
+```
+
+Body: `programs`, an array of 1–20 objects containing `platform` and `handle`; unknown body fields are rejected. Results preserve input order (including duplicates), with individual statuses. Response: `ok`, `partial`, `api_version`, `count`, `succeeded`, `programs`.
+
+If any lookup succeeds, HTTP status is 200, even when other items fail. If all fail, status is 503 **with the same batch result envelope**, so you can still inspect every item. A success that uses dataset fallback can itself be partial.
+
+## Service metadata
+
+| Endpoint | Behavior |
+| --- | --- |
+| `GET /health` | Application readiness/version, `authentication_required: false`, `upstream_checked: false`; no upstream calls |
+| `GET /sources` | Per-platform capabilities, cache times, public-access status, and request limits; no upstream calls |
+| `GET /stats?platforms=hackerone,bugcrowd` | Directory totals, known bounty counts, unknown bounty status, and source states; all four platforms by default |
+
+Healthy application status does not guarantee that every upstream platform is reachable. Stats do not fetch individual live detail documents. Free hosted Spaces can sleep, making the first request slow.
+
+## Errors, capacity, and retries
+
+| HTTP status | Meaning | Action |
+| --- | --- | --- |
+| `200` | Successful response, possibly partial | Check `ok`, `partial`, and per-item/source states |
+| `400` | Invalid filter combination/platform list | Fix the query; see error message |
+| `422` | Invalid type, enum, bounds, or JSON body | Inspect `error.details` |
+| `429` | Both REST collection slots are busy | Wait for `Retry-After` seconds, then retry |
+| `503` | No usable upstream data, or every batch item failed | Retry with backoff; inspect fallback/field availability |
+
+Ordinary errors use:
+
+```json
+{"ok":false,"api_version":"1.0.0","error":{"code":"invalid_filters","message":"Set currency when filtering or sorting by bounty amounts; currencies are never mixed."}}
+```
+
+Validation errors add `error.details` entries with `location`, `message`, `type`. All-failed batch responses use the batch envelope instead. Responses include `X-API-Version`. REST responses are not HTTP-cached (`Cache-Control: no-store`), while source data is shared in an in-memory cache.
+
+REST collection capacity is two concurrent requests per application process, not a per-user key quota. Each batch/detail page uses at most four workers. Prefer summary listings, small full-detail pages, and bounded retries on 429/503. Do not run unbounded parallel crawls. There is no key to obtain or refresh.
+
+## Freshness and pagination
+
+Successful source requests are cached for 900 seconds; failed requests for 60. The cache is lost on restart/sleep. There is no persistent database, background schedule, refresh-bypass parameter, or atomic multi-platform snapshot. Source `fetched_at` and `source_updated_at` have different meanings. Dates may be ISO timestamps or native upstream HTTP dates.
+
+Offset pages can shift as directories change or caches expire. Follow `next_offset` and deduplicate by `id` for long crawls. Native rule and asset fields are intentionally preserved and can evolve with platforms; common REST envelope fields use the versioned `/api/v1` path.
+
+## Gradio compatibility
+
+The Gradio UI and event API remain at the same host/root paths.
+
+| Endpoint | Positional inputs | Behavior |
+| --- | --- | --- |
+| `/collect` | `[platforms, search, bounty_only, limit, offset]` | Exactly five values; full details |
+| `/collect_options` | `[platforms, search, bounty_only, limit, offset, include_details]` | Exactly six values; optional summary mode |
+| `/details` | `[platform, handle]` | Complete live detail document |
+| `/scope` | `[platform, handle]` | Alias of `/details` |
+| `/sources` | `[]` | Source capabilities |
+
+Gradio `/collect` defaults to full details; REST `/api/v1/programs` defaults to summaries. This deliberate difference preserves existing clients.
+
+For raw Gradio HTTP, POST `{"data":[["hackerone"],"",false,10,0]}` to `/gradio_api/call/collect`, read `event_id`, then GET `/gradio_api/call/collect/EVENT_ID` as an event stream. The `complete` event's JSON array contains the result at index 0; `error` events indicate processing failures. Heartbeats are not results. Python defaults cannot replace missing Gradio positional values.
+
+New integrations should use REST unless they specifically need Gradio queue/event behavior.

docs/DEPLOYMENT.md

@@ -0,0 +1,48 @@
+# Development and deployment
+
+## Local run
+
+Python 3.13 is used by the hosted Space and CI.
+
+```bash
+git clone https://github.com/Wuhpondiscord/bbcollect.git
+cd bbcollect
+python -m venv .venv
+```
+
+Activate `.venv` (`.venv\Scripts\Activate.ps1` in PowerShell; `source .venv/bin/activate` on macOS/Linux), then:
+
+```bash
+pip install -r requirements.txt
+python -m unittest discover -s tests -v
+python app.py
+```
+
+Gradio: `http://localhost:7860` · REST: `http://localhost:7860/api/v1` · Docs: `http://localhost:7860/api/docs`.
+
+`app.py` mounts Gradio at `/` on the FastAPI application after registering REST routes, then starts Uvicorn. Keep the REST route registration before the root mount. No platform tokens or runtime secrets are needed. Set `PORT` to change port 7860.
+
+## Hugging Face Space
+
+The `README.md` front matter configures the Gradio Space with Python 3.13, Gradio 6.31.0, and `app_file: app.py`. Keep the Space public for anonymous access. Use CPU Basic unless your own load requires more resources; the app does not run a model.
+
+GitHub Actions tests every push and pull request. Successful pushes to `main` and manual workflow dispatches upload application files to the existing `wuhp/bbconnectapi` Space. `scripts/deploy.py` explicitly selects app modules, docs, and examples; it never uploads secrets, `.git`, virtual environments, or tests, and does not mirror-delete unrelated Space files.
+
+The GitHub Actions repository secret `HF_TOKEN` needs write access to the destination Space. **This secret is only for publishing. API callers do not need it, and the running application does not read it.** For a fork, change `repo_id` in `scripts/deploy.py`, add your own deployment secret, and update the public URLs in docs/examples.
+
+## Public hosting behavior
+
+- REST and Gradio are deliberately unauthenticated. Browser CORS allows public GET/POST calls without credentials.
+- A per-process capacity gate permits two concurrent REST collection requests; busy callers get `429` with `Retry-After: 5`. Health/source descriptions stay lightweight.
+- Source cache is process-local: 900 seconds for successes, 60 for failures. Multiple processes have independent caches/capacity gates. Use one Uvicorn process on the free Space.
+- Free Spaces can sleep/restart. Long live scope requests may exceed a client's default timeout; start with small pages and allow a larger timeout. Summary discovery plus selected batch details is usually faster.
+- There is no persistent storage, scheduled refresh, force-refresh bypass, service-level agreement, or API-key subscription mechanism.
+
+## Release checklist
+
+1. Run the test suite and verify generated OpenAPI schemas.
+2. Keep the five-input Gradio `/collect` contract and six-input `/collect_options` contract intact.
+3. Test `/api/v1/health`, `/api/docs`, a summary page, a detail document, normalized assets, and a mixed batch.
+4. Confirm the Gradio UI and its event API still work after mounting.
+5. Check `ok`, `partial`, and source states rather than treating missing rules as an empty policy.
+6. Update the changelog, API reference, and examples for intentional behavior changes. Make breaking REST changes under a new versioned path.

docs/EXAMPLES.md

@@ -0,0 +1,148 @@
+# Usage recipes
+
+No API key or login is needed. All examples call the public REST API. Start with summaries, then retrieve the details you need.
+
+## Python: summary search and full details
+
+```bash
+pip install requests
+```
+
+```python
+import requests
+
+base = "https://wuhp-bbconnectapi.hf.space/api/v1"
+session = requests.Session()
+
+def get(path, **params):
+    response = session.get(base + path, params=params, timeout=180)
+    response.raise_for_status()
+    result = response.json()
+    if not result["ok"]:
+        raise RuntimeError(result)
+    return result
+
+page = get("/programs", platforms="hackerone,bugcrowd", search="", limit=10)
+print(page["total"], page["next_offset"])
+
+for program in page["programs"][:2]:
+    detail = get(f"/programs/{program['platform']}/{program['handle']}")
+    print(program["name"], detail["min_bounty"], detail["max_bounty"], detail["currency"])
+    print("partial:", detail["partial"], "fields:", detail["field_status"])
+    print(detail["targets"], detail["rules"])
+```
+
+Details may use a fallback. A successful HTTP response is not proof that every field is present; inspect `partial`/`field_status` before accessing optional values.
+
+## Crawl directory pages and export JSON/CSV
+
+Run the [complete example](../examples/export_programs.py):
+
+```bash
+python examples/export_programs.py --platforms hackerone,bugcrowd,intigriti,yeswehack --output programs.json --csv programs.csv
+```
+
+The script follows `next_offset`, deduplicates IDs, validates statuses, and retries a bounded number of 429/503 responses. It exports directory summaries, not every policy document. To point it at a local instance, add `--base-url http://localhost:7860/api/v1`.
+
+Equivalent pagination pattern:
+
+```python
+programs = {}
+offset = 0
+while True:
+    page = get("/programs", platforms="hackerone,bugcrowd", limit=100, offset=offset)
+    if page["partial"]:
+        raise RuntimeError("A directory source is unavailable; inspect page['sources']")
+    programs.update({p["id"]: p for p in page["programs"]})
+    if page["next_offset"] is None:
+        break
+    offset = page["next_offset"]
+```
+
+For large collections, use summaries to discover IDs, then batch selected detail lookups. Offset pagination is not an atomic snapshot and can shift after cache expiration.
+
+## Filter advertised rewards without mixing currencies
+
+```python
+page = get("/programs", platforms="hackerone", currency="USD",
+           reward_at_least=1000, sort_by="max_bounty", descending=True, limit=10)
+```
+
+This uses directory `max_bounty`. Unknown amounts/currencies are excluded. It does not convert EUR to USD, infer minimums, or crawl all live tables before filtering.
+
+## Get normalized assets while retaining exclusions
+
+```python
+assets = get("/programs/hackerone/security/assets", asset_type="wildcard", limit=100)
+for asset in assets["in_scope"]:
+    print(asset["identifier"], asset["type"], asset["instructions"])
+print("All exclusions:", assets["out_of_scope"])
+print("Program rules:", assets["rules"])
+```
+
+If `next_offset` is present, continue paging the included assets. All exclusions and rules remain in each response. Native target-group restrictions are in each asset's `raw` record; a normalized identifier alone is not sufficient authorization to test.
+
+## Batch detail lookup
+
+```python
+response = session.post(base + "/programs/batch", json={"programs": [
+    {"platform": "hackerone", "handle": "security"},
+    {"platform": "yeswehack", "handle": "outscale"},
+]}, timeout=180)
+# Batch status 503 still contains individual results if all lookups failed.
+if response.status_code not in (200, 503):
+    response.raise_for_status()
+batch = response.json()
+if "programs" not in batch:
+    raise RuntimeError(batch)
+for detail in batch["programs"]:
+    print(detail["platform"], detail["handle"], detail["ok"], detail.get("partial"))
+```
+
+## JavaScript / browser fetch
+
+```javascript
+const base = "https://wuhp-bbconnectapi.hf.space/api/v1";
+const query = new URLSearchParams({
+  platforms: "hackerone,bugcrowd", limit: "10", include_details: "false"
+});
+const response = await fetch(`${base}/programs?${query}`);
+const page = await response.json();
+if (!response.ok || !page.ok) throw new Error(JSON.stringify(page));
+console.log(page.programs, page.next_offset);
+```
+
+CORS is enabled for public GET/POST requests. Do not add credentials, tokens, or an Authorization header. For batch requests use `method: "POST"`, `headers: {"Content-Type": "application/json"}`, and `body: JSON.stringify({programs: [...]})`.
+
+## curl
+
+```bash
+# Is the application awake? (No upstream check.)
+curl 'https://wuhp-bbconnectapi.hf.space/api/v1/health'
+
+# Program counts and source status
+curl 'https://wuhp-bbconnectapi.hf.space/api/v1/stats'
+
+# A small page with full scopes and rules
+curl 'https://wuhp-bbconnectapi.hf.space/api/v1/programs?platforms=hackerone&include_details=true&limit=5'
+
+# Save a complete policy/scope response
+curl 'https://wuhp-bbconnectapi.hf.space/api/v1/programs/yeswehack/outscale' -o outscale.json
+```
+
+In Windows PowerShell, use `curl.exe` if `curl` is an alias in your shell. For JSON POSTs, Python or a saved request-body file avoids shell-specific quote escaping.
+
+## Existing Gradio integrations
+
+```python
+from gradio_client import Client
+client = Client("wuhp/bbconnectapi")
+
+# Original contract: exactly five positional values, always full details.
+page = client.predict(["hackerone"], "", False, 10, 0, api_name="/collect")
+
+# Six values only on collect_options; False selects summaries.
+summary = client.predict(["hackerone"], "", False, 10, 0, False, api_name="/collect_options")
+```
+
+For direct REST examples, `requests` is the only client dependency. `gradio_client` is needed only for Gradio calls.

examples/export_programs.py

@@ -0,0 +1,59 @@
+"""Export all selected directory summaries. Requires only `pip install requests`."""
+import argparse
+import csv
+import json
+from pathlib import Path
+import time
+
+import requests
+
+
+def get_json(client, url, params):
+    for attempt in range(5):
+        response = client.get(url, params=params, timeout=180)
+        if response.status_code in (429, 503) and attempt < 4:
+            retry = response.headers.get("Retry-After", "")
+            delay = min(int(retry), 60) if retry.isdigit() else min(2 ** (attempt + 1), 30)
+            time.sleep(delay)
+            continue
+        response.raise_for_status()
+        data = response.json()
+        if not data.get("ok") or data.get("partial"):
+            raise RuntimeError("Incomplete directory response; inspect source status: " + json.dumps(data.get("sources")))
+        return data
+    raise RuntimeError("Retry limit reached")
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--base-url", default="https://wuhp-bbconnectapi.hf.space/api/v1")
+    parser.add_argument("--platforms", default="hackerone,bugcrowd,intigriti,yeswehack")
+    parser.add_argument("--output", type=Path, default=Path("programs.json"))
+    parser.add_argument("--csv", type=Path, help="Optional CSV of summary columns")
+    args = parser.parse_args()
+    programs, seen_offsets = {}, set()
+    offset = 0
+    with requests.Session() as client:
+        while True:
+            if offset in seen_offsets:
+                raise RuntimeError("Repeated pagination offset")
+            seen_offsets.add(offset)
+            page = get_json(client, args.base_url.rstrip("/") + "/programs", {
+                "platforms": args.platforms, "limit": 100, "offset": offset})
+            programs.update({p["id"]: p for p in page["programs"]})
+            offset = page["next_offset"]
+            if offset is None:
+                break
+    rows = list(programs.values())
+    args.output.write_text(json.dumps({"count": len(rows), "programs": rows}, indent=2, ensure_ascii=False), encoding="utf-8")
+    if args.csv:
+        columns = ["id", "platform", "handle", "name", "url", "offers_bounties", "min_bounty", "max_bounty", "currency"]
+        with args.csv.open("w", newline="", encoding="utf-8") as stream:
+            writer = csv.DictWriter(stream, columns, extrasaction="ignore")
+            writer.writeheader()
+            writer.writerows(rows)
+    print(f"Exported {len(rows)} programs to {args.output}")
+
+
+if __name__ == "__main__":
+    main()

requirements.txt

@@ -1,2 +1,4 @@
 gradio==6.31.0
 requests==2.32.5
+fastapi==0.162.0
+uvicorn==0.54.0