> โญ **Support Open Source:** If you find OpticParse or PhishVision useful, please **give us a Star on GitHub**! It helps us maintain free edge scrapers and datasets for everyone.
---
## โก Feature & Architecture Comparison
Looking for a **Firecrawl alternative**, **Crawl4AI alternative**, or a **Zero-Day Phishing & Visual Shield** for your autonomous AI agents? Here is how OpticParse + PhishVision compares to traditional scraping engines and legacy security blacklists:
| Feature / Capability | OpticParse & PhishVision | Firecrawl | Crawl4AI | Jina Reader | Google Safe Browsing / VirusTotal |
| :--- | :---: | :---: | :---: | :---: | :---: |
| **Primary Extraction Engine** | **Multimodal AI Vision (Zero-CSS)** | HTML / Markdown | DOM Tree / XPath | Regex / Markdown | N/A (Security Only) |
| **Visual Prompt Injection Shield (`ToxicCanvas`)** | ๐ก๏ธ **Built-in (Zero-Opacity & CSS Sanitization)** | โ None (Vulnerable) | โ None (Vulnerable) | โ None | N/A |
| **0-Day Phishing & Wallet Drainer Shield** | ๐ก๏ธ **Built-in (Instant Day 0)** | โ None | โ None | โ None | โ ๏ธ Historical Blacklist (12-48h delay) |
| **Machine Discovery Standard (A2A)** | โ
**Native `agent.json` & `agent-service.json`** | โ None | โ None | โ None | โ None |
| **In-IDE Developer Free Trial** | โ
**200 Requests Out-of-the-Box** | โ Mandatory Credit Card | โ Self-hosted Setup | โ ๏ธ Limited Free Tier | โ ๏ธ Strict Rate Limits |
| **In-Terminal Instant Checkout** | โ
**ASCII QR in Console + .env Save**| โ Web Portal Only | โ None | โ Web Portal Only | โ Enterprise Contract |
| **Agent Ecosystem Toolkits** | **LangChain, ElizaOS, AgentKit, MCP** | LangChain, LlamaIndex | Generic Python | Generic HTTP | None (Raw API) |
| **Payment & Settlement Flexibility** | **USDC (Base/Polygon) / $0.01 M2M** | Stripe ($16+/mo SaaS) | None (Open Source) | Stripe ($$$) | Enterprise ($$$) |
| **Token Optimization Noise Reduction** | **96% Token Reduction** | ~80% | ~70% | ~85% | N/A |
---
## ๐ Why This Stack Exists: The Dual-Engine Agent Architecture
Giving an autonomous AI agent unrestricted internet access introduces three critical vulnerabilities:
1. **Context Window & DOM Fragility (OpticParse solves this):** Raw HTML consumes 90%+ of LLM token context, while rigid CSS/XPath selectors break the moment websites change frontend frameworks.
2. **Visual Prompt Injection & Screen Poisoning ([ToxicCanvas](https://opticparse.com/toxic-canvas.html) solves this):** Adversarial sites embed invisible CSS, zero-opacity layers, and prompt injection strings to hijack multimodal vision models (Claude Computer Use, GPT-4o, Operator). ToxicCanvas sanitizes DOM tokens before ingestion.
3. **The Autonomous Link Trap (PhishVision solves this):** When agents follow links autonomously, malicious actors exploit them with zero-day credential harvesting kits, fake Web3 dApps, and wallet drainers.
**OpticParse + PhishVision provides the complete solution:**
- **The Shield (PhishVision + ToxicCanvas):** Inspects target URLs in real-time for SSL age, brand spoofing, and visual injection attacks *before* the agent executes navigation.
- **The Engine (OpticParse):** Visually renders authenticated pages at the edge, converting dynamic JavaScript into token-optimized clean Markdown (96% noise reduction) without brittle CSS selectors.
- **Embodied AI & Edge Hardware Compatible:** Lightweight token streams structured for low-power edge processors (NVIDIA Jetson, embedded robotics telemetry) that cannot waste local battery and GPU cycles running headless browsers ([Learn more](https://opticparse.com/embodied-ai.html)).
Operating **150 autonomous extraction & threat pipelines** across **13 industries**, the network auto-harvests **1,250+ verified intelligence records every 24 hours** at the global edge on Cloudflare Workers, R2, and D1.
---
## ๐๏ธ Autonomous Agent Dual-Shield Workflow
```
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Autonomous AI Agent (LangChain / LlamaIndex) โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Target URL
โผ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ PhishVision Security Shield โ
โ - Heuristic Brand Distance โ
โ - Zero-Day Phishing Kits โ
โ - Web3 Drainer Signatures โ
โโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโ
โ
โโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโ
โ Verdict โ
[ MALICIOUS ] [ SAFE ]
โ โ
โผ โผ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Halt & Shield Agent Loop โ โ OpticParse Edge Extractor โ
โ (Protect API & Wallet) โ โ - Zero-CSS Visual Parse โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ - 96% Token Reduction โ
โโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโ
โ Clean Markdown
โผ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ LLM Context Window / RAG โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
```
---
## ๐๏ธ System Architecture
```
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ OpticParse System Architecture โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ โ
โ โโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโ โ
โ โ 150 Scraping โโโโถโ Cloudflare โโโโถโ Cloudflare โ โ
โ โ Pipelines โ โ Workers Edge โ โ D1 + R2 Lake โ โ
โ โโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโ โโโโโโโโฌโโโโโโโโ โ
โ โ โ
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโ โ
โ โ Distribution โ โ โ
โ โผ โผ โผ โผ โ
โ โโโโโโโโโโโโ โโโโโโโโโโโโ โโโโโโโโโโโโ โโโโโโโ โ
โ โ Kaggle โ โ Hugging โ โ RapidAPI โ โOceanโ โ
โ โ Hub โ โ Face Hub โ โ Gateway โ โ NFTsโ โ
โ โโโโโโโโโโโโ โโโโโโโโโโโโ โโโโโโโโโโโโ โโโโโโโ โ
โ โ
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
โ โ Anthropic Model Context Protocol (MCP) Server for Agents โ โ
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
```
---
## ๐ฏ 150-Pipeline Industry Coverage Matrix
| Industry Vertical | Active Pipelines | Core Extractions & Capabilities |
| :--- | :---: | :--- |
| **๐ก๏ธ Cybersecurity & Threat Intel** | **70** | Zero-day phishing kits, typosquats, crypto-drainers, brand impersonation |
| **๐ E-Commerce & Retail Arbitrage** | **15** | Amazon undercut alerts, Shopify inventory snipers, price elasticity |
| **๐ผ B2B Growth & Lead Signals** | **45** | Executive hiring telemetry, YC startup surge, wage trends, SEC filings |
| **โก Finance, Crypto & DEX Arbitrage**| **20** | Liquidity pool spreads, token mint monitors, funding rate surveillance |
---
## ๐ Verified Adversarial Stress Benchmarks (Live September 2026 Audit)
Both engines were independently benchmarked against live adversarial threats and enterprise-grade anti-bot defenses:
### 1. OpticParse: Zero-CSS Visual Extraction Stress Test
Tested live on production backend against complex dynamic JavaScript, nested schemas, and anti-bot protected targets:
| Target Page | Challenge Architecture | Result | Latency | Data Extracted |
| :--- | :--- | :---: | :---: | :--- |
| **Cloudflare Official Plans** | Enterprise Bot-Shield & obfuscated DOM | **100% PASS** | 42.1s | Extracted complete tiered pricing structure (`Free`, `Pro`, `Business`) |
| **Stripe Global Pricing** | Heavy React hydration & dynamic cards | **100% PASS** | 38.2s | Extracted standard card processing fees (`2.9% + $0.30`) |
| **Hacker News Frontpage** | Real-time community feed & score metadata | **100% PASS** | 42.7s | Extracted top 5 stories with real ranks and point totals |
| **GitHub Trending Hub** | Nested repository schemas & live star counts | **100% PASS** | 32.3s | Extracted trending repositories (`skills`, `ponytail`) with exact stars |
* **Zero-CSS Extraction Accuracy:** **4 / 4 (100.0% Success Rate)**
* **DOM Fragility:** Zero broken CSS/XPath selectors; resilient to class renaming and layout refactors.
> ๐ก **Understanding the ~35s Latency: Why Multimodal Vision Beats Brittle 2-Second Scrapers**
> * Traditional scrapers finish in 2 seconds because they only regex raw HTML text. The moment a site deploys dynamic React hydration, anti-bot protection, or changes CSS class names, **traditional scrapers break and return empty data or 403 Forbidden errors**.
> * **OpticParse prioritizes zero-failure autonomous agent execution:** It spins up a full headless Chromium browser, bypasses anti-bot layers, captures a high-resolution visual buffer, and has a Multimodal Vision LLM reason over the page like a human.
> * **Result:** 100% schema accuracy with **zero broken CSS selectors**, guaranteed for autonomous background agents where data integrity matters more than raw millisecond speed.
---
### 2. PhishVision: 0-Day Adversarial Threat & Crypto Drainer Audit
Audited against **50 live zero-day malicious URLs** from the **OpenPhish global threat feed** (created within hours of test) + **20 difficult authentic authentication portals**:
| Security Benchmark Metric | PhishVision Score | Traditional Baseline (Google Safe Browsing / VirusTotal on Day 0) |
| :--- | :---: | :---: |
| **0-Day Threat Catch Rate** | **68.0% (34 / 50)** | **~10% โ 15%** (Fails because new domains lack historical reports) |
| **Authentic Auth Portals (Clean)** | **95.0% (19 / 20)** | **>90%** (Industry standard for benign enterprise login portals) |
| **Median Execution Latency** | **1.58 seconds** | **5 โ 15 seconds** (Heavy commercial sandbox scanners) |
#### Verified Live Threats Neutralized On First Contact:
* ๐ก๏ธ **Uniswap Wallet Drainer:** `uniswap-interface.vercel.app` (BLOCKED)
* ๐ก๏ธ **Trezor Hardware Seed Stealer:** `sso-trezor-com-start-x-auth.typedream.app` (BLOCKED)
* ๐ก๏ธ **TrustWallet Crypto Harvester:** `trust-wallet-liart.vercel.app` (BLOCKED)
* ๐ก๏ธ **Compromised WordPress Injections:** `bruceleephilosophy.com/texts/` (BLOCKED)
* ๐ก๏ธ **Apple Brand Impersonator:** `apple-fruit.xyz` (BLOCKED)
#### Verified Authentic Portals Cleared as Safe:
* `accounts.google.com` • `github.com/login` • `dashboard.stripe.com/login` • `appleid.apple.com` • `reddit.com/login` • `auth.openai.com` (All CLEARED with zero false alarm interruptions).
---
## ๐ Python & LangChain Quickstart (PyPI)
Install the official Python SDK or LangChain multi-agent toolkit:
```bash
# Core Python SDK (OpticParse AI Scraper + PhishVision)
pip install opticparse-py
# LangChain & CrewAI Tool Integration
pip install langchain-opticparse
# LlamaIndex Tool Integration
pip install llama-index-tools-opticparse
```
### LangChain Dual-Shield Agent Usage:
```python
from langchain_opticparse import OpticParseTool, PhishVisionTool
# Initialize tools
phish_shield = PhishVisionTool(api_key="your_api_key")
optic_scraper = OpticParseTool(api_key="your_api_key")
target_url = "https://example.com/pricing"
# 1. Pre-flight security audit: Block phishing kits & wallet drainers
threat_audit = phish_shield.run({"url": target_url})
if threat_audit.get("verdict") == "MALICIOUS":
print(f"๐จ Security Alert: Agent halted. Threat detected (Score: {threat_audit['threat_score']}/100)")
else:
# 2. Extract clean visual Markdown without brittle CSS selectors
clean_data = optic_scraper.run({
"url": target_url,
"query": "Extract tier prices, plan limits, and feature comparisons"
})
print(clean_data)
```
> โก **200 Free Trial Extractions & In-IDE Activation:**
> Every IP receives **200 free trial requests** out of the box. When your free trial completes, the terminal automatically prints an ASCII QR code + 1-click checkout link. Upon paying on-chain ($10 Starter / $50 Growth / $200 Scale / $0.05 M2M via MetaMask or mobile wallet), your live API key is automatically minted, saved directly to your local `.env`, and your code resumes execution seamlessly.
### LlamaIndex Usage:
```python
from llama_index.tools.opticparse import OpticParseToolSpec
from llama_index.core.agent import FunctionCallingAgentWorker
# Initialize tool spec and extract documents
tool_spec = OpticParseToolSpec(api_key="your_api_key")
docs = tool_spec.extract(url="https://example.com", query="Extract specs")
print(docs[0].text)
# Convert directly to agent tool list
tools = tool_spec.to_tool_list()
```
### Python Agent Usage:
```python
from opticparse import OpticParse
client = OpticParse(api_key="YOUR_API_KEY")
# 1. Clean Markdown Extraction for LLM RAG pipelines
res = client.extract_markdown("https://news.ycombinator.com")
print(res["markdown"])
# 2. Autonomous Zero-Day Threat Inspection
safety = client.detect_phishing("https://suspicious-dapp-claim.xyz")
print(f"Verdict: {safety['verdict']} | Threat Score: {safety['threat_score']}/100")
```
---
## ๐ค ElizaOS Autonomous Agent Plugin (npm)
Install the official ElizaOS autonomous agent plugin:
```bash
npm install opticparse-eliza-plugin
```
### ElizaOS Usage:
```typescript
import { AgentRuntime } from "@elizaos/core";
import { opticParsePlugin } from "opticparse-eliza-plugin";
export const webScoutAgent = {
name: "WebScoutAgent",
plugins: [opticParsePlugin],
settings: {
secrets: {
OPTICPARSE_API_KEY: process.env.OPTICPARSE_API_KEY
}
}
};
```
---
## ๐ค Model Context Protocol (MCP) Integration
Connect **Claude Desktop, Cursor IDE, or AutoGen** directly to OpticParse in 1 click:
### 1. Install via Smithery
```bash
npx -y @smithery/cli install @parastejpal987-cmyk/opticparse --client claude
```
### 2. Manual Configuration (`claude_desktop_config.json`)
```json
{
"mcpServers": {
"opticparse": {
"command": "python",
"args": ["-m", "mcp_server"],
"env": {
"OPTICPARSE_API_KEY": "YOUR_API_KEY"
}
}
}
}
```
### ๐ ๏ธ Exposed AI Agent Tools:
* `opticparse_scrape`: Vision-based structured data extraction from any web URL.
* `phishvision_detect`: Real-time phishing and brand impersonation heuristic scanner.
* `search_lessons`: Query indexed threat telemetry records.
---
## ๐ฎ Postman Public API Collections
Test the live endpoints instantly in Postman with **200 free trial requests** built-in:
* **OpticParse Visual Web Scraper Collection:** [Run in Postman](https://god.net.objectstorage.softlayer.net/v1/AUTH_084e34cb-d467-422f-87d2-7c3a81283d08/postman-run-button/run-button.svg) โ Includes visual extraction, token-optimized RAG markdown parsing, and live system health checks.
* **PhishVision AI Threat Shield Collection:** Includes 0-day phishing scanning, brand impersonation detection, and permit2 drainer audits in <1.6s.
* **Offline Import:** Both collections are available directly in this repository under [`postman/`](postman/):
- `postman/opticparse_api_collection.json`
- `postman/phishvision_api_collection.json`
---
## ๐ก๏ธ Embed Dynamic Scrapeability & Security Badges
Showcase that your open-source repository, documentation, or SaaS application is scrapeable by autonomous AI agents or verified safe against zero-day phishing:
### 1. OpticParse Scrapeability Badge
```markdown
[](https://opticparse.com)
```
Output:
### 2. PhishVision 0-Day Safe Badge
```markdown
[](https://opticparse.com)
```
Output:
---
## ๐ฆ Master Public Datasets (Hugging Face & Kaggle)
All master datasets are public, verified, and streamable in **Apache Parquet & CSV format**:
| Dataset Name | Records | Format | Direct Access |
| :--- | :---: | :---: | :--- |
| **Master 150-Template Catalog** | `150` | `CSV / Parquet` | [Hugging Face](https://huggingface.co/datasets/paras9909/opticparse-150-template-web-corpus) / [Kaggle](https://www.kaggle.com/datasets/parastejpal/opticparse-150-template-web-corpus) |
| **PhishVision Threat Intelligence** | `70` | `CSV / Parquet` | [Hugging Face](https://huggingface.co/datasets/paras9909/opticparse-150-template-web-corpus) / [Kaggle](https://www.kaggle.com/datasets/parastejpal/opticparse-150-template-web-corpus) |
| **E-Commerce & Retail Arbitrage** | `15` | `CSV / Parquet` | [Hugging Face](https://huggingface.co/datasets/paras9909/opticparse-150-template-web-corpus) / [Kaggle](https://www.kaggle.com/datasets/parastejpal/opticparse-150-template-web-corpus) |
| **B2B Growth & Financial Signals** | `45` | `CSV / Parquet` | [Hugging Face](https://huggingface.co/datasets/paras9909/opticparse-150-template-web-corpus) / [Kaggle](https://www.kaggle.com/datasets/parastejpal/opticparse-150-template-web-corpus) |
### ๐ป Load in Python (1-Line Quickstart):
```python
import pandas as pd
# Load master threat intelligence dataset
df_threat = pd.read_csv("https://opticparse.com/threat_intel_dataset.csv")
print(f"Loaded {len(df_threat)} live threat vectors")
print(df_threat.head())
```
---
## โก Commercial RapidAPI Gateway
Pay-As-You-Go developer access with **sub-350ms global edge latency** at `$0.008/request`:
1. **`01-ai-web-scraper`**: Autonomous AI vision web scraper
2. **`02-dom-poisoner`**: Streaming adversarial anti-scraping tag injector
3. **`03-edge-proxy`**: Global unblockable fetcher proxy
4. **`04-rate-limit-bypasser`**: Residential edge IP rotator
5. **`05-realtime-sentiment`**: Live threat & telemetry database feed
๐ **Target Gateway URL:** `https://opticparse-rapidapi-gateway.parastejpal987.workers.dev`
---
## ๐ฆ Autonomous AI Agent Micropayments (HTTP 402 Machine Paywall)
For autonomous bots, multi-agent frameworks (ElizaOS, AutoGPT, CrewAI), and automated scrapers with no human in the loop, OpticParse supports **instant on-chain settlement**:
* **Price**: `$0.01 USDC` per request
* **Supported Chains**: **Base**, **Polygon**, **Arbitrum**
* **Treasury Address**: `0xd458E709e7d54fd3659EF66624A621Cde74EDD27`
* **Machine Manifest**: [`https://opticparse.com/.well-known/agent.json`](https://opticparse.com/.well-known/agent.json) (Google A2A Standard)
### ๐ค 1-Line Autonomous Agent Request:
```bash
curl -X POST https://opticparse-edge.parastejpal987.workers.dev/api/edge/scrape \
-H "X-Payment-TxHash: 
