Spaces:
Sleeping
Sleeping
Commit History
-
fix: add OpenEnv root reset and step aliases
Expose OpenEnv-compatible root endpoints (/reset, /step, /state/{episode_id}) and API aliases (/api/reset, /api/step, /api/state/{episode_id}) to prevent 405 on evaluator probes. Add endpoint coercion for relaxed action payloads and tests for alias behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>b4b210e -
feat: add dynamic registry-driven agent tool runtime
- Added AgentToolCaller with LLM-driven tool planning from plugin registry metadata\n- Added ToolExecutor with runtime namespace dispatch and non-hardcoded execution\n- Expanded plugin registry from 71 to 82 tools (parser/data/analysis/extraction/validation additions)\n- Integrated runtime tool decisions and tool observations into agentic scrape flow\n- Verified selected tools execute and vary by prompt for different scraping tasks\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>5b2dac6 -
fix: improve LLM extraction prompts and column name parsing
- Added _parse_column_names() helper to properly extract column names from output_instructions - Fixed extraction prompt to guide LLM to extract actual content not empty strings - Updated requirements to emphasize extracting real data from HTML elements - Added backend/README.md for build compatibility - Created comprehensive LLM integration status report in docs/ VERIFIED: Streaming response DOES return output field with extracted data ISSUE IDENTIFIED: LLM extraction code quality needs improvement - often returns empty values NEXT: Test with improved prompts on diverse sites06af10e -
feat: prioritize user-site extraction over external search, improve metric patterns for HN-style sites
fc5088a
-
fix: improve complex link extraction for Twitch-style markdown
9c16219 -
feat: intelligent search-based navigation for trending/popular content
- Add _infer_navigation_paths() for generic path inference based on intent - Add _build_search_navigation_url() for search-based fallback - Prefer search URLs for top/trending/popular queries (work without auth) - Rewrite _extract_markdown_link_rows() with smarter content detection: - Skip image links (![Image]) in markdown - Extract metrics (views, likes, comments) from nearby lines - Filter boilerplate labels and URL tokens aggressively - Rank results by keyword relevance and engagement metrics - Expand boilerplate filtering in _fallback_extraction_code() - Fix alias matching in registry.py (boundary-aware matching) - Add regression tests for alias matching This enables generic extraction for JS-heavy sites like YouTube by using search URLs that return content without requiring authentication.0735175 -
fix: enforce template-hint agentic flow and strict output schema
- keep templates as navigation/extraction hints instead of hard dependencies - remove site-specific fallback navigation branches in favor of plan target URL hints - include assets and output format explicitly in agent prompts - enforce requested output columns by projecting extracted rows to output instructions schema - preserve deterministic fallback behavior for offline/test execution without hardcoded scrapers - keep regression coverage green for scrape API suite Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>bfec523 -
refactor: remove hardcoded strategy scrapers
- delete legacy _scrape_github_trending/_scrape_reddit_trending/_scrape_single_page/_scrape_with_exploration flows - keep a single agentic scraper path via _scrape_with_agentic_llm - add deterministic fallback planning and extraction when live LLM calls are unavailable - emit validate.url, html.extract, extract.urls, and extract.emails tool calls in agentic flow - restore _fetch_reddit_communities compatibility helper for E2E monkeypatch fixtures - keep gold-data verifier as partial signal without forcing session-level partial failures - validate with scrape API regression tests (asset resolution + 100-case E2E suite) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>0a43df3 -
feat: Improve LLM extraction prompts for better data quality
- Increase HTML sample from 5000 to 15000 chars to include content - Add explicit example output format in extraction prompt - Clarify column name requirements matching output_instructions - Add temperature=0.3 for more deterministic code generation - Log generated code on execution failures for debugging - Test results: username/repo/stars now extracted correctly - Forks field empty (acceptable - LLM extracts what's visible in HTML)1421a1a -
fix: Kill stale backend process and verify agentic scraping works
- Identified two backend processes listening on port 8000 - Killed stale process (PID 32444) that was serving old hardcoded code - Verified agentic scraper now executes with LLM-driven decisions - LLM successfully decided navigation URL (github.com/trending?since=weekly) - LLM generated BeautifulSoup extraction code dynamically - Extraction partially working (stars/forks OK, username/repo need refinement) - System now operates without hardcoded scraping logic405f6a4 -
feat: Replace hardcoded scraping with truly agentic LLM-driven approach
- Add _scrape_with_agentic_llm() function that uses model router for all decisions - LLM now decides navigation URLs based on user instructions + template hints - LLM generates BeautifulSoup extraction code dynamically from HTML + instructions - Execute generated code in sandbox for flexible data extraction - Templates serve as reference hints only, not rigid execution scripts - Works even without templates (pure agentic mode) - Output columns now driven by user's output_instructions - Replace all hardcoded strategy routing (github_trending, reddit_trending, etc) with single agentic pathf946069 -
fix: GitHub trending CSV output returns correct columns
- Strategy detection now properly triggers github_trending for requests like 'Extract the top 5 repos of this week' - CSV output now returns exactly the requested columns (username, repo, stars, forks) - Extracted data structure includes pre-formatted csv_output for direct download - Fixed issue where Docker container was intercepting requests with stale code The fix ensures the scraper: 1. Detects GitHub trending signals ('top repos', 'this week', 'trending', etc.) 2. Navigates to github.com/trending instead of github.com homepage 3. Extracts repository data with correct column mapping 4. Formats output according to user's output_instructions4ece098 -
fix: resolve bare-domain assets to direct URLs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>4b354aa
-
feat: add validate.url and html.parse tool calls for comprehensive visibility
b136a9f -
feat: add comprehensive plugin registry with 71 tools across 12 plugins
fa40af9 -
feat: add comprehensive tool call visibility in step accordion
48f04de -
feat: add tool call visibility in step accordion
- emit tool_call step events for browser.navigate, html.parse, html.select, csv.generate - show tool name, description, parameters, and result in frontend accordion - add Zap icon for tool_call action with yellow highlight - display tool call details section with structured parameter/result views This enables users to see exactly which tools agents invoke during scraping.64ae2f9 -
feat: add site template registry and agent integration
- add backend/app/sites template catalog with 56 templates - expose /api/sites list/get/match endpoints - wire scrape planner/navigator to resolve and reference templates - add per-url template-aware strategy selection for scraping Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>e13f862 -
fix: proper rewards for all steps and clean CSV output
- Added rewards to plugin, planner, planner_python, navigator, navigator_python steps - Navigation step now gets 0.5 reward on success - Extraction step calculates reward based on repo count (0.5 per repo + 1.0 bonus) - CSV output now returns clean data without nested structure - format_output() checks for csv_output key and returns it directly - Total rewards now accumulate correctly across all steps13402be -
fix: intelligent navigation scraper with GitHub trending support
a04acb3 -
feat: implement intelligent agentic web scraper
- Added intelligent navigation planning based on instructions - Created GitHub trending repository scraper strategy - Enhanced planner to understand goals like 'trending repos' - Added CSV output for repository data (username, repo_name, stars, forks) - Improved agent orchestration for goal-oriented scraping - Added intelligent exploration vs single-page strategies The scraper now understands instructions like 'Get me all trending repo' and navigates intelligently to GitHub trending pages instead of just doing basic field extraction from the homepage.82fb385 -
fix: resolve scraper functionality and plugin issues
- Fixed plugin registry missing web_scraper and python_sandbox - Removed locals() from sandbox BLOCKED_CALLS for analysis - Fixed frontend health check API response parsing - Added comprehensive test validation framework - Verified all agents (planner, navigator, extractor, verifier) working - Confirmed Python sandbox execution with numpy/pandas/bs4 - Validated real-world URL scraping capabilities - Added session artifact management and memory integration54ec9cb
-
test: add end-to-end scraper test script with successful validation
8512126 -
feat: add WebSocket support for real-time scraper progress updates
4afa792 -
feat: add providers API endpoints with model and cost tracking
1226a86 -
feat: add embeddings service and update plugins with NVIDIA support
e5b4f8d
-
fix: support settings endpoint without trailing slash
f659d59 -
fix: support plugins endpoint without trailing slash
3232831 -
test: add comprehensive API and core module tests
- Add 30+ new tests for memory, tasks, episode modules - Episode tests cover lifecycle, steps, manager operations - Memory API tests cover store, query, delete operations - Tasks API tests cover list, filter, create operations - All 101 tests passing with 44% coveragee8d7c11 -
fix: add list_providers method and agents list endpoint
e497b8a
-
feat: add modular plugin system for extensions
942fc42 -
feat: add client API key configuration and model selection
5c03d80 -
feat: add API routes and utility modules
27cde0c