Spaces:
Sleeping
Sleeping
Commit History
-
feat: Major scraper enhancements - consistent format, better extraction, per-month costs
Riley +1: ClaudeThis commit significantly improves the grant scraper to extract more complete and consistent data: ## Scraper Improvements (snapshot.py) ### 1. Consistent JSON Structure - Added `id` and `competition_id` fields to all snapshots - Ensures compatibility between manual (snapshot.py) and auto (discover_grants.py) scraping - All existing code handles both old and new formats gracefully ### 2. Enhanced Funding Rates Extraction - Pattern 1: Standard format "up to 60% if you are micro/small" - Pattern 2: Combined format "up to 60% if you are a micro, small or medium" - Pattern 3: KTP format "large companies...50%...SMEs...67%" - Now extracts rates for diverse grant types ### 3. Enhanced Duration Extraction - Range: "last/must be between 12 and 36 months" - Single max: "last up to nine months" (supports word numbers like "nine") - Maximum: "maximum 24 months" - Added word-to-number conversion for written numbers ### 4. Per-Month Cost Calculation (NEW) - Detects patterns like "£8,500 per mo…2ae7490 -
feat: Merge GPT-5 enhancements with working c72a240 base + restore crawler
Riley +1: ClaudeMajor update combining the stable c72a240 Hugging Face deployment base with GPT-5 improvements from 057c21e. ## Core Enhancements Added: ### GPT-5 Model Support (Step 1 of 15) - Three-tier model system: nano/mini/main for different use cases - model_router (gpt-5-nano): Fast routing and classification - model_translator (gpt-5-mini): Translations and summaries - model_analyzer (gpt-5): Complex analysis and comparisons - Verbosity control: low/medium/high response lengths - Reasoning effort: minimal/medium/high thinking depth - Task-specific presets for optimal parameters ### Files Modified: - analyzer/llm_client.py: Added GPT-5 support with model types - analyzer/config.py: Added GPT-5 model configurations - src/analyzer/*: Synced all files to match analyzer/ base ### Crawler Restored: - analyzer/crawler/snapshot.py: Grant page scraping - analyzer/crawler/discover_grants.py: Grant URL discovery - Both restored from 057c21e commit ### Preserved from c72a240 Working Base: - ✓ 7 pres…cf9b3dc -
feat: Major system enhancements - GPT-5 support, monitoring, translation, and optimizations
Riley +1: ClaudeThis commit includes 15 major improvements to the grant-analyst system: ## New Features ### GPT-5 Model Support (Step 1) - Implemented three-tier GPT-5 model system: - gpt-5-nano for routing and simple decisions - gpt-5-mini for translations and summaries - gpt-5 for complex analysis - Updated token encoding to use o200k_base (GPT-5) with cl100k_base fallback - Updated all configuration files and documentation ### Grant Translation Endpoint (Step 14) - New `/translate` API endpoint for converting grant jargon to simple language - Structured output with 4 sections: eligibility, process, timeline, requirements - Uses gpt-5-mini model for cost-effective translations - Permanent caching in MongoDB (translations don't change) - Batch translation support via `/translate/batch` ### Monitoring & Metrics System (Step 15) - New metrics collection system tracking: - P50/P95/P99 latency per endpoint - Token usage per request (prompt + completion) - Cache hit/miss rates (Redis + Mon…057c21e
-
docs: remove unnecessary markdown files - keep only README and DEPLOY
Riley Coleman3b4387f
-
feat: add PDF & supporting materials integration to context
Riley Coleman- Add build_context_with_supporting() function to embed extracted PDF text - Add get_supporting_docs_for_grant() to retrieve supporting materials - Add _load_supporting_docs_jsonl() helper to index 1,124 supporting documents - Auto-extract grant ID from URL if not present - Support filtering by document type (PDF only or mixed) - Include top k supporting docs with full text in LLM context Benefits: ✓ Model now has direct access to PDF content (not just URLs) ✓ Can answer questions about briefing materials and policies ✓ Can cite specific sections from supporting documents ✓ ~1,120 HTML sections + 4 PDFs available for context ✓ No external API calls needed - content is embedded Usage: from src.analyzer.context_builder import build_context_with_supporting context = build_context_with_supporting(grant, k=5) # Pass to LLM with full supporting materials embedded See PDF_CONTEXT_INTEGRATION.md for complete integration guide. 🤖 Generated with Claude Codebfcc872