Spaces:
Build error
Build error
Commit History
-
fix: centralize + desktop-mode-adapt the logs directory (P-1)
New finding while implementing this: config/settings.py's LOGS_PATH constant was dead code -- grepped every consumer, zero hits anywhere outside its own definition. The 3 real log-writing call sites each independently hardcoded or recomputed their own path instead: - scrapers/base_scraper.py: a bare CWD-relative "logs" string - api/middleware/audit_logger.py: __file__-relative, 3 dirname() calls up to project root - blockchain/audit_chain.py: __file__-relative, 2 dirname() calls up to project root (different file depth, same target) All 3 resolve to the identical directory TODAY only because Docker's fixed WORKDIR makes CWD == project root -- a coincidence, not a guarantee. A desktop build has no such guarantee: a double-clicked app's CWD is unpredictable, so the CWD-relative scraper logs and the __file__-relative audit logs could silently end up in two different, unrelated directories the moment this ships as a desktop app. Fix: LOGS_PATH is now desktop-mode-aware (same…6d12c24
-
fix(ascii): replace all non-ASCII chars in Python source files
CI check requires all .py files (except transliteration/multilingual_ner/ translator/languages) to be pure ASCII. Pre-existing violations fixed: U+03BB lambda -> lambda_val U+2014 em-dash -> -- U+2013 en-dash -> -- U+20B9 rupee Rs -> Rs. U+2192 arrow -> -> U+2500 box-h -> - U+2502 box-v -> | U+2081 subscript -> 1fa87992
-
fix(BUG-C5): GemScraper never returns fake sample data; returns None/[] with explicit warning on failure
0c26e07
-
fix(M-09): guard os.makedirs() in save_json() against empty dirname -- filenames without path component no longer crash
0e43bbf
-
fix(NEW-A3 part1): replace em-dashes/smart-quotes in 56+ Python files -- CI requires pure ASCII source
8a0fba4 -
fix(BUG-29): add haryana_2024, jk_2024, odisha_2024, ap_2024, sikkim_2024, arunachal_2024, bihar_2025 election URLs
67cb2b8
-
fix(encoding): strip non-ASCII from 4 remaining files
investigation.py, myneta_scraper.py, components.js, and settings.py contained UTF-8 em dashes and curly quotes in comments which cause UnicodeDecodeError on GitHub Actions runners.23cf286 -
fix(scrapers+pipeline): H-03+L-02+H-12 -- MyNeta elections and git bloat
H-03: STATE_PAGES only had 3 stale elections (lok_sabha_2024, delhi_2020, tamil_nadu_2021). 10+ major 2022-2024 elections were completely missed: UP 2022, Gujarat 2022, Karnataka 2023, Rajasthan 2023, MP 2023, Telangana 2023, Maharashtra 2024, Jharkhand 2024, etc. Fix: expanded to 14 elections covering all recent major elections. L-02: delhi_2020 was defined in STATE_PAGES but fetch_sample_data() only iterated ['lok_sabha_2024', 'tamil_nadu_2021'] -- delhi_2020 was always skipped. Fix: iterate list(self.STATE_PAGES.keys()) instead. H-12: pipeline.save() included the full 'raw' key (all scraper output) in the JSON committed to git by the GitHub Action. With 20 scrapers x 50 records each plus nested objects, these files grew large and accumulated in git history indefinitely. Fix: save() now only writes summary+links; raw is kept in memory only.1ffe057
-
fix(scrapers): BUG-29 -- logs dir may not exist in Docker environments
scrapers/base_scraper.py called logger.add('logs/{name}.log') in every scraper __init__ without ensuring the directory exists first. In Docker and HuggingFace Spaces, logs/ may be read-only or absent. loguru.add() would silently fail or throw FileNotFoundError. Fix: added os.makedirs('logs', exist_ok=True) before logger.add and added enqueue=True, catch=True to prevent log write failures from crashing the scraper process.b62bf48
-
feat(scrapers): complete — 8 new Indian sources, total 21 scrapers
Confirmed results: - njdg: LIVE — 39 real records from njdg.ecourts.gov.in - ed: 404 on /press-release — sample fallback working - cvc: 0 rows parsed from live page — sample fallback working - ncrb: OpenCity returns HTML not JSON — sample fallback working - lgd: DataGov API key required — sample fallback working - ibbi: 500 server error — sample fallback working - ngo_darpan: DataGov API key required — sample fallback working - cppp: DataGov API key required — sample fallback working All 8 scrapers: syntax clean, sample fallback correct, JSON saved. Total scrapers: 216bcb5d3
-
fix( ): add 5 scrapers, fix README links
README FIX: - Converted footer links to clickable markdown format - Added PHASE_ROADMAP.md link SCRAPERS RESTORED: - scrapers/loksabha_scraper.py - scrapers/sebi_scraper.py - scrapers/electoral_bond_scraper.py - scrapers/opensanctions_scraper.py - scrapers/wikidata_scraper.py8af87f7 -
fix(scrapers): ICIJ HTML scraping instead of JSON API, restore .env.example
- icij_scraper.py: ICIJ /search returns HTML not JSON Switched from get_json to get_html + BeautifulSoup table parsing Handles 0-result responses gracefully with clear logging - .env.example: was empty - restored with all 5 required keys DATAGOV_API_KEY, NEO4J_URI/USER/PASSWORD, OPENSANCTIONS_API_KEY, NEWSAPI_KEY8958e86
-
fix(pib+gitignore): PIB HTML scraper + clean gitignore
- pib_scraper.py: HTML scraping pib.gov.in/allRel.aspx CONFIRMED WORKING: fetched 2 real live PIB press releases Removed feedparser completely (PIB server blocks it) Browser User-Agent added to prevent blocking - .gitignore: completely rewritten, no duplicates Added data/samples/*.json and data/processed/*.json8e77b70
-
fix(scrapers): add missing import os to mca_scraper.py
- mca_scraper.py had os.getenv() at class level without importing os - base_scraper.py already fixed with load_dotenv() in previous commit - All 7 scrapers now import os correctly421aac9 -
fix(security): remove real API key from repo, fix gem_scraper bugs
Fixes #4 - .env.example: replaced real DATAGOV_API_KEY with placeholder text (real key was accidentally committed - security risk) - config/settings.py: removed hardcoded key fallback (now empty string) - gem_scraper.py: fixed 3 bugs: 1. Added missing 'import os' at top 2. Fixed empty __init__ - added self.api_key = os.getenv() 3. Fixed fetch_gem_stats() indentation (body was at class level) - All 7 scrapers now pass syntax check7590455
-
fix(scrapers): update PIB RSS feed URLs, add empty feed handling
Fixes #4 - Updated RSS_FEEDS with verified working PIB URLs - Added fallback HTML fetch when RSS returns 0 entries - PIB may require session cookies - will investigate in next iteration - Logs HTML response size to help debug future RSS issues0c2b3a5
-
fix(scrapers): correct GeM stats endpoint, use gem.gov.in/statistics
Fixes #3 - Old URL mkp.gem.gov.in/api/v2/public/stats returned 404 - Updated to use gem.gov.in/statistics (public stats page) - fetch_gem_stats() now fetches HTML page instead of broken JSON API - Sample data still returned as fallback with correct structurea4f3e18 -
fix(scrapers): load DataGov API key from .env, remove hardcoded key
Fixes #2 - All scrapers now read DATAGOV_API_KEY from .env via os.getenv() - Removed hardcoded public key (was causing 403 errors) - Added clear warning when key is missing - Updated .env.example with setup instructions - Register free at: https://data.gov.in/user/register4baa251 -
docs(scrapers): update README with all 6 scrapers + known issues
- Added CAG and GeM scrapers to table - Documented known 403 issues with DataGov API - Added setup instructions with .env API key note2a9acf6 -
feat(scrapers): add GeM government contracts scraper
- Scrapes procurement data from Government e-Marketplace (gem.gov.in) - Extracts: order ID, buyer org, seller name, GSTIN, amount, date - find_repeated_winners() detects contract concentration (key risk flag) - Sample data shows same company winning 2 contracts (pattern detection) - Links to politician graph: seller company → directors → politicians - Falls back to structured sample data when API unavailable0e9b6d0 -
feat(scrapers): add CAG audit report scraper
- Scrapes Comptroller & Auditor General reports from cag.gov.in - Detects fraud keywords: misappropriation, ghost beneficiaries, etc. - get_high_value_irregularities() filters reports by crore amount - Sample data based on real CAG findings (TN MGNREGA, Assam PM-KISAN) - Key for graph: links audit findings to schemes, states, departments - 3s polite delay, falls back to structured sample data563cd5e -
fix(scrapers): update PIB RSS URLs, add .env API key setup
- PIB RSS feed URLs corrected - .env.example updated with DATAGOV_API_KEY placeholder - Users must register free at data.gov.in for personal API keyf0cecec -
docs(scrapers): add README documenting all scrapers and usage
- Documents all 4 scrapers with source and data collected - Includes run instructions and legal note - Will be updated as more scrapers are addedf452d21 -
feat(scrapers): add MCA company registry scraper
- Fetches company data from Ministry of Corporate Affairs - Uses data.gov.in public API (free, no login needed) - Extracts: CIN, company name, status, state, registration date - search_company_by_name() enables politician-company linking - Falls back to sample data if API unavailable - Key use: connect politicians → company directors → contracts - Supports Tamil Nadu, Maharashtra, Delhi (expandable)50f9e90 -
feat(scrapers): add MyNeta/ADR candidate affidavit scraper
- Scrapes politician data from myneta.info (ADR project) - Extracts: name, party, assets, liabilities, criminal cases, education - Data source: Election Commission of India affidavits (100% public) - Auto-flags candidates with declared criminal cases - Supports Lok Sabha 2024, Tamil Nadu 2021, Delhi 2020 - 3s polite delay between requests - Saves output to data/samples/ with timestamp67940af -
fix: add __init__.py to all packages, create venv setup
- Added __init__.py to scrapers, processing, graph, ai, api, blockchain, config, tests folders - Fixes ModuleNotFoundError when running scrapers - Added main.py as project entry point - Updated .gitignore to exclude venv/ and __pycache__/ - All scripts must now run from project root with: python -m scrapers.Xd76057f -
feat(scrapers): add PIB press release scraper
- Fetches official press releases from Press Information Bureau - Parses RSS feeds from all ministries - Auto-detects alert keywords (scheme, crore, contract, fund etc.) - get_high_alert_articles() filters money/scheme-related posts - Deduplicates articles across feeds - Fully legal: PIB is 100% public government datad0cb0ea -
feat(scrapers): add DataGov scraper for govt open datasets
- Fetches datasets from India's data.gov.in portal - Supports MGNREGA and PM-KISAN state-level data - Auto-saves JSON to data/samples/ with timestamp - Inherits rate limiting from BaseScraper (polite 2s delay) - Uses public API key (no registration needed)a2a0f68
-
feat: add requirements.txt and base_scraper.py - Day 2
1c02240