Spaces:
Build error
Build error
Commit Β·
2a9acf6
1
Parent(s): 0e9b6d0
docs(scrapers): update README with all 6 scrapers + known issues
Browse files- Added CAG and GeM scrapers to table
- Documented known 403 issues with DataGov API
- Added setup instructions with .env API key note
- scrapers/README.md +28 -18
scrapers/README.md
CHANGED
|
@@ -1,32 +1,42 @@
|
|
| 1 |
-
cat > scrapers/README.md << 'EOF'
|
| 2 |
# BharatGraph Scrapers
|
| 3 |
|
| 4 |
All scrapers collect **public data only**. No private data, no logins.
|
| 5 |
|
| 6 |
## Scrapers
|
| 7 |
|
| 8 |
-
| File | Source | Data Collected |
|
| 9 |
-
|------|--------|----------------|
|
| 10 |
-
| `base_scraper.py` | - | Base class, rate limiting, logging |
|
| 11 |
-
| `datagov_scraper.py` | data.gov.in | MGNREGA, PM-KISAN datasets |
|
| 12 |
-
| `pib_scraper.py` | pib.gov.in | Govt press releases, cabinet decisions |
|
| 13 |
-
| `myneta_scraper.py` | myneta.info | Politician assets, criminal cases, education |
|
| 14 |
-
| `mca_scraper.py` | MCA / data.gov.in | Company registrations, directors |
|
|
|
|
|
|
|
| 15 |
|
| 16 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 17 |
```bash
|
| 18 |
-
#
|
| 19 |
source venv/Scripts/activate
|
| 20 |
|
| 21 |
-
#
|
| 22 |
-
|
| 23 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 24 |
python -m scrapers.myneta_scraper
|
| 25 |
-
python -m scrapers.mca_scraper
|
| 26 |
```
|
| 27 |
|
| 28 |
## Legal Note
|
| 29 |
-
All data
|
| 30 |
-
|
| 31 |
-
legal disclaimer.
|
| 32 |
-
EOF
|
|
|
|
|
|
|
| 1 |
# BharatGraph Scrapers
|
| 2 |
|
| 3 |
All scrapers collect **public data only**. No private data, no logins.
|
| 4 |
|
| 5 |
## Scrapers
|
| 6 |
|
| 7 |
+
| File | Source | Data Collected | Status |
|
| 8 |
+
|------|--------|----------------|--------|
|
| 9 |
+
| `base_scraper.py` | - | Base class, rate limiting, logging | β
|
|
| 10 |
+
| `datagov_scraper.py` | data.gov.in | MGNREGA, PM-KISAN datasets | β
|
|
| 11 |
+
| `pib_scraper.py` | pib.gov.in | Govt press releases, cabinet decisions | β
|
|
| 12 |
+
| `myneta_scraper.py` | myneta.info | Politician assets, criminal cases, education | β
|
|
| 13 |
+
| `mca_scraper.py` | MCA / data.gov.in | Company registrations, directors | β
|
|
| 14 |
+
| `cag_scraper.py` | cag.gov.in | Audit reports, financial irregularities | β
|
|
| 15 |
+
| `gem_scraper.py` | gem.gov.in | Govt contracts, procurement orders | β
|
|
| 16 |
|
| 17 |
+
## Known Issues
|
| 18 |
+
|
| 19 |
+
| Scraper | Issue | Fix |
|
| 20 |
+
|---------|-------|-----|
|
| 21 |
+
| `datagov_scraper.py` | 403 error with default key | Register at data.gov.in for free key |
|
| 22 |
+
| `mca_scraper.py` | 403 error | Same - needs personal API key in .env |
|
| 23 |
+
| `pib_scraper.py` | 0 articles | RSS URL under investigation |
|
| 24 |
+
|
| 25 |
+
## Setup
|
| 26 |
```bash
|
| 27 |
+
# 1. Activate venv
|
| 28 |
source venv/Scripts/activate
|
| 29 |
|
| 30 |
+
# 2. Copy env file and add your API key
|
| 31 |
+
cp .env.example .env
|
| 32 |
+
# Edit .env and add DATAGOV_API_KEY=your_key
|
| 33 |
+
|
| 34 |
+
# 3. Run any scraper
|
| 35 |
+
python -m scrapers.cag_scraper
|
| 36 |
+
python -m scrapers.gem_scraper
|
| 37 |
python -m scrapers.myneta_scraper
|
|
|
|
| 38 |
```
|
| 39 |
|
| 40 |
## Legal Note
|
| 41 |
+
All data from official government portals and public civic databases.
|
| 42 |
+
No private data collected. See project README for full legal disclaimer.
|
|
|
|
|
|