abinazebinoy commited on
Commit
2a9acf6
Β·
1 Parent(s): 0e9b6d0

docs(scrapers): update README with all 6 scrapers + known issues

Browse files

- Added CAG and GeM scrapers to table
- Documented known 403 issues with DataGov API
- Added setup instructions with .env API key note

Files changed (1) hide show
  1. scrapers/README.md +28 -18
scrapers/README.md CHANGED
@@ -1,32 +1,42 @@
1
- cat > scrapers/README.md << 'EOF'
2
  # BharatGraph Scrapers
3
 
4
  All scrapers collect **public data only**. No private data, no logins.
5
 
6
  ## Scrapers
7
 
8
- | File | Source | Data Collected |
9
- |------|--------|----------------|
10
- | `base_scraper.py` | - | Base class, rate limiting, logging |
11
- | `datagov_scraper.py` | data.gov.in | MGNREGA, PM-KISAN datasets |
12
- | `pib_scraper.py` | pib.gov.in | Govt press releases, cabinet decisions |
13
- | `myneta_scraper.py` | myneta.info | Politician assets, criminal cases, education |
14
- | `mca_scraper.py` | MCA / data.gov.in | Company registrations, directors |
 
 
15
 
16
- ## How to run
 
 
 
 
 
 
 
 
17
  ```bash
18
- # Always activate venv first
19
  source venv/Scripts/activate
20
 
21
- # Run any scraper
22
- python -m scrapers.datagov_scraper
23
- python -m scrapers.pib_scraper
 
 
 
 
24
  python -m scrapers.myneta_scraper
25
- python -m scrapers.mca_scraper
26
  ```
27
 
28
  ## Legal Note
29
- All data collected is from official government portals and public civic
30
- databases. No private data is collected. See project README for full
31
- legal disclaimer.
32
- EOF
 
 
1
  # BharatGraph Scrapers
2
 
3
  All scrapers collect **public data only**. No private data, no logins.
4
 
5
  ## Scrapers
6
 
7
+ | File | Source | Data Collected | Status |
8
+ |------|--------|----------------|--------|
9
+ | `base_scraper.py` | - | Base class, rate limiting, logging | βœ… |
10
+ | `datagov_scraper.py` | data.gov.in | MGNREGA, PM-KISAN datasets | βœ… |
11
+ | `pib_scraper.py` | pib.gov.in | Govt press releases, cabinet decisions | βœ… |
12
+ | `myneta_scraper.py` | myneta.info | Politician assets, criminal cases, education | βœ… |
13
+ | `mca_scraper.py` | MCA / data.gov.in | Company registrations, directors | βœ… |
14
+ | `cag_scraper.py` | cag.gov.in | Audit reports, financial irregularities | βœ… |
15
+ | `gem_scraper.py` | gem.gov.in | Govt contracts, procurement orders | βœ… |
16
 
17
+ ## Known Issues
18
+
19
+ | Scraper | Issue | Fix |
20
+ |---------|-------|-----|
21
+ | `datagov_scraper.py` | 403 error with default key | Register at data.gov.in for free key |
22
+ | `mca_scraper.py` | 403 error | Same - needs personal API key in .env |
23
+ | `pib_scraper.py` | 0 articles | RSS URL under investigation |
24
+
25
+ ## Setup
26
  ```bash
27
+ # 1. Activate venv
28
  source venv/Scripts/activate
29
 
30
+ # 2. Copy env file and add your API key
31
+ cp .env.example .env
32
+ # Edit .env and add DATAGOV_API_KEY=your_key
33
+
34
+ # 3. Run any scraper
35
+ python -m scrapers.cag_scraper
36
+ python -m scrapers.gem_scraper
37
  python -m scrapers.myneta_scraper
 
38
  ```
39
 
40
  ## Legal Note
41
+ All data from official government portals and public civic databases.
42
+ No private data collected. See project README for full legal disclaimer.