abinazebinoy commited on
Commit
03a5dbe
·
1 Parent(s): 5657d91

fix(ci): C-01 -- daily scrape never loaded data into Neo4j

Browse files

The daily_scrape.yml workflow ran the pipeline (scraping + saving JSON
to data/processed/) and committed the files to git but had NO step that
actually imported the data into Neo4j. The live site always queried a
stale or empty database.

Fix: added a 'load pipeline output into Neo4j' step that POSTs to
/admin/pipeline after the scrape completes. This triggers the existing
background loader (SCRAPER_TO_LOADER dict in admin.py) which imports
all 20 datasets into Neo4j.

Files changed (1) hide show
  1. .github/workflows/daily_scrape.yml +14 -0
.github/workflows/daily_scrape.yml CHANGED
@@ -69,6 +69,20 @@ jobs:
69
  echo "Check pipeline status at /admin/pipeline/status"
70
  exit 0
71
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
72
  - name: commit updated data artifacts
73
  run: |
74
  git config user.name "github-actions[bot]"
 
69
  echo "Check pipeline status at /admin/pipeline/status"
70
  exit 0
71
 
72
+ - name: load pipeline output into Neo4j
73
+ env:
74
+ NEO4J_URI: ${{ secrets.NEO4J_URI }}
75
+ NEO4J_USER: ${{ secrets.NEO4J_USER }}
76
+ NEO4J_PASSWORD: ${{ secrets.NEO4J_PASSWORD }}
77
+ run: |
78
+ # C-01 FIX: pipeline was saving JSON but never importing into Neo4j.
79
+ # POST to /admin/pipeline triggers background load via the API.
80
+ API_BASE="${{ secrets.API_BASE_URL }}"
81
+ API_BASE="${API_BASE:-https://abinazebinoly-bharatgraph.hf.space}"
82
+ echo "Triggering graph load via API..."
83
+ curl -s -X POST "${API_BASE}/admin/pipeline" -H "Content-Type: application/json" --max-time 30 || echo "Warning: pipeline trigger returned non-200 (may still be running)"
84
+ echo "Graph load triggered. Check /admin/pipeline/status for progress."
85
+
86
  - name: commit updated data artifacts
87
  run: |
88
  git config user.name "github-actions[bot]"