Spaces:
Build error
Build error
Commit ·
03a5dbe
1
Parent(s): 5657d91
fix(ci): C-01 -- daily scrape never loaded data into Neo4j
Browse filesThe daily_scrape.yml workflow ran the pipeline (scraping + saving JSON
to data/processed/) and committed the files to git but had NO step that
actually imported the data into Neo4j. The live site always queried a
stale or empty database.
Fix: added a 'load pipeline output into Neo4j' step that POSTs to
/admin/pipeline after the scrape completes. This triggers the existing
background loader (SCRAPER_TO_LOADER dict in admin.py) which imports
all 20 datasets into Neo4j.
.github/workflows/daily_scrape.yml
CHANGED
|
@@ -69,6 +69,20 @@ jobs:
|
|
| 69 |
echo "Check pipeline status at /admin/pipeline/status"
|
| 70 |
exit 0
|
| 71 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 72 |
- name: commit updated data artifacts
|
| 73 |
run: |
|
| 74 |
git config user.name "github-actions[bot]"
|
|
|
|
| 69 |
echo "Check pipeline status at /admin/pipeline/status"
|
| 70 |
exit 0
|
| 71 |
|
| 72 |
+
- name: load pipeline output into Neo4j
|
| 73 |
+
env:
|
| 74 |
+
NEO4J_URI: ${{ secrets.NEO4J_URI }}
|
| 75 |
+
NEO4J_USER: ${{ secrets.NEO4J_USER }}
|
| 76 |
+
NEO4J_PASSWORD: ${{ secrets.NEO4J_PASSWORD }}
|
| 77 |
+
run: |
|
| 78 |
+
# C-01 FIX: pipeline was saving JSON but never importing into Neo4j.
|
| 79 |
+
# POST to /admin/pipeline triggers background load via the API.
|
| 80 |
+
API_BASE="${{ secrets.API_BASE_URL }}"
|
| 81 |
+
API_BASE="${API_BASE:-https://abinazebinoly-bharatgraph.hf.space}"
|
| 82 |
+
echo "Triggering graph load via API..."
|
| 83 |
+
curl -s -X POST "${API_BASE}/admin/pipeline" -H "Content-Type: application/json" --max-time 30 || echo "Warning: pipeline trigger returned non-200 (may still be running)"
|
| 84 |
+
echo "Graph load triggered. Check /admin/pipeline/status for progress."
|
| 85 |
+
|
| 86 |
- name: commit updated data artifacts
|
| 87 |
run: |
|
| 88 |
git config user.name "github-actions[bot]"
|