qshub / validation_set /README.md
basel00omar's picture
THE QS HUB v2.0 - Complete React + FastAPI Platform (clean deploy)
1b4b6d2
|
Raw History Blame Contribute Delete
2.21 kB
# Validation set — measure real drawing→BOQ accuracy
This harness references real villa projects **in place** (it never copies the large
PDFs), so the repo stays lean and runtime performance is unaffected.
## What qualifies as a validation project
A project folder must contain all three:
- architectural drawing β€” PDF whose name contains `arch`
- structural drawing β€” PDF whose name contains `str`/`struct`
- ground-truth BOQ β€” an Excel file that is **not** a system output
(`*_QTO_HYBRID_TEST.xlsx` etc. are ignored)
Incomplete folders are skipped automatically and listed under `incomplete_projects`.
## Run (cheap β€” no AI, no copying)
```powershell
# default root = B:\work\trainning (the 8 clean arch+str+BOQ pairs)
python validation_set/run_validation.py
# point at another folder
python validation_set/run_validation.py --root "B:\work\projects estimation\projects\..."
```
Output: `validation_set/manifest.json` β€” paths + each project's ground-truth BOQ
parsed into normalized engine keys (excavation, foundation_concrete, …) via the
existing `_extract_real_boqs.extract_from_workbook`.
`manifest.json` is git-ignored (machine-specific absolute paths; regenerate anytime).
## Current verified set (root = trainning)
All **11** projects (1–11) qualify β€” each has an architectural PDF + structural PDF
+ ground-truth BOQ (parsed to 23–29 items). The detector handles real-world naming
variants (`AR-…` for architectural, `STC`/`stc` for structural).
Note: 5 and 7 share the same BOQ file β€” treat as one if deduping.
## Next step β€” the accuracy sweep (heavy, opt-in)
`--run-engine` is the place to wire the full pass: load each project's two PDFs β†’
run the extraction + `build_boq_dataframe_from_project` β†’ diff computed vs ground-truth
per item β†’ emit a per-item accuracy report (% deviation, 🟒/🟑/πŸ”΄).
It is **not** run automatically because it calls the AI extractor (Gemini cost + time).
## Adding more projects
Point `--root` at any folder of clean `arch + str + BOQ` projects. Noisy folders with
40–100 PDFs (NOCs, soil reports, approvals) need the right two sheets isolated first β€”
the auto-detector picks one `arch` + one `str` PDF per folder.