GeoCarto model v0.2 (research preview): segmentation + naming + GeoPackage export
Browse files- .gitattributes +3 -0
- README.md +101 -0
- best_fp16.pt +3 -0
- exemple/carte_geologique.gpkg +3 -0
- exemple/legende_exemple.csv +6 -0
- exemple/rendu_qgis.png +3 -0
- gc_model.py +127 -0
- leader.py +79 -0
- legend_ranker.pkl +3 -0
- naming_model.pkl +3 -0
- naming_model.py +203 -0
- pipeline.py +288 -0
- qgis_test.py +55 -0
- qml_faults.xml +17 -0
- qml_units.xml +41 -0
- requirements.txt +13 -0
- training/e2e_naming_9_zones.csv +10 -0
- training/history.csv +13 -0
- training/lab_round4_ablation.csv +4 -0
- training/metrics.json +257 -0
- training/test_examples.png +3 -0
- training/test_metrics.csv +15 -0
- training/train_final.py +471 -0
- vectorize.py +56 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
exemple/carte_geologique.gpkg filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
exemple/rendu_qgis.png filter=lfs diff=lfs merge=lfs -text
|
| 38 |
+
training/test_examples.png filter=lfs diff=lfs merge=lfs -text
|
README.md
ADDED
|
@@ -0,0 +1,101 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: cc-by-sa-4.0
|
| 3 |
+
library_name: pytorch
|
| 4 |
+
pipeline_tag: image-segmentation
|
| 5 |
+
datasets:
|
| 6 |
+
- PerfectWin/geocarto
|
| 7 |
+
tags:
|
| 8 |
+
- geology
|
| 9 |
+
- geological-map
|
| 10 |
+
- vectorization
|
| 11 |
+
- instance-segmentation
|
| 12 |
+
- gis
|
| 13 |
+
- qgis
|
| 14 |
+
---
|
| 15 |
+
|
| 16 |
+
# GeoCarto model – geological map raster → GIS polygons, faults and unit names (GeoPackage)
|
| 17 |
+
|
| 18 |
+
**Research preview (v0.2).** Turns a scanned / rasterised geological map into a GeoPackage that opens directly in
|
| 19 |
+
QGIS or ArcGIS: one polygon layer of geological units (with code, name, age, lithology when they can be read),
|
| 20 |
+
one line layer of faults, the legend table and default QGIS styles.
|
| 21 |
+
|
| 22 |
+
Trained on [PerfectWin/geocarto](https://huggingface.co/datasets/PerfectWin/geocarto) **v2.0** (4 continents,
|
| 23 |
+
real USGS GMNA scans + realistic synthetic maps).
|
| 24 |
+
|
| 25 |
+
## How it works
|
| 26 |
+
1. **Segmentation** – U-Net++ / EfficientNet-B4, input RGB + CIE-Lab (6 channels), predicts 10 pixel affinities
|
| 27 |
+
(offsets 1–27 px) + a fault map; sliding window 512 px, stride 384, Hann blending.
|
| 28 |
+
2. **Polygons** – mutex watershed on the affinities (full resolution), minimum region 40 px, vectorised with rasterio.
|
| 29 |
+
3. **Faults** – skeleton of the fault map, traced into lines.
|
| 30 |
+
4. **Text** – EasyOCR on the whole map.
|
| 31 |
+
5. **Naming** – names come **only from text printed on the map and from the legend** (no propagation of names between
|
| 32 |
+
polygons that merely look alike). A learned ranker (gradient boosting) scores every (polygon, candidate code) pair
|
| 33 |
+
from: OCR evidence and leader-line features, distance to the legend swatch (relative to the other swatches),
|
| 34 |
+
where else the code is read on the map. A name is written only when the confidence is ≥ 0.8; otherwise the polygon
|
| 35 |
+
is `Unité non identifiée` with the 3 best ranked suggestions in the `suggestions` column.
|
| 36 |
+
6. **Export** – GeoPackage with layers `unites_geologiques`, `failles`, table `legende`, styles in `layer_styles`.
|
| 37 |
+
|
| 38 |
+
## Usage
|
| 39 |
+
```bash
|
| 40 |
+
pip install -r requirements.txt
|
| 41 |
+
python vectorize.py carte.tif legende.csv sortie.gpkg
|
| 42 |
+
```
|
| 43 |
+
`carte.tif` must be georeferenced (any CRS). `legende.csv` has one row per legend entry: `code` (as printed on the
|
| 44 |
+
map), optional `nom`, `age`, `lithologie`, and either the pixel box of the legend swatch in the raster
|
| 45 |
+
(`x0,y0,x1,y1`, recommended) or its colour `couleur_hex`. See `exemple/legende_exemple.csv`.
|
| 46 |
+
`qgis_test.py` checks a produced GeoPackage with PyQGIS (layers, styles, geometry validity, rendering).
|
| 47 |
+
|
| 48 |
+
Attribute table of `unites_geologiques`: `id, code, nom, age, lithologie, source_nom` (code read on the map / legend
|
| 49 |
+
colour / not identified), `confiance, texte_lu, suggestions, couleur_hex` (legend colour), `couleur_carte` (colour
|
| 50 |
+
measured on the scan), `surface_km2`.
|
| 51 |
+
|
| 52 |
+
## Results (held-out test split, never seen in training, 608 tiles)
|
| 53 |
+
Recall = share of reference polygons recovered with IoU ≥ 0.5. *Small* < 1000 px, *large* ≥ 1000 px (512 px tiles).
|
| 54 |
+
*Correct area* = share of the map area that falls in a correctly recovered polygon.
|
| 55 |
+
|
| 56 |
+
| group | ARAND ↓ | small recall | large recall | correct area | fault F1 (2 px) |
|
| 57 |
+
|---|---|---|---|---|---|
|
| 58 |
+
| Africa (synthetic) | 0.030 | 89.6 % | 97.9 % | 99.3 % | – |
|
| 59 |
+
| Europe (synthetic) | 0.099 | 82.2 % | 90.4 % | 92.0 % | – |
|
| 60 |
+
| North America (all) | 0.091 | 72.5 % | 90.5 % | 95.1 % | 0.95 |
|
| 61 |
+
| Oceania (synthetic) | 0.176 | 63.9 % | 75.6 % | 83.5 % | 0.99 |
|
| 62 |
+
| **GMNA real scans** | 0.052 | **74.7 %** | **93.4 %** | **98.3 %** | 0.63 |
|
| 63 |
+
| **full test set** | 0.107 | 70.5 % | 86.2 % | 92.1 % | 0.97 |
|
| 64 |
+
| trap set: neighbouring units with the same colour | 0.151 | 62.8 % | 76.9 % | 84.9 % | 0.97 |
|
| 65 |
+
| trap set: patterned units | 0.099 | 74.3 % | 89.8 % | 93.9 % | 0.98 |
|
| 66 |
+
|
| 67 |
+
Full table: `training/test_metrics.csv`; ablation of the training choices: `training/lab_round4_ablation.csv`
|
| 68 |
+
(small-polygon loss re-weighting and OHEM were tested and rejected: they over-segment real maps).
|
| 69 |
+
|
| 70 |
+
### Naming (real GMNA scans, map sheet never used to train the ranker)
|
| 71 |
+
End-to-end test (segmentation by this model + OCR + naming) on 9 real 2048 × 2048 px windows of the GMNA north sheet;
|
| 72 |
+
legend = all units of the sheet, swatches measured outside the tested window; ranker trained on the south sheet only.
|
| 73 |
+
|
| 74 |
+
| | value |
|
| 75 |
+
|---|---|
|
| 76 |
+
| area that receives a name (confidence ≥ 0.8) | **30.0 %** (per window 4 % – 57 %) |
|
| 77 |
+
| name correct, on the named area | **91.6 %** (per window 76 % – 98 %) |
|
| 78 |
+
| area correctly named, over the whole map | 27.5 % |
|
| 79 |
+
|
| 80 |
+
Cross-sheet benchmark on reference polygons (38 windows, both directions): names given at confidence ≥ 0.8 are
|
| 81 |
+
94–96 % correct on ~1/3 of the area. The rest of the area is mostly units without a legible label in the window.
|
| 82 |
+
Per-window results: `training/e2e_naming_9_zones.csv`. Example output: `exemple/carte_geologique.gpkg`
|
| 83 |
+
(rendered in QGIS: `exemple/rendu_qgis.png`).
|
| 84 |
+
|
| 85 |
+
## Limitations
|
| 86 |
+
* Small units: about 1 in 4 small polygons on real scans is still missed or merged.
|
| 87 |
+
* Naming depends on labels being printed and legible; unlabelled units are left as *Unité non identifiée* with
|
| 88 |
+
suggestions rather than guessed. Text recognition is trained for Latin script codes.
|
| 89 |
+
* Real-scan validation is so far only on the USGS Geologic Map of North America; other continents are evaluated on
|
| 90 |
+
realistic synthetic renderings of official vector maps.
|
| 91 |
+
* Fault detection on real scans (F1 0.63) is much weaker than on synthetic maps.
|
| 92 |
+
* Large maps are processed in one piece for the watershed: keep inputs below ~6000 × 6000 px or split them.
|
| 93 |
+
|
| 94 |
+
## Files
|
| 95 |
+
`best_fp16.pt` weights (step 22 500) · `gc_model.py` network + mutex watershed · `pipeline.py` full pipeline ·
|
| 96 |
+
`naming_model.py`, `leader.py`, `naming_model.pkl` OCR-to-polygon model · `legend_ranker.pkl` naming ranker ·
|
| 97 |
+
`vectorize.py` command line · `qml_*.xml` QGIS styles · `training/` training script, curves and metrics.
|
| 98 |
+
The `.pkl` files are scikit-learn pickles: only load files from this repository.
|
| 99 |
+
|
| 100 |
+
## Citation
|
| 101 |
+
Please cite the dataset sources listed in the [dataset card](https://huggingface.co/datasets/PerfectWin/geocarto).
|
best_fp16.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:fe12b55b0b460bafe50628b32372c857b3d446b6a5b329a2ad0274b2b0bb5f54
|
| 3 |
+
size 42158475
|
exemple/carte_geologique.gpkg
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:5d49b7700611a18c8d02636eae0bdc6485eb661d5d625424eceae6b8ccc18a20
|
| 3 |
+
size 557056
|
exemple/legende_exemple.csv
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
code,nom,age,lithologie,couleur_hex
|
| 2 |
+
X²,X²,Early Proterozoic – Early Proterozoic,Sedimentary,#bda8c6
|
| 3 |
+
X²g,X²g,Early Proterozoic – Early Proterozoic,Plutonic undivided granitic rocks,#d96288
|
| 4 |
+
At,At,Archean – Archean,Plutonic tonalite,#efadcf
|
| 5 |
+
Agn,Agn,Archean – Archean,Metamorphic and undivided crystalline orthogneiss,#ebd0df
|
| 6 |
+
Wtn,Wtn,Late Archean – Late Archean,Metamorphic and undivided crystalline mainly tonalite gneiss,#aaddf1
|
exemple/rendu_qgis.png
ADDED
|
Git LFS Details
|
gc_model.py
ADDED
|
@@ -0,0 +1,127 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Définitions du modèle GeoCarto (extraites de train_final.py) pour l'inférence hors Kaggle.
|
| 2 |
+
import numpy as np, torch, torch.nn as nn
|
| 3 |
+
from scipy import ndimage as ndi
|
| 4 |
+
from numba import njit
|
| 5 |
+
import segmentation_models_pytorch as smp
|
| 6 |
+
|
| 7 |
+
CFG = dict(arch="unetpp", encoder="efficientnet-b4", input="rgb+lab",
|
| 8 |
+
offsets=[(1, 0), (0, 1), (3, 0), (0, 3), (9, 0), (0, 9), (9, 9), (9, -9), (27, 0), (0, 27)],
|
| 9 |
+
n_short=2, mws_stride=1, mws_min_size=40, tile=512)
|
| 10 |
+
OFFSETS, N_SHORT, K = CFG["offsets"], CFG["n_short"], len(CFG["offsets"])
|
| 11 |
+
|
| 12 |
+
# ------------------------------------------------------------------ affinités + mutex watershed
|
| 13 |
+
def fill_holes(inst):
|
| 14 |
+
"""Chaque zone sans polygone (mer, lac, halo) devient sa propre région."""
|
| 15 |
+
holes, n = ndi.label(inst == 0)
|
| 16 |
+
if n == 0:
|
| 17 |
+
return inst
|
| 18 |
+
out = inst.copy()
|
| 19 |
+
out[holes > 0] = holes[holes > 0] + inst.max()
|
| 20 |
+
return out
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
def affinity_targets(inst):
|
| 24 |
+
H, W = inst.shape
|
| 25 |
+
aff, mask = np.zeros((K, H, W), np.float32), np.zeros((K, H, W), np.float32)
|
| 26 |
+
for k, (dy, dx) in enumerate(OFFSETS):
|
| 27 |
+
ys, ye, xs, xe = max(0, -dy), H - max(0, dy), max(0, -dx), W - max(0, dx)
|
| 28 |
+
a, b = inst[ys:ye, xs:xe], inst[ys + dy:ye + dy, xs + dx:xe + dx]
|
| 29 |
+
aff[k, ys:ye, xs:xe] = a == b
|
| 30 |
+
mask[k, ys:ye, xs:xe] = 1 # après fill_holes, tous les pixels ont une région
|
| 31 |
+
return aff, mask
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
@njit
|
| 35 |
+
def _find(parent, x):
|
| 36 |
+
while parent[x] != x:
|
| 37 |
+
parent[x] = parent[parent[x]]
|
| 38 |
+
x = parent[x]
|
| 39 |
+
return x
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
@njit
|
| 43 |
+
def _mws_core(n, u, v, rep):
|
| 44 |
+
parent = np.arange(n); size = np.ones(n, np.int64); m = len(u)
|
| 45 |
+
mutex_a = np.empty(m, np.int64); mutex_b = np.empty(m, np.int64); nm = 0
|
| 46 |
+
head = -np.ones(n, np.int64); nxt = np.empty(2 * m, np.int64); val = np.empty(2 * m, np.int64); nl = 0
|
| 47 |
+
for e in range(m):
|
| 48 |
+
ra, rb = _find(parent, u[e]), _find(parent, v[e])
|
| 49 |
+
if ra == rb:
|
| 50 |
+
continue
|
| 51 |
+
if rep[e]:
|
| 52 |
+
val[nl] = nm; nxt[nl] = head[ra]; head[ra] = nl; nl += 1
|
| 53 |
+
val[nl] = nm; nxt[nl] = head[rb]; head[rb] = nl; nl += 1
|
| 54 |
+
mutex_a[nm] = ra; mutex_b[nm] = rb; nm += 1
|
| 55 |
+
else:
|
| 56 |
+
small, other = (ra, rb) if size[ra] <= size[rb] else (rb, ra)
|
| 57 |
+
blocked = False; p = head[small]
|
| 58 |
+
while p != -1:
|
| 59 |
+
k = val[p]; x = _find(parent, mutex_a[k]); y = _find(parent, mutex_b[k])
|
| 60 |
+
if (x == small and y == other) or (x == other and y == small):
|
| 61 |
+
blocked = True; break
|
| 62 |
+
p = nxt[p]
|
| 63 |
+
if blocked:
|
| 64 |
+
continue
|
| 65 |
+
parent[small] = other; size[other] += size[small]; p = head[small]
|
| 66 |
+
if p != -1:
|
| 67 |
+
while nxt[p] != -1:
|
| 68 |
+
p = nxt[p]
|
| 69 |
+
nxt[p] = head[other]; head[other] = head[small]
|
| 70 |
+
out = np.empty(n, np.int64)
|
| 71 |
+
for i in range(n):
|
| 72 |
+
out[i] = _find(parent, i)
|
| 73 |
+
return out
|
| 74 |
+
|
| 75 |
+
|
| 76 |
+
def mutex_watershed(aff, stride=CFG["mws_stride"], min_size=CFG["mws_min_size"]):
|
| 77 |
+
_, H, W = aff.shape
|
| 78 |
+
idx = np.arange(H * W).reshape(H, W)
|
| 79 |
+
us, vs, ws, rep = [], [], [], []
|
| 80 |
+
for k, (dy, dx) in enumerate(OFFSETS):
|
| 81 |
+
a = idx[max(0, -dy):H - max(0, dy), max(0, -dx):W - max(0, dx)]
|
| 82 |
+
b = idx[max(0, dy):H - max(0, -dy), max(0, dx):W - max(0, -dx)]
|
| 83 |
+
w = aff[k, max(0, -dy):H - max(0, dy), max(0, -dx):W - max(0, dx)].copy()
|
| 84 |
+
if k >= N_SHORT:
|
| 85 |
+
a, b, w = a[::stride, ::stride], b[::stride, ::stride], 1 - w[::stride, ::stride]
|
| 86 |
+
us.append(a.ravel()); vs.append(b.ravel()); ws.append(w.ravel()); rep.append(np.full(a.size, k >= N_SHORT))
|
| 87 |
+
u, v, w, rep = map(np.concatenate, (us, vs, ws, rep))
|
| 88 |
+
order = np.argsort(-w, kind="stable")
|
| 89 |
+
lab = _mws_core(H * W, u[order], v[order], rep[order]).reshape(H, W)
|
| 90 |
+
_, lab = np.unique(lab, return_inverse=True); lab = lab.reshape(H, W) + 1
|
| 91 |
+
sizes = np.bincount(lab.ravel()); tiny = sizes < min_size; tiny[0] = False
|
| 92 |
+
if tiny[lab].any():
|
| 93 |
+
_, (iy, ix) = ndi.distance_transform_edt(tiny[lab], return_indices=True)
|
| 94 |
+
lab = lab[iy, ix]; _, lab = np.unique(lab, return_inverse=True); lab = lab.reshape(H, W) + 1
|
| 95 |
+
return lab.astype(np.int32)
|
| 96 |
+
|
| 97 |
+
|
| 98 |
+
# ------------------------------------------------------------------ modèle
|
| 99 |
+
def rgb2lab(x):
|
| 100 |
+
"""x : B x 3 x H x W dans [0,1] -> Lab normalisé (L/50-1, a/64, b/64)."""
|
| 101 |
+
lin = torch.where(x <= 0.04045, x / 12.92, ((x + 0.055) / 1.055) ** 2.4)
|
| 102 |
+
M = torch.tensor([[0.4124, 0.3576, 0.1805], [0.2126, 0.7152, 0.0722], [0.0193, 0.1192, 0.9505]], device=x.device)
|
| 103 |
+
xyz = torch.einsum("ij,bjhw->bihw", M, lin) / torch.tensor([0.95047, 1.0, 1.08883], device=x.device).view(1, 3, 1, 1)
|
| 104 |
+
f = torch.where(xyz > 0.008856, xyz.clamp(min=1e-6) ** (1 / 3), 7.787 * xyz + 16 / 116)
|
| 105 |
+
L, a, b = 116 * f[:, 1] - 16, 500 * (f[:, 0] - f[:, 1]), 200 * (f[:, 1] - f[:, 2])
|
| 106 |
+
return torch.stack([L / 50 - 1, a / 64, b / 64], 1)
|
| 107 |
+
|
| 108 |
+
|
| 109 |
+
class GeoCartoNet(nn.Module):
|
| 110 |
+
"""Entrée : image uint8 B x 3 x H x W (H, W multiples de 32). Sortie : K affinités + 1 carte de failles (logits)."""
|
| 111 |
+
|
| 112 |
+
def __init__(self, cfg, pretrained=True):
|
| 113 |
+
super().__init__()
|
| 114 |
+
self.cfg = cfg
|
| 115 |
+
self.register_buffer("mean", torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1))
|
| 116 |
+
self.register_buffer("std", torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1))
|
| 117 |
+
Arch = {"unet": smp.Unet, "unetpp": smp.UnetPlusPlus}[cfg["arch"]]
|
| 118 |
+
self.net = Arch(cfg["encoder"], encoder_weights="imagenet" if pretrained else None, in_channels=6,
|
| 119 |
+
classes=len(cfg["offsets"]) + 1)
|
| 120 |
+
|
| 121 |
+
def forward(self, x8):
|
| 122 |
+
with torch.autocast(x8.device.type, enabled=False): # conversion de couleur en précision complète
|
| 123 |
+
x = x8.float() / 255
|
| 124 |
+
x = torch.cat([(x - self.mean) / self.std, rgb2lab(x)], 1)
|
| 125 |
+
return self.net(x)
|
| 126 |
+
|
| 127 |
+
|
leader.py
ADDED
|
@@ -0,0 +1,79 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Détection des traits de rappel : trait sombre fin qui part du bord d'une étiquette, dans n'importe quelle direction.
|
| 2 |
+
import numpy as np
|
| 3 |
+
from scipy import ndimage as ndi
|
| 4 |
+
from skimage.segmentation import find_boundaries
|
| 5 |
+
|
| 6 |
+
|
| 7 |
+
def prepare(reg):
|
| 8 |
+
"""Masque des pixels 'trait candidat' : sombres, hors contacts, hors failles. Calculé une fois par zone."""
|
| 9 |
+
if "lead_mask" in reg:
|
| 10 |
+
return
|
| 11 |
+
lum, inst = reg["lum"], reg["inst"]
|
| 12 |
+
bd = find_boundaries(inst, mode="thick")
|
| 13 |
+
nearbd = ndi.binary_dilation(bd, iterations=1)
|
| 14 |
+
flt = reg.get("flt")
|
| 15 |
+
nf = ndi.binary_dilation(flt > 0, iterations=3) if flt is not None else np.zeros_like(bd)
|
| 16 |
+
rgb = reg["rgb"].astype(np.int16)
|
| 17 |
+
sat = rgb.max(-1) - rgb.min(-1)
|
| 18 |
+
dark = (lum < 100) & (sat < 70) # trait noir/gris (pas de texte ou de contour coloré)
|
| 19 |
+
reg["lead_dark"] = dark
|
| 20 |
+
reg["lead_mask"] = dark & ~nearbd & ~nf
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
def trace(reg, wd, max_len=160, gap=6):
|
| 24 |
+
"""Renvoie la liste des extrémités (y, x, longueur) des traits partant de la boîte du mot."""
|
| 25 |
+
prepare(reg)
|
| 26 |
+
M, D, inst = reg["lead_mask"], reg["lead_dark"], reg["inst"]
|
| 27 |
+
N = inst.shape[0]
|
| 28 |
+
x0, y0, x1, y1 = int(max(wd.x0, 0)), int(max(wd.y0, 0)), int(min(wd.x1, N - 1)), int(min(wd.y1, N - 1))
|
| 29 |
+
P = 14
|
| 30 |
+
Y0, X0, Y1, X1 = max(y0 - P, 0), max(x0 - P, 0), min(y1 + P, N - 1), min(x1 + P, N - 1)
|
| 31 |
+
sub = M[Y0:Y1 + 1, X0:X1 + 1].copy()
|
| 32 |
+
# on enlève la boîte de texte (+1 px) : l'encre des lettres n'est pas un trait
|
| 33 |
+
sub[max(y0 - 1 - Y0, 0):y1 + 2 - Y0, max(x0 - 1 - X0, 0):x1 + 2 - X0] = False
|
| 34 |
+
lab, n = ndi.label(sub, structure=np.ones((3, 3)))
|
| 35 |
+
if n == 0:
|
| 36 |
+
return []
|
| 37 |
+
yy, xx = np.nonzero(lab)
|
| 38 |
+
ll = lab[yy, xx]
|
| 39 |
+
gy, gx = yy + Y0, xx + X0
|
| 40 |
+
# distance à la boîte
|
| 41 |
+
db = np.hypot(np.maximum(np.maximum(y0 - gy, gy - y1), 0), np.maximum(np.maximum(x0 - gx, gx - x1), 0))
|
| 42 |
+
out = []
|
| 43 |
+
for c in range(1, n + 1):
|
| 44 |
+
s = ll == c
|
| 45 |
+
if s.sum() < 5 or db[s].min() > 2.5: # doit toucher (presque) la boîte
|
| 46 |
+
continue
|
| 47 |
+
py, px, d = gy[s].astype(float), gx[s].astype(float), db[s]
|
| 48 |
+
# un trait est fin et allongé
|
| 49 |
+
cov = np.cov(np.stack([py, px]))
|
| 50 |
+
ev, evec = np.linalg.eigh(cov)
|
| 51 |
+
v = evec[:, 1]
|
| 52 |
+
if ev[1] < 9: # trop court (< ~10 px)
|
| 53 |
+
continue
|
| 54 |
+
res = np.abs((py - py.mean()) * v[1] - (px - px.mean()) * v[0])
|
| 55 |
+
if np.percentile(res, 90) > 1.3: # pas droit ou trop épais
|
| 56 |
+
continue
|
| 57 |
+
a = np.argmin(d); b = np.argmax(d)
|
| 58 |
+
if d[b] - d[a] < 4: # doit s'éloigner de la boîte
|
| 59 |
+
continue
|
| 60 |
+
if np.dot(v, [py[b] - py[a], px[b] - px[a]]) < 0:
|
| 61 |
+
v = -v
|
| 62 |
+
# suivi dans la direction, au-delà du morceau visible, en tolérant les coupures aux contacts
|
| 63 |
+
y, x = py[b], px[b]; miss = 0; L = float(np.hypot(py[b] - py[a], px[b] - px[a]))
|
| 64 |
+
ey, ex = y, x
|
| 65 |
+
while L < max_len:
|
| 66 |
+
y += v[0]; x += v[1]; L += 1
|
| 67 |
+
iy, ix = int(round(y)), int(round(x))
|
| 68 |
+
if not (0 <= iy < N and 0 <= ix < N):
|
| 69 |
+
break
|
| 70 |
+
if D[max(iy - 1, 0):iy + 2, max(ix - 1, 0):ix + 2].any():
|
| 71 |
+
miss = 0; ey, ex = y, x
|
| 72 |
+
else:
|
| 73 |
+
miss += 1
|
| 74 |
+
if miss > gap:
|
| 75 |
+
break
|
| 76 |
+
# pointe : on avance de 2 px au-delà de la fin (le trait se termine DANS le polygone visé)
|
| 77 |
+
ty, tx = int(np.clip(round(ey + 2 * v[0]), 0, N - 1)), int(np.clip(round(ex + 2 * v[1]), 0, N - 1))
|
| 78 |
+
out.append((ty, tx, float(np.hypot(ty - py[a], tx - px[a]))))
|
| 79 |
+
return out
|
legend_ranker.pkl
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0b99fd9736d2bd63a138a9dd12c7edfe2d46128e9b7454d6c1aeea7166be7a15
|
| 3 |
+
size 550205
|
naming_model.pkl
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:c9e2a5367d325a5b4455073597354f815453042a44c604ae82050a5b42ac5046
|
| 3 |
+
size 838928
|
naming_model.py
ADDED
|
@@ -0,0 +1,203 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# =====================================================================================
|
| 2 |
+
# MODULE DE NOMMAGE : à quel polygone appartient chaque code lu sur la carte ?
|
| 3 |
+
# Module séparé du modèle de segmentation. Pour chaque code lu (OCR), on note chaque polygone
|
| 4 |
+
# voisin avec des indices simples (distance, trait de rappel sombre, place disponible pour le
|
| 5 |
+
# texte, recouvrement...) et un petit modèle appris sur de VRAIES étiquettes GMNA choisit.
|
| 6 |
+
# Apprentissage : zones "south" ; test : zones "north" (jamais vues).
|
| 7 |
+
# =====================================================================================
|
| 8 |
+
import sys, glob, pickle, re, difflib, time
|
| 9 |
+
import numpy as np, pandas as pd
|
| 10 |
+
from scipy import ndimage as ndi
|
| 11 |
+
from sklearn.ensemble import HistGradientBoostingClassifier
|
| 12 |
+
|
| 13 |
+
RADIUS = 110 # rayon de recherche des polygones candidats autour d'une étiquette (px)
|
| 14 |
+
norm = lambda c: re.sub(r"[\*\?\s]", "", str(c))
|
| 15 |
+
import os
|
| 16 |
+
if os.path.exists("naming_cache/vocab.pkl"):
|
| 17 |
+
_codes = pickle.load(open("naming_cache/vocab.pkl", "rb"))
|
| 18 |
+
else: # lecture encore en cours : vocabulaire = codes des zones déjà lues
|
| 19 |
+
_codes = [c for f in glob.glob("naming_cache/*_*.pkl") for c in pickle.load(open(f, "rb"))["codes"]]
|
| 20 |
+
VOCAB = sorted({norm(c) for c in _codes} - {""})
|
| 21 |
+
VSET = set(VOCAB)
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
def to_code(txt):
|
| 25 |
+
t = re.sub(r"[^A-Za-z0-9€Є³]", "", txt).replace("€", "Є")
|
| 26 |
+
if not t:
|
| 27 |
+
return None, 0.0
|
| 28 |
+
if t in VSET:
|
| 29 |
+
return t, 1.0
|
| 30 |
+
m = difflib.get_close_matches(t, VOCAB, n=1, cutoff=0.75)
|
| 31 |
+
return (m[0], 0.75) if m else (None, 0.0)
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
FEATS = ["dist_centre", "dist_boite", "sous_centre", "part_boite", "rang_dist", "n_cand", "aire_log", "rayon_int",
|
| 35 |
+
"rayon_sur_texte", "trait_sombre", "trait_lum", "limites_croisees", "long_trait", "h_texte", "l_texte",
|
| 36 |
+
"conf", "qualite_code", "dist_rel", "plus_proche_non_sous", "part_boite_max_autre", "vise_trait", "trait_L"]
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
def candidates(reg, wd):
|
| 40 |
+
"""Lignes de caractéristiques (une par polygone candidat) pour un mot lu."""
|
| 41 |
+
inst, lum = reg["inst"], reg["lum"]
|
| 42 |
+
N = inst.shape[0]
|
| 43 |
+
x0, y0, x1, y1 = int(max(wd.x0, 0)), int(max(wd.y0, 0)), int(min(wd.x1, N - 1)), int(min(wd.y1, N - 1))
|
| 44 |
+
if x1 - x0 < 2 or y1 - y0 < 2:
|
| 45 |
+
return []
|
| 46 |
+
cx, cy = (x0 + x1) / 2, (y0 + y1) / 2
|
| 47 |
+
X0, Y0, X1, Y1 = max(x0 - RADIUS, 0), max(y0 - RADIUS, 0), min(x1 + RADIUS, N - 1), min(y1 + RADIUS, N - 1)
|
| 48 |
+
sub, slum = inst[Y0:Y1 + 1, X0:X1 + 1], lum[Y0:Y1 + 1, X0:X1 + 1]
|
| 49 |
+
yy, xx = np.mgrid[Y0:Y1 + 1, X0:X1 + 1]
|
| 50 |
+
dc = np.hypot(yy - cy, xx - cx) # distance au centre de la boîte
|
| 51 |
+
db = np.hypot(np.maximum(np.maximum(y0 - yy, yy - y1), 0), np.maximum(np.maximum(x0 - xx, xx - x1), 0)) # à la boîte
|
| 52 |
+
ids = np.unique(sub); ids = ids[ids > 0]
|
| 53 |
+
if not len(ids):
|
| 54 |
+
return []
|
| 55 |
+
box = sub[y0 - Y0:y1 - Y0 + 1, x0 - X0:x1 - X0 + 1]
|
| 56 |
+
light = slum[y0 - Y0:y1 - Y0 + 1, x0 - X0:x1 - X0 + 1] >= 120 # hors encre du texte
|
| 57 |
+
nb = max(light.sum(), 1)
|
| 58 |
+
under = inst[int(cy), int(cx)]
|
| 59 |
+
th, tw = y1 - y0, x1 - x0
|
| 60 |
+
dmin_c = ndi.minimum(dc, sub, ids); dmin_b = ndi.minimum(db, sub, ids)
|
| 61 |
+
pos_b = ndi.minimum_position(db, sub, ids)
|
| 62 |
+
edt = ndi.distance_transform_edt(sub > 0) if False else None
|
| 63 |
+
from leader import trace
|
| 64 |
+
tips = trace(reg, wd)
|
| 65 |
+
rows = []
|
| 66 |
+
order = np.argsort(dmin_b)
|
| 67 |
+
rank = np.empty(len(ids)); rank[order] = np.arange(len(ids))
|
| 68 |
+
part = np.array([(box[light] == i).sum() / nb for i in ids])
|
| 69 |
+
for j, i in enumerate(ids):
|
| 70 |
+
m = sub == i
|
| 71 |
+
# rayon intérieur du polygone (dans la fenêtre) : peut-il contenir le texte ?
|
| 72 |
+
rin = ndi.distance_transform_edt(np.pad(m, 1))[1:-1, 1:-1].max()
|
| 73 |
+
# segment droit boîte -> point le plus proche du polygone : un trait de rappel est sombre et droit
|
| 74 |
+
py, px = pos_b[j][0] + Y0, pos_b[j][1] + X0
|
| 75 |
+
qy, qx = min(max(py, y0), y1), min(max(px, x0), x1)
|
| 76 |
+
L = float(np.hypot(py - qy, px - qx))
|
| 77 |
+
if L >= 3:
|
| 78 |
+
n = int(L) + 1
|
| 79 |
+
sy = np.clip(np.round(np.linspace(qy, py, n)).astype(int), 0, N - 1)
|
| 80 |
+
sx = np.clip(np.round(np.linspace(qx, px, n)).astype(int), 0, N - 1)
|
| 81 |
+
ls = np.minimum.reduce([lum[np.clip(sy + a, 0, N - 1), np.clip(sx + b, 0, N - 1)] for a in (-1, 0, 1) for b in (-1, 0, 1)])
|
| 82 |
+
dark, lmean = float((ls[1:-1] < 120).mean()) if n > 2 else 0.0, float(ls.mean())
|
| 83 |
+
cross = int((np.diff(inst[sy, sx]) != 0).sum())
|
| 84 |
+
else:
|
| 85 |
+
dark, lmean, cross = 0.0, 255.0, 0
|
| 86 |
+
others = np.delete(part, j)
|
| 87 |
+
tl = [t[2] for t in tips if inst[t[0], t[1]] == i]
|
| 88 |
+
rows.append(dict(poly=int(i), dist_centre=float(dmin_c[j]), dist_boite=float(dmin_b[j]), sous_centre=float(i == under),
|
| 89 |
+
part_boite=float(part[j]), rang_dist=float(rank[j]), n_cand=float(len(ids)),
|
| 90 |
+
aire_log=float(np.log1p(reg["area"][i])), rayon_int=float(rin), rayon_sur_texte=float(rin / max(th / 2, 1)),
|
| 91 |
+
trait_sombre=dark, trait_lum=lmean, limites_croisees=float(cross), long_trait=L, h_texte=float(th),
|
| 92 |
+
l_texte=float(tw), conf=float(wd.conf), qualite_code=float(wd.q),
|
| 93 |
+
dist_rel=float(dmin_b[j] / max(th, 1)), plus_proche_non_sous=float(rank[j] == (1 if under in ids else 0) and i != under),
|
| 94 |
+
part_boite_max_autre=float(others.max() if len(others) else 0),
|
| 95 |
+
vise_trait=float(len(tl)), trait_L=float(min(tl)) if tl else -1.0))
|
| 96 |
+
return rows
|
| 97 |
+
|
| 98 |
+
|
| 99 |
+
def load_region(f):
|
| 100 |
+
r = pickle.load(open(f, "rb"))
|
| 101 |
+
r["lum"] = (r["rgb"].astype(np.float32) @ np.array([0.299, 0.587, 0.114], np.float32))
|
| 102 |
+
r["area"] = np.bincount(r["inst"].ravel())
|
| 103 |
+
r["true"] = {i + 1: norm(c) for i, c in enumerate(r["codes"])}
|
| 104 |
+
return r
|
| 105 |
+
|
| 106 |
+
|
| 107 |
+
def build(files):
|
| 108 |
+
out = []
|
| 109 |
+
for f in files:
|
| 110 |
+
reg = load_region(f)
|
| 111 |
+
W_ = reg["words"]
|
| 112 |
+
for k, wd in enumerate(W_.itertuples()):
|
| 113 |
+
code, q = to_code(wd.text)
|
| 114 |
+
if code is None:
|
| 115 |
+
continue
|
| 116 |
+
wd = wd._replace(conf=wd.conf) if False else wd
|
| 117 |
+
class _W: pass
|
| 118 |
+
w = _W(); w.__dict__.update(wd._asdict()); w.q = q
|
| 119 |
+
for row in candidates(reg, w):
|
| 120 |
+
row.update(region=f, word=k, code=code, y=float(reg["true"][row["poly"]] == code),
|
| 121 |
+
vrai=reg["true"][row["poly"]], aire=float(reg["area"][row["poly"]]))
|
| 122 |
+
out.append(row)
|
| 123 |
+
return pd.DataFrame(out)
|
| 124 |
+
|
| 125 |
+
|
| 126 |
+
def assign(D, score):
|
| 127 |
+
"""Choix final : pour chaque mot le polygone le mieux noté ; un polygone ne garde qu'UN code
|
| 128 |
+
(celui qui a la meilleure note), les mots perdants passent à leur candidat suivant."""
|
| 129 |
+
D = D.assign(s=score)
|
| 130 |
+
choice = {}
|
| 131 |
+
words = {k: g.sort_values("s", ascending=False) for k, g in D.groupby(["region", "word"])}
|
| 132 |
+
ptr = {k: 0 for k in words}
|
| 133 |
+
for _ in range(4):
|
| 134 |
+
for k, g in words.items():
|
| 135 |
+
if ptr[k] < len(g):
|
| 136 |
+
r = g.iloc[ptr[k]]
|
| 137 |
+
choice[k] = (r.region, int(r.poly), r.code, float(r.s))
|
| 138 |
+
else:
|
| 139 |
+
choice.pop(k, None)
|
| 140 |
+
best = {}
|
| 141 |
+
for k, (reg, p, c, s) in choice.items():
|
| 142 |
+
key = (reg, p)
|
| 143 |
+
if key not in best or s > best[key][1]:
|
| 144 |
+
best[key] = (c, s)
|
| 145 |
+
moved = False
|
| 146 |
+
for k, (reg, p, c, s) in list(choice.items()):
|
| 147 |
+
if best[(reg, p)][0] != c and ptr[k] + 1 < len(words[k]) and words[k].iloc[ptr[k] + 1].s > 0.15:
|
| 148 |
+
ptr[k] += 1; moved = True
|
| 149 |
+
if not moved:
|
| 150 |
+
break
|
| 151 |
+
return choice
|
| 152 |
+
|
| 153 |
+
|
| 154 |
+
def report(D, choice, title):
|
| 155 |
+
key = D.set_index(["region", "word", "poly"])
|
| 156 |
+
evaluable = D.groupby(["region", "word"]).y.max() > 0 # le bon code existe parmi les candidats (lecture juste)
|
| 157 |
+
ok = n = 0
|
| 158 |
+
polys = {}
|
| 159 |
+
for k, (reg, p, c, s) in choice.items():
|
| 160 |
+
r = key.loc[(k[0], k[1], p)]
|
| 161 |
+
good = bool(np.atleast_1d(r.y)[0])
|
| 162 |
+
if evaluable[k]:
|
| 163 |
+
n += 1; ok += good
|
| 164 |
+
cur = polys.get((reg, p))
|
| 165 |
+
if cur is None or s > cur[1]:
|
| 166 |
+
polys[(reg, p)] = (good, s, float(np.atleast_1d(r.aire)[0]))
|
| 167 |
+
pg = np.array([v[0] for v in polys.values()]); pa = np.array([v[2] for v in polys.values()])
|
| 168 |
+
print(f"{title:<46} codes bien rattachés : {ok / max(n, 1):5.1%} ({n} codes) | polygones nommés : {len(polys)}, "
|
| 169 |
+
f"noms justes : {pg.mean():5.1%} (en surface : {(pa * pg).sum() / pa.sum():5.1%})")
|
| 170 |
+
return ok / max(n, 1), polys
|
| 171 |
+
|
| 172 |
+
|
| 173 |
+
if __name__ == "__main__":
|
| 174 |
+
t0 = time.time()
|
| 175 |
+
tr_files, te_files = sorted(glob.glob("naming_cache/south_*.pkl")), sorted(glob.glob("naming_cache/north_*.pkl"))
|
| 176 |
+
if len(te_files) < 3: # provisoire : pas encore de zones nord -> 1 zone sud sur 3 en test
|
| 177 |
+
allf = tr_files + te_files
|
| 178 |
+
tr_files, te_files = [f for i, f in enumerate(allf) if i % 3], [f for i, f in enumerate(allf) if i % 3 == 0]
|
| 179 |
+
Dtr, Dte = build(tr_files), build(te_files)
|
| 180 |
+
print(f"apprentissage : {len(tr_files)} zones, {Dtr.groupby(['region', 'word']).ngroups} codes lus, {len(Dtr)} paires | "
|
| 181 |
+
f"test : {len(te_files)} zones, {Dte.groupby(['region', 'word']).ngroups} codes lus ({time.time() - t0:.0f} s)")
|
| 182 |
+
ev = Dtr.groupby(["region", "word"]).y.transform("max") > 0 # on n'apprend que sur les mots correctement lus
|
| 183 |
+
clf = HistGradientBoostingClassifier(max_iter=300, learning_rate=0.06, max_leaf_nodes=24, l2_regularization=1.0,
|
| 184 |
+
class_weight="balanced", random_state=0)
|
| 185 |
+
clf.fit(Dtr.loc[ev, FEATS], Dtr.loc[ev, "y"])
|
| 186 |
+
pickle.dump({"model": clf, "features": FEATS, "radius": RADIUS}, open("naming_model.pkl", "wb"))
|
| 187 |
+
print("\nSUR LES ZONES DE TEST (north, jamais vues) :")
|
| 188 |
+
naive = Dte.sous_centre + 0.001 * Dte.part_boite
|
| 189 |
+
a0, _ = report(Dte, {k: (g.region.iloc[0], int(g.poly.iloc[np.argmax(n_)]), g.code.iloc[0], 1.0)
|
| 190 |
+
for (k, g), n_ in ((kg, naive.loc[kg[1].index].values) for kg in Dte.groupby(["region", "word"]))},
|
| 191 |
+
"Règle naïve (polygone sous l'étiquette)")
|
| 192 |
+
p = clf.predict_proba(Dte[FEATS])[:, 1]
|
| 193 |
+
a1, _ = report(Dte, {k: (g.region.iloc[0], int(g.poly.iloc[np.argmax(p[Dte.index.get_indexer(g.index)])]), g.code.iloc[0],
|
| 194 |
+
float(p[Dte.index.get_indexer(g.index)].max())) for k, g in Dte.groupby(["region", "word"])},
|
| 195 |
+
"Module appris (meilleure note)")
|
| 196 |
+
ch = assign(Dte, p)
|
| 197 |
+
a2, polys = report(Dte, ch, "Module appris + un seul code par polygone")
|
| 198 |
+
# avec seuil de confiance : on ne nomme que si la note est élevée
|
| 199 |
+
for thr in (0.5, 0.7, 0.85):
|
| 200 |
+
report(Dte, {k: v for k, v in ch.items() if v[3] >= thr}, f" ... en ne gardant que les notes >= {thr}")
|
| 201 |
+
from sklearn.inspection import permutation_importance
|
| 202 |
+
imp = permutation_importance(clf, Dte[FEATS], Dte["y"], n_repeats=3, random_state=0, scoring="average_precision")
|
| 203 |
+
print("\nIndices les plus utiles :", ", ".join(f"{FEATS[i]} ({imp.importances_mean[i]:.3f})" for i in np.argsort(-imp.importances_mean)[:8]))
|
pipeline.py
ADDED
|
@@ -0,0 +1,288 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# =====================================================================================
|
| 2 |
+
# GEOCARTO – PIPELINE COMPLET : raster de carte géologique -> GeoPackage QGIS / ArcGIS
|
| 3 |
+
# 1. modèle (affinités + failles) par fenêtres glissantes 2. mutex watershed -> polygones
|
| 4 |
+
# 3. vectorisation (unités + failles) 4. OCR des textes de la carte
|
| 5 |
+
# 5. nommage : code écrit (module appris) > case de légende (si sûr) > "Unité non identifiée" + 3 suggestions
|
| 6 |
+
# 6. export GeoPackage (couches unites_geologiques, failles, table legende + styles QGIS par défaut)
|
| 7 |
+
# =====================================================================================
|
| 8 |
+
import os, sys, re, sqlite3, pickle, time
|
| 9 |
+
import numpy as np, pandas as pd, torch, geopandas as gpd
|
| 10 |
+
from scipy import ndimage as ndi
|
| 11 |
+
from rasterio.features import shapes
|
| 12 |
+
from rasterio.transform import Affine
|
| 13 |
+
from shapely.geometry import shape, LineString
|
| 14 |
+
from skimage.color import rgb2lab as sk_rgb2lab
|
| 15 |
+
from skimage.morphology import skeletonize
|
| 16 |
+
|
| 17 |
+
HERE = os.path.dirname(os.path.abspath(__file__))
|
| 18 |
+
sys.path.insert(0, HERE)
|
| 19 |
+
import gc_model as M
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
# ------------------------------------------------------------------ 1. modèle
|
| 23 |
+
def load_model(path):
|
| 24 |
+
ck = torch.load(path, map_location="cpu", weights_only=True)
|
| 25 |
+
cfg = ck["cfg"]
|
| 26 |
+
M.CFG.update(cfg); M.OFFSETS[:] = cfg["offsets"]
|
| 27 |
+
model = M.GeoCartoNet(cfg, pretrained=False)
|
| 28 |
+
model.load_state_dict({k: v.float() for k, v in ck["state_dict"].items()})
|
| 29 |
+
return model.eval()
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
@torch.no_grad()
|
| 33 |
+
def _run(model, im):
|
| 34 |
+
x = torch.from_numpy(np.ascontiguousarray(im).transpose(2, 0, 1).copy())[None]
|
| 35 |
+
return torch.sigmoid(model(x).float())[0].numpy()
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
def _tta(model, im, offsets):
|
| 39 |
+
"""Moyenne sur 4 miroirs ; chaque décalage est ramené dans le repère d'origine (testé : exact)."""
|
| 40 |
+
H, W = im.shape[:2]
|
| 41 |
+
Y, X = np.mgrid[:H, :W]
|
| 42 |
+
idx = {tuple(o): k for k, o in enumerate(offsets)}
|
| 43 |
+
K = len(offsets)
|
| 44 |
+
acc = 0
|
| 45 |
+
for fy in (0, 1):
|
| 46 |
+
for fx in (0, 1):
|
| 47 |
+
v = im[::-1] if fy else im
|
| 48 |
+
v = v[:, ::-1] if fx else v
|
| 49 |
+
o = _run(model, v)
|
| 50 |
+
A, pf = o[:K], o[K]
|
| 51 |
+
py, px = (H - 1 - Y) if fy else Y, (W - 1 - X) if fx else X
|
| 52 |
+
out = np.empty_like(o)
|
| 53 |
+
for k, (dy, dx) in enumerate(offsets):
|
| 54 |
+
t = (-dy if fy else dy, -dx if fx else dx)
|
| 55 |
+
if t in idx:
|
| 56 |
+
out[k] = A[idx[t]][py, px]
|
| 57 |
+
else:
|
| 58 |
+
out[k] = A[idx[(-t[0], -t[1])]][np.clip(py + t[0], 0, H - 1), np.clip(px + t[1], 0, W - 1)]
|
| 59 |
+
out[K] = pf[py, px]
|
| 60 |
+
acc = acc + out
|
| 61 |
+
return acc / 4
|
| 62 |
+
|
| 63 |
+
|
| 64 |
+
def infer(model, rgb, tile=512, stride=384, tta=False):
|
| 65 |
+
"""Fenêtres glissantes avec pondération de Hann -> affinités (K x H x W) + probabilité de faille."""
|
| 66 |
+
H, W = rgb.shape[:2]
|
| 67 |
+
ph, pw = max(tile - H, 0), max(tile - W, 0)
|
| 68 |
+
img = np.pad(rgb, ((0, ph), (0, pw), (0, 0)), mode="edge")
|
| 69 |
+
Hp, Wp = img.shape[:2]
|
| 70 |
+
K = len(M.OFFSETS)
|
| 71 |
+
acc = np.zeros((K + 1, Hp, Wp), np.float32); ws = np.zeros((Hp, Wp), np.float32)
|
| 72 |
+
hw = np.maximum(np.outer(np.hanning(tile), np.hanning(tile)), 1e-3).astype(np.float32)
|
| 73 |
+
pos = lambda n: sorted(set(list(range(0, n - tile + 1, stride)) + [n - tile]))
|
| 74 |
+
for y in pos(Hp):
|
| 75 |
+
for x in pos(Wp):
|
| 76 |
+
im = img[y:y + tile, x:x + tile]
|
| 77 |
+
o = _tta(model, im, M.OFFSETS) if tta else _run(model, im)
|
| 78 |
+
acc[:, y:y + tile, x:x + tile] += o * hw; ws[y:y + tile, x:x + tile] += hw
|
| 79 |
+
o = acc / ws
|
| 80 |
+
return o[:K, :H, :W], o[K, :H, :W]
|
| 81 |
+
|
| 82 |
+
|
| 83 |
+
# ------------------------------------------------------------------ 3. vectorisation
|
| 84 |
+
def vectorize_units(lab, transform, crs):
|
| 85 |
+
geoms, ids = [], []
|
| 86 |
+
for g, v in shapes(lab.astype(np.int32), mask=lab > 0, transform=transform):
|
| 87 |
+
geoms.append(shape(g)); ids.append(int(v))
|
| 88 |
+
P = gpd.GeoDataFrame({"pid": ids}, geometry=geoms, crs=crs).dissolve("pid").reset_index()
|
| 89 |
+
P["geometry"] = P.geometry.simplify(abs(transform.a) * 0.75, preserve_topology=True)
|
| 90 |
+
return P
|
| 91 |
+
|
| 92 |
+
|
| 93 |
+
def vectorize_faults(pf, transform, crs, thr=0.5, min_px=15):
|
| 94 |
+
sk = skeletonize(pf > thr)
|
| 95 |
+
lab, n = ndi.label(sk, structure=np.ones((3, 3)))
|
| 96 |
+
sizes = np.bincount(lab.ravel())
|
| 97 |
+
nb = ndi.convolve(sk.astype(np.uint8), np.ones((3, 3), np.uint8), mode="constant") - 1
|
| 98 |
+
lines = []
|
| 99 |
+
seen = np.zeros_like(sk)
|
| 100 |
+
for c in range(1, n + 1):
|
| 101 |
+
if sizes[c] < min_px:
|
| 102 |
+
continue
|
| 103 |
+
ys, xs = np.nonzero(lab == c)
|
| 104 |
+
pts = set(zip(ys.tolist(), xs.tolist()))
|
| 105 |
+
ends = [p for p in pts if nb[p] == 1] or [next(iter(pts))]
|
| 106 |
+
for st in ends:
|
| 107 |
+
if seen[st]:
|
| 108 |
+
continue
|
| 109 |
+
path, cur = [st], st
|
| 110 |
+
seen[st] = True
|
| 111 |
+
while True:
|
| 112 |
+
nxt = [(cur[0] + a, cur[1] + b) for a in (-1, 0, 1) for b in (-1, 0, 1) if (a or b)
|
| 113 |
+
and (cur[0] + a, cur[1] + b) in pts and not seen[cur[0] + a, cur[1] + b]]
|
| 114 |
+
if not nxt:
|
| 115 |
+
break
|
| 116 |
+
cur = nxt[0]; seen[cur] = True; path.append(cur)
|
| 117 |
+
if len(path) >= min_px:
|
| 118 |
+
lines.append(LineString([transform * (x + 0.5, y + 0.5) for y, x in path]).simplify(abs(transform.a)))
|
| 119 |
+
F = gpd.GeoDataFrame({"type": ["indéterminée"] * len(lines)}, geometry=lines, crs=crs)
|
| 120 |
+
F["longueur_km"] = (F.length / 1000).round(2)
|
| 121 |
+
return F
|
| 122 |
+
|
| 123 |
+
|
| 124 |
+
# ------------------------------------------------------------------ 4. OCR
|
| 125 |
+
def read_text(rgb, reader, B=1024, OV=96):
|
| 126 |
+
words = []
|
| 127 |
+
H, W = rgb.shape[:2]
|
| 128 |
+
for y in range(0, H, B - OV):
|
| 129 |
+
for x in range(0, W, B - OV):
|
| 130 |
+
for box, txt, conf in reader.readtext(rgb[y:y + B, x:x + B], mag_ratio=2.0, text_threshold=0.6, low_text=0.3):
|
| 131 |
+
b = np.array(box) + [x, y]; cx, cy = b.mean(0)
|
| 132 |
+
if (x > 0 and cx < x + OV / 2) or (y > 0 and cy < y + OV / 2):
|
| 133 |
+
continue
|
| 134 |
+
words.append(dict(text=txt, conf=conf, cx=cx, cy=cy, x0=b[:, 0].min(), x1=b[:, 0].max(), y0=b[:, 1].min(), y1=b[:, 1].max()))
|
| 135 |
+
return pd.DataFrame(words, columns=["text", "conf", "cx", "cy", "x0", "x1", "y0", "y1"])
|
| 136 |
+
|
| 137 |
+
|
| 138 |
+
# ------------------------------------------------------------------ 5. nommage
|
| 139 |
+
def swatch_features(lab_pixels):
|
| 140 |
+
"""Signature d'une case de légende / d'un polygone : couleur dominante, couleurs claire et foncée du motif, texture."""
|
| 141 |
+
px = lab_pixels[lab_pixels[:, 0] > 45]
|
| 142 |
+
if len(px) < 20:
|
| 143 |
+
px = lab_pixels
|
| 144 |
+
L = px[:, 0]; med = np.percentile(L, 50)
|
| 145 |
+
return np.concatenate([np.median(px, 0), np.median(px[L >= med], 0), np.median(px[L <= med], 0), px.std(0) / 4])
|
| 146 |
+
|
| 147 |
+
|
| 148 |
+
def ocr_candidates(rgb, lab, words, vocab, naming_pkl, flt=None):
|
| 149 |
+
"""Toutes les paires (code lu, polygone candidat) avec leur score appris ; et l'affectation un code -> un polygone."""
|
| 150 |
+
import naming_model as NM
|
| 151 |
+
NM.VOCAB = sorted(vocab); NM.VSET = set(NM.VOCAB)
|
| 152 |
+
mdl = pickle.load(open(naming_pkl, "rb"))
|
| 153 |
+
reg = dict(inst=lab, rgb=rgb, lum=rgb.astype(np.float32) @ np.array([0.299, 0.587, 0.114], np.float32),
|
| 154 |
+
area=np.bincount(lab.ravel()), flt=flt)
|
| 155 |
+
rows = []
|
| 156 |
+
for k, wd in enumerate(words.itertuples()):
|
| 157 |
+
code, q = NM.to_code(wd.text)
|
| 158 |
+
if code is None:
|
| 159 |
+
continue
|
| 160 |
+
class _W: pass
|
| 161 |
+
w = _W(); w.__dict__.update(wd._asdict()); w.q = q
|
| 162 |
+
for r in NM.candidates(reg, w):
|
| 163 |
+
r.update(region="carte", word=k, code=code, lu=wd.text); rows.append(r)
|
| 164 |
+
if not rows:
|
| 165 |
+
return pd.DataFrame(columns=["word", "code", "s"]), {}
|
| 166 |
+
D = pd.DataFrame(rows)
|
| 167 |
+
D["s"] = mdl["model"].predict_proba(D[mdl["features"]])[:, 1]
|
| 168 |
+
return D, NM.assign(D, D["s"].values)
|
| 169 |
+
|
| 170 |
+
RANK_FEATS = ["d_norm", "d_rel", "rang", "gap12", "d_ratio", "ocr", "ocr_assigne", "ocr_rel", "lu", "n_lu", "dist_lu", "log_aire"]
|
| 171 |
+
|
| 172 |
+
def rank_pairs(pids, feats, areas, cents, codes, Fm, D, ch, words, K=12):
|
| 173 |
+
"""Paires (polygone, code candidat) et leurs indices. Candidats = K cases de légende les plus proches + codes lus par l'OCR.
|
| 174 |
+
Indices : uniquement le texte lu sur la carte et la légende (pas de propagation d'apparence entre polygones)."""
|
| 175 |
+
cidx = {c: i for i, c in enumerate(codes)}
|
| 176 |
+
d = np.linalg.norm(feats[:, None, :] - Fm[None], axis=2)
|
| 177 |
+
E, lu_pos, asg = {}, {}, {}
|
| 178 |
+
if len(D):
|
| 179 |
+
for (p, c), s in D.groupby(["poly", "code"]).s.max().items():
|
| 180 |
+
E[(p, c)] = s
|
| 181 |
+
for w, c in D[["word", "code"]].drop_duplicates().values:
|
| 182 |
+
lu_pos.setdefault(c, []).append((words.iloc[w].cy, words.iloc[w].cx))
|
| 183 |
+
for (_, w), (_, p, c, s) in ch.items():
|
| 184 |
+
asg[(p, c)] = max(asg.get((p, c), 0), s)
|
| 185 |
+
Ep = {}
|
| 186 |
+
for (p, c), s in E.items():
|
| 187 |
+
Ep.setdefault(p, []).append(c)
|
| 188 |
+
lu_pos = {c: np.array(v) for c, v in lu_pos.items()}
|
| 189 |
+
rows = []
|
| 190 |
+
for i, p in enumerate(pids):
|
| 191 |
+
o = np.argsort(d[i]); rank = np.empty(len(codes), int); rank[o] = np.arange(len(codes))
|
| 192 |
+
cand = list(dict.fromkeys(list(codes[o[:K]]) + [c for c in Ep.get(p, []) if c in cidx]))
|
| 193 |
+
emax = max([E[(p, c)] for c in Ep.get(p, [])] + [0]); gap = d[i, o[1]] - d[i, o[0]] if len(o) > 1 else 0
|
| 194 |
+
for c in cand:
|
| 195 |
+
j = cidx[c]; pos = lu_pos.get(c)
|
| 196 |
+
rows.append(dict(poly=p, code=c, d=d[i, j], d_rel=d[i, j] - d[i, o[0]], rang=rank[j], gap12=gap,
|
| 197 |
+
ocr=E.get((p, c), 0), ocr_assigne=asg.get((p, c), 0), ocr_max_poly=emax,
|
| 198 |
+
lu=int(pos is not None), n_lu=0 if pos is None else len(pos),
|
| 199 |
+
dist_lu=5000 if pos is None else np.hypot(pos[:, 0] - cents[i][0], pos[:, 1] - cents[i][1]).min(),
|
| 200 |
+
log_aire=np.log(areas[i])))
|
| 201 |
+
X = pd.DataFrame(rows)
|
| 202 |
+
X["d_ratio"] = X.d_rel / (X.gap12 + 1); X["d_norm"] = X.d / X.d.median(); X["ocr_rel"] = X.ocr - X.ocr_max_poly
|
| 203 |
+
return X
|
| 204 |
+
|
| 205 |
+
def name_polygons(rgb, lab, words, legend, naming_pkl, flt=None, ranker_pkl="legend_ranker.pkl", sure=0.8, check=0.5):
|
| 206 |
+
"""legend : DataFrame index=code, colonnes nom, age, lithologie, couleur_hex, feat (signature de la case).
|
| 207 |
+
Retourne par polygone : code, source_nom, confiance, texte_lu, suggestions.
|
| 208 |
+
source_nom : 'code lu sur la carte' / 'légende' (confiance >= sure), '… – à vérifier' (check <= confiance < sure), 'non identifiée'."""
|
| 209 |
+
D, ch = ocr_candidates(rgb, lab, words, legend.index, naming_pkl, flt)
|
| 210 |
+
R = pickle.load(open(ranker_pkl, "rb"))
|
| 211 |
+
labc = sk_rgb2lab(rgb).reshape(-1, 3)
|
| 212 |
+
ids = np.unique(lab[lab > 0])
|
| 213 |
+
order = np.argsort(lab.ravel(), kind="stable"); sl = lab.ravel()[order]
|
| 214 |
+
starts = np.searchsorted(sl, ids); ends = np.searchsorted(sl, ids, side="right")
|
| 215 |
+
feats = np.stack([swatch_features(labc[order[a:b]]) for a, b in zip(starts, ends)])
|
| 216 |
+
cents = ndi.center_of_mass(np.ones_like(lab), lab, ids)
|
| 217 |
+
codes = np.array(legend.index); Fm = np.stack(legend.feat.values)
|
| 218 |
+
X = rank_pairs(ids, feats, ends - starts, cents, codes, Fm, D, ch, words, K=R.get("K", 12))
|
| 219 |
+
X["p"] = R["model"].predict_proba(X[R["features"]])[:, 1]
|
| 220 |
+
X = X.sort_values(["poly", "p"], ascending=[True, False])
|
| 221 |
+
lu_txt = {}
|
| 222 |
+
for (_, w), (_, p, c, s) in ch.items():
|
| 223 |
+
lu_txt[(p, c)] = words.iloc[w].text
|
| 224 |
+
out = {}
|
| 225 |
+
for p, g in X.groupby("poly", sort=False):
|
| 226 |
+
top = g.head(3); c, pr = top.code.iloc[0], float(top.p.iloc[0])
|
| 227 |
+
sug = ", ".join(f"{a} ({b:.0%})" for a, b in zip(top.code, top.p))
|
| 228 |
+
ocr = top.ocr.iloc[0] > 0
|
| 229 |
+
src = "code lu sur la carte" if ocr else "couleur de la légende"
|
| 230 |
+
if pr >= sure:
|
| 231 |
+
out[p] = dict(code=c, source_nom=src, confiance=round(pr, 2), texte_lu=lu_txt.get((p, c), ""), suggestions=sug)
|
| 232 |
+
elif pr >= check: # trop peu fiable sur cartes réelles (31-64 % juste) : pas de nom, mais suggestion classée en tête
|
| 233 |
+
out[p] = dict(code="", source_nom="non identifiée – suggestion probable", confiance=round(pr, 2), texte_lu=lu_txt.get((p, c), ""), suggestions=sug)
|
| 234 |
+
else:
|
| 235 |
+
out[p] = dict(code="", source_nom="non identifiée", confiance=round(pr, 2), texte_lu="", suggestions=sug)
|
| 236 |
+
return pd.DataFrame.from_dict(out, orient="index")
|
| 237 |
+
|
| 238 |
+
# ------------------------------------------------------------------ 6. export
|
| 239 |
+
QML_UNITS = open(os.path.join(HERE, "qml_units.xml")).read() if os.path.exists(os.path.join(HERE, "qml_units.xml")) else None
|
| 240 |
+
QML_FAULTS = open(os.path.join(HERE, "qml_faults.xml")).read() if os.path.exists(os.path.join(HERE, "qml_faults.xml")) else None
|
| 241 |
+
|
| 242 |
+
|
| 243 |
+
def export_gpkg(path, P, F, legend):
|
| 244 |
+
if os.path.exists(path):
|
| 245 |
+
os.remove(path)
|
| 246 |
+
P.to_file(path, layer="unites_geologiques", driver="GPKG")
|
| 247 |
+
if len(F):
|
| 248 |
+
F.to_file(path, layer="failles", driver="GPKG")
|
| 249 |
+
con = sqlite3.connect(path)
|
| 250 |
+
legend.drop(columns=["feat"]).reset_index().rename(columns={"index": "code"}).to_sql("legende", con, index=False, if_exists="replace")
|
| 251 |
+
con.execute("""CREATE TABLE IF NOT EXISTS layer_styles (id INTEGER PRIMARY KEY AUTOINCREMENT, f_table_catalog TEXT(256),
|
| 252 |
+
f_table_schema TEXT(256), f_table_name TEXT(256), f_geometry_column TEXT(256), styleName TEXT(30), styleQML TEXT,
|
| 253 |
+
styleSLD TEXT, useAsDefault BOOLEAN, description TEXT, owner TEXT(30), ui TEXT(30), update_time DATETIME DEFAULT CURRENT_TIMESTAMP)""")
|
| 254 |
+
for tbl, qml in (("unites_geologiques", QML_UNITS), ("failles", QML_FAULTS)):
|
| 255 |
+
if qml and (tbl != "failles" or len(F)):
|
| 256 |
+
con.execute("INSERT INTO layer_styles (f_table_catalog,f_table_schema,f_table_name,f_geometry_column,styleName,styleQML,styleSLD,useAsDefault,description,owner,update_time) "
|
| 257 |
+
"VALUES ('','',?,'geom',?,?,'',1,'style GeoCarto','',strftime('%Y-%m-%dT%H:%M:%fZ','now'))", (tbl, tbl, qml))
|
| 258 |
+
con.execute("INSERT INTO gpkg_contents (table_name, data_type, identifier) SELECT 'layer_styles','attributes','layer_styles' "
|
| 259 |
+
"WHERE NOT EXISTS (SELECT 1 FROM gpkg_contents WHERE table_name='layer_styles')")
|
| 260 |
+
con.execute("INSERT INTO gpkg_contents (table_name, data_type, identifier) SELECT 'legende','attributes','legende' "
|
| 261 |
+
"WHERE NOT EXISTS (SELECT 1 FROM gpkg_contents WHERE table_name='legende')")
|
| 262 |
+
con.commit(); con.close()
|
| 263 |
+
|
| 264 |
+
|
| 265 |
+
def run(rgb, transform, crs, model, legend, reader, naming_pkl, out_gpkg, tta=False, log=print, ranker_pkl="legend_ranker.pkl"):
|
| 266 |
+
t = time.time()
|
| 267 |
+
aff, pf = infer(model, rgb, tta=tta); log(f"modèle : {time.time() - t:.0f} s")
|
| 268 |
+
lab = M.mutex_watershed(aff); log(f"segmentation : {lab.max()} polygones ({time.time() - t:.0f} s)")
|
| 269 |
+
words = read_text(rgb, reader); log(f"OCR : {len(words)} textes ({time.time() - t:.0f} s)")
|
| 270 |
+
N = name_polygons(rgb, lab, words, legend, naming_pkl, flt=(pf > 0.5).astype(np.uint8), ranker_pkl=ranker_pkl); log(f"nommage ({time.time() - t:.0f} s)")
|
| 271 |
+
P = vectorize_units(lab, transform, crs)
|
| 272 |
+
P = P.join(N, on="pid")
|
| 273 |
+
P["code"] = P.code.fillna(""); P["source_nom"] = P.source_nom.fillna("non identifiée")
|
| 274 |
+
lg = legend.reindex(P.code)
|
| 275 |
+
P["nom"] = np.where(P.code != "", (P.code + " – " + lg.lithologie.fillna("").values).str.strip(" –"), "Unité non identifiée")
|
| 276 |
+
P["age"] = lg.age.fillna("").values; P["lithologie"] = lg.lithologie.fillna("").values
|
| 277 |
+
med = [ndi.median(rgb[..., c], lab, P.pid.values) for c in range(3)]
|
| 278 |
+
P["couleur_carte"] = ["#%02x%02x%02x" % tuple(int(v) for v in x) for x in zip(*med)]
|
| 279 |
+
P["couleur_hex"] = np.where(P.code != "", lg.couleur_hex.fillna("").values, P.couleur_carte)
|
| 280 |
+
P["couleur_hex"] = np.where(P.couleur_hex == "", P.couleur_carte, P.couleur_hex)
|
| 281 |
+
P["surface_km2"] = (P.area / 1e6).round(3)
|
| 282 |
+
P["id"] = np.arange(1, len(P) + 1)
|
| 283 |
+
P = P[["id", "code", "nom", "age", "lithologie", "source_nom", "confiance", "texte_lu", "suggestions",
|
| 284 |
+
"couleur_hex", "couleur_carte", "surface_km2", "geometry"]]
|
| 285 |
+
F = vectorize_faults(pf, transform, crs)
|
| 286 |
+
export_gpkg(out_gpkg, P, F, legend.loc[legend.index.isin(P.code)])
|
| 287 |
+
log(f"export : {out_gpkg} | {len(P)} unités, {len(F)} failles ({time.time() - t:.0f} s)")
|
| 288 |
+
return P, F, lab, pf
|
qgis_test.py
ADDED
|
@@ -0,0 +1,55 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Vrai test QGIS (PyQGIS 3.34, sans interface) d'un GeoPackage produit par pipeline.py
|
| 2 |
+
# QT_QPA_PLATFORM=offscreen python3.12 qgis_test.py <carte.gpkg> <rendu.png>
|
| 3 |
+
import sys
|
| 4 |
+
from qgis.core import (QgsApplication, QgsVectorLayer, QgsProject, QgsMapSettings, QgsMapRendererParallelJob,
|
| 5 |
+
QgsGeometry, QgsWkbTypes)
|
| 6 |
+
from qgis.PyQt.QtCore import QSize
|
| 7 |
+
from qgis.PyQt.QtGui import QColor
|
| 8 |
+
|
| 9 |
+
gpkg, png = sys.argv[1], sys.argv[2]
|
| 10 |
+
app = QgsApplication([], False); app.initQgis()
|
| 11 |
+
ok = True
|
| 12 |
+
def check(cond, msg):
|
| 13 |
+
global ok
|
| 14 |
+
print(("OK " if cond else "ÉCHEC"), msg); ok &= bool(cond)
|
| 15 |
+
|
| 16 |
+
layers = []
|
| 17 |
+
for name, geom in (("unites_geologiques", QgsWkbTypes.PolygonGeometry), ("failles", QgsWkbTypes.LineGeometry)):
|
| 18 |
+
lyr = QgsVectorLayer(f"{gpkg}|layername={name}", name, "ogr")
|
| 19 |
+
check(lyr.isValid(), f"couche {name} chargée")
|
| 20 |
+
if not lyr.isValid():
|
| 21 |
+
continue
|
| 22 |
+
msg, styled = lyr.loadDefaultStyle() # style QML stocké dans la table layer_styles du GeoPackage
|
| 23 |
+
check(styled, f"{name} : style intégré chargé automatiquement ({msg})")
|
| 24 |
+
check(lyr.geometryType() == geom, f"{name} : type de géométrie {QgsWkbTypes.geometryDisplayString(lyr.geometryType())}")
|
| 25 |
+
check(lyr.crs().isValid(), f"{name} : SCR {lyr.crs().authid() or lyr.crs().description()}")
|
| 26 |
+
n = lyr.featureCount(); check(n > 0, f"{name} : {n} entités")
|
| 27 |
+
bad = sum(1 for f in lyr.getFeatures() if not f.geometry().isGeosValid())
|
| 28 |
+
check(bad == 0, f"{name} : géométries invalides = {bad}")
|
| 29 |
+
print(" champs :", [f.name() for f in lyr.fields()])
|
| 30 |
+
r = lyr.renderer()
|
| 31 |
+
print(" rendu :", r.type(), (f"{len(r.categories())} catégories" if hasattr(r, "categories") else ""))
|
| 32 |
+
if name == "unites_geologiques":
|
| 33 |
+
need = {"code", "nom", "age", "lithologie", "source_nom", "confiance", "suggestions"}
|
| 34 |
+
check(need <= {f.name() for f in lyr.fields()}, "colonnes de nommage présentes")
|
| 35 |
+
# filtre attributaire comme le ferait un utilisateur QGIS
|
| 36 |
+
lyr.setSubsetString("\"source_nom\" = 'code lu sur la carte'")
|
| 37 |
+
print(" filtre source_nom='code lu sur la carte' :", lyr.featureCount(), "entités"); lyr.setSubsetString("")
|
| 38 |
+
ex = next(lyr.getFeatures())
|
| 39 |
+
print(" exemple :", {k: ex[k] for k in ("code", "nom", "age", "source_nom", "confiance")})
|
| 40 |
+
layers.append(lyr)
|
| 41 |
+
|
| 42 |
+
leg = QgsVectorLayer(f"{gpkg}|layername=legende", "legende", "ogr")
|
| 43 |
+
check(leg.isValid() and leg.featureCount() > 0, f"table legende : {leg.featureCount() if leg.isValid() else 0} lignes")
|
| 44 |
+
|
| 45 |
+
if layers:
|
| 46 |
+
QgsProject.instance().addMapLayers(layers)
|
| 47 |
+
ms = QgsMapSettings(); ms.setLayers(layers[::-1]); ms.setDestinationCrs(layers[0].crs())
|
| 48 |
+
ext = layers[0].extent(); ms.setExtent(ext); ms.setOutputSize(QSize(1400, int(1400 * ext.height() / ext.width())))
|
| 49 |
+
ms.setBackgroundColor(QColor("white"))
|
| 50 |
+
job = QgsMapRendererParallelJob(ms); job.start(); job.waitForFinished()
|
| 51 |
+
img = job.renderedImage(); check(img.save(png), f"rendu carte -> {png}")
|
| 52 |
+
QgsProject.instance().write(png.rsplit(".", 1)[0] + ".qgz")
|
| 53 |
+
print(" projet QGIS écrit :", png.rsplit(".", 1)[0] + ".qgz")
|
| 54 |
+
print("\nRÉSULTAT QGIS :", "TOUT OK" if ok else "PROBLÈMES")
|
| 55 |
+
app.exitQgis()
|
qml_faults.xml
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<!DOCTYPE qgis PUBLIC 'http://mrcc.com/qgis.dtd' 'SYSTEM'>
|
| 2 |
+
<qgis version="3.34.0" styleCategories="Symbology">
|
| 3 |
+
<renderer-v2 type="singleSymbol" symbollevels="0" enableorderby="0" forceraster="0">
|
| 4 |
+
<symbols>
|
| 5 |
+
<symbol type="line" name="0" alpha="1" clip_to_extent="1" force_rhr="0">
|
| 6 |
+
<layer class="SimpleLine" enabled="1" locked="0" pass="0">
|
| 7 |
+
<Option type="Map">
|
| 8 |
+
<Option name="line_color" type="QString" value="0,0,0,255"/>
|
| 9 |
+
<Option name="line_width" type="QString" value="0.6"/>
|
| 10 |
+
<Option name="line_width_unit" type="QString" value="MM"/>
|
| 11 |
+
<Option name="line_style" type="QString" value="solid"/>
|
| 12 |
+
</Option>
|
| 13 |
+
</layer>
|
| 14 |
+
</symbol>
|
| 15 |
+
</symbols>
|
| 16 |
+
</renderer-v2>
|
| 17 |
+
</qgis>
|
qml_units.xml
ADDED
|
@@ -0,0 +1,41 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<!DOCTYPE qgis PUBLIC 'http://mrcc.com/qgis.dtd' 'SYSTEM'>
|
| 2 |
+
<qgis version="3.34.0" styleCategories="Symbology|Labeling">
|
| 3 |
+
<renderer-v2 type="singleSymbol" symbollevels="0" enableorderby="0" forceraster="0">
|
| 4 |
+
<symbols>
|
| 5 |
+
<symbol type="fill" name="0" alpha="1" clip_to_extent="1" force_rhr="0">
|
| 6 |
+
<layer class="SimpleFill" enabled="1" locked="0" pass="0">
|
| 7 |
+
<Option type="Map">
|
| 8 |
+
<Option name="color" type="QString" value="200,200,200,255"/>
|
| 9 |
+
<Option name="outline_color" type="QString" value="40,40,40,255"/>
|
| 10 |
+
<Option name="outline_style" type="QString" value="solid"/>
|
| 11 |
+
<Option name="outline_width" type="QString" value="0.15"/>
|
| 12 |
+
<Option name="outline_width_unit" type="QString" value="MM"/>
|
| 13 |
+
<Option name="style" type="QString" value="solid"/>
|
| 14 |
+
</Option>
|
| 15 |
+
<data_defined_properties>
|
| 16 |
+
<Option type="Map">
|
| 17 |
+
<Option name="name" type="QString" value=""/>
|
| 18 |
+
<Option name="properties" type="Map">
|
| 19 |
+
<Option name="fillColor" type="Map">
|
| 20 |
+
<Option name="active" type="bool" value="true"/>
|
| 21 |
+
<Option name="expression" type="QString" value=""couleur_hex""/>
|
| 22 |
+
<Option name="type" type="int" value="3"/>
|
| 23 |
+
</Option>
|
| 24 |
+
</Option>
|
| 25 |
+
<Option name="type" type="QString" value="collection"/>
|
| 26 |
+
</Option>
|
| 27 |
+
</data_defined_properties>
|
| 28 |
+
</layer>
|
| 29 |
+
</symbol>
|
| 30 |
+
</symbols>
|
| 31 |
+
</renderer-v2>
|
| 32 |
+
<labeling type="simple">
|
| 33 |
+
<settings calloutType="simple">
|
| 34 |
+
<text-style fieldName="code" isExpression="0" fontSize="8" fontSizeUnit="Point" fontFamily="Arial" textColor="20,20,20,255" fontWeight="50">
|
| 35 |
+
<text-buffer bufferDraw="1" bufferSize="0.6" bufferSizeUnits="MM" bufferColor="255,255,255,200"/>
|
| 36 |
+
</text-style>
|
| 37 |
+
<placement placement="1" placementFlags="10" centroidInside="1" fitInPolygonOnly="1"/>
|
| 38 |
+
<rendering scaleVisibility="0" obstacle="1" limitNumLabels="0" displayAll="0"/>
|
| 39 |
+
</settings>
|
| 40 |
+
</labeling>
|
| 41 |
+
</qgis>
|
requirements.txt
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
torch>=2.2
|
| 2 |
+
segmentation-models-pytorch==0.6.0
|
| 3 |
+
timm==1.0.31
|
| 4 |
+
numba>=0.60
|
| 5 |
+
numpy
|
| 6 |
+
scipy
|
| 7 |
+
pandas
|
| 8 |
+
scikit-learn==1.9.1
|
| 9 |
+
scikit-image>=0.24
|
| 10 |
+
rasterio>=1.4
|
| 11 |
+
geopandas>=1.0
|
| 12 |
+
shapely>=2.0
|
| 13 |
+
easyocr==1.7.2
|
training/e2e_naming_9_zones.csv
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
zone,modele,surface,nommee,juste,sure,sure_juste
|
| 2 |
+
north_2048_10240,geocarto_model/best_fp16.pt,3599656,1499573,1385929,1499573,1385929
|
| 3 |
+
north_0_14336,geocarto_model/best_fp16.pt,3946304,1366865,1312457,1366865,1312457
|
| 4 |
+
north_4096_8192,geocarto_model/best_fp16.pt,3701097,629318,476448,629318,476448
|
| 5 |
+
north_4096_18432,geocarto_model/best_fp16.pt,4194304,162091,145539,162091,145539
|
| 6 |
+
north_8192_16384,geocarto_model/best_fp16.pt,4012500,835803,665180,835803,665180
|
| 7 |
+
north_6144_16384,geocarto_model/best_fp16.pt,3958564,1188440,1148893,1188440,1148893
|
| 8 |
+
north_8192_10240,geocarto_model/best_fp16.pt,3799155,2138119,1859959,2138119,1859959
|
| 9 |
+
north_2048_16384,geocarto_model/best_fp16.pt,4184402,491224,465825,491224,465825
|
| 10 |
+
north_0_2048,geocarto_model/best_fp16.pt,4074680,2335504,2287133,2335504,2287133
|
training/history.csv
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
step,loss,heures,modele_score,modele_reel,modele_synth,meilleur,ema_score,ema_reel,ema_synth
|
| 2 |
+
2500,1.2678692572832106,0.48,0.1156618180162316,0.05801492429830092,0.17330871173416226,modele,0.1257966551104029,0.05539731695128261,0.1961959932695232
|
| 3 |
+
5000,0.7672936572551727,0.91,0.09529314214143128,0.052139002017007524,0.13844728226585504,modele,0.0973675850057942,0.04910327124788072,0.1456318987637077
|
| 4 |
+
7500,0.7159945685982704,1.34,0.08752472791366481,0.05359620966079264,0.12145324616653697,modele,0.09248038060147537,0.04966643006802915,0.1352943311349216
|
| 5 |
+
10000,0.685878133893013,1.76,0.0836330135427693,0.050465125732164565,0.11680090135337402,modele,0.08596079908456222,0.05022387633190165,0.12169772183722279
|
| 6 |
+
12500,0.6719268432140351,2.18,0.08175700726637768,0.04745522503445026,0.1160587894983051,ema,0.07972059465372879,0.0443447903329452,0.11509639897451236
|
| 7 |
+
15000,0.6572574779510498,2.61,0.07931626369027464,0.049011944981386724,0.10962058239916257,modele,0.08061698872873928,0.04628204825805805,0.1149519291994205
|
| 8 |
+
17500,0.6435735144972802,3.03,0.07761612833109743,0.04601976537745372,0.10921249128474114,modele,0.07827108745221906,0.04586425730415124,0.11067791760028689
|
| 9 |
+
20000,0.6359998931407929,3.45,0.07532928573818343,0.04557381860105503,0.10508475287531183,modele,0.07670002805089449,0.046568494432873635,0.10683156166891535
|
| 10 |
+
22500,0.626114172899723,3.87,0.07367399082233667,0.043336204269031287,0.10401177737564203,modele,0.07437775992976638,0.044113875308752916,0.10464164455077984
|
| 11 |
+
25000,0.6217467277765274,4.3,0.07452715778951396,0.0464807556550982,0.10257355992392973,,0.07528071106283755,0.04745712026447532,0.10310430186119979
|
| 12 |
+
27500,0.6217535385727883,4.72,0.07442850533802978,0.045681225671011565,0.103175785005048,,0.0742625662646455,0.04581297982583516,0.10271215270345585
|
| 13 |
+
30000,0.6190747852683067,5.14,0.07454904397298479,0.04589639765003511,0.10320169029593447,,0.07448182474760545,0.04589398591477694,0.10306966358043397
|
training/lab_round4_ablation.csv
ADDED
|
@@ -0,0 +1,4 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
variante,desc,train_s,réel|ARAND,réel|split,réel|merge,réel|rappel_petites,réel|rappel_grandes,réel|F1_failles,synthétique|ARAND,synthétique|split,synthétique|merge,synthétique|rappel_petites,synthétique|rappel_grandes,synthétique|F1_failles,piège_mono|ARAND,piège_mono|split,piège_mono|merge,piège_mono|rappel_petites,piège_mono|rappel_grandes,piège_mono|F1_failles,piège_motifs|ARAND,piège_motifs|split,piège_motifs|merge,piège_motifs|rappel_petites,piège_motifs|rappel_grandes,piège_motifs|F1_failles,réel@s3m15|ARAND,réel@s3m15|split,réel@s3m15|merge,réel@s3m15|rappel_petites,réel@s3m15|rappel_grandes,réel@s3m15|F1_failles,synthétique@s3m15|ARAND,synthétique@s3m15|split,synthétique@s3m15|merge,synthétique@s3m15|rappel_petites,synthétique@s3m15|rappel_grandes,synthétique@s3m15|F1_failles,piège_mono@s3m15|ARAND,piège_mono@s3m15|split,piège_mono@s3m15|merge,piège_mono@s3m15|rappel_petites,piège_mono@s3m15|rappel_grandes,piège_mono@s3m15|F1_failles,piège_motifs@s3m15|ARAND,piège_motifs@s3m15|split,piège_motifs@s3m15|merge,piège_motifs@s3m15|rappel_petites,piège_motifs@s3m15|rappel_grandes,piège_motifs@s3m15|F1_failles,réel@s1m40|ARAND,réel@s1m40|split,réel@s1m40|merge,réel@s1m40|rappel_petites,réel@s1m40|rappel_grandes,réel@s1m40|F1_failles,synthétique@s1m40|ARAND,synthétique@s1m40|split,synthétique@s1m40|merge,synthétique@s1m40|rappel_petites,synthétique@s1m40|rappel_grandes,synthétique@s1m40|F1_failles,piège_mono@s1m40|ARAND,piège_mono@s1m40|split,piège_mono@s1m40|merge,piège_mono@s1m40|rappel_petites,piège_mono@s1m40|rappel_grandes,piège_mono@s1m40|F1_failles,piège_motifs@s1m40|ARAND,piège_motifs@s1m40|split,piège_motifs@s1m40|merge,piège_motifs@s1m40|rappel_petites,piège_motifs@s1m40|rappel_grandes,piège_motifs@s1m40|F1_failles,réel@sttam40|ARAND,réel@sttam40|split,réel@sttam40|merge,réel@sttam40|rappel_petites,réel@sttam40|rappel_grandes,réel@sttam40|F1_failles,synthétique@sttam40|ARAND,synthétique@sttam40|split,synthétique@sttam40|merge,synthétique@sttam40|rappel_petites,synthétique@sttam40|rappel_grandes,synthétique@sttam40|F1_failles,piège_mono@sttam40|ARAND,piège_mono@sttam40|split,piège_mono@sttam40|merge,piège_mono@sttam40|rappel_petites,piège_mono@sttam40|rappel_grandes,piège_mono@sttam40|F1_failles,piège_motifs@sttam40|ARAND,piège_motifs@sttam40|split,piège_motifs@sttam40|merge,piège_motifs@sttam40|rappel_petites,piège_motifs@sttam40|rappel_grandes,piège_motifs@sttam40|F1_failles
|
| 2 |
+
M0,R3 sur données généralisées (unité minimale de cartographie),1748,0.0672,0.2439,0.2701,0.7482,0.8923,0.7426,0.1463,0.295,0.6285,0.7225,0.8621,0.9339,0.2164,0.6157,0.8184,0.5046,0.6552,0.9545,0.1749,0.405,0.794,0.6166,0.8019,0.9502,0.0672,0.2428,0.2724,0.754,0.8923,0.7426,0.1464,0.2838,0.6452,0.7378,0.8608,0.9339,0.2165,0.6035,0.8388,0.5083,0.6562,0.9545,0.1748,0.3917,0.8149,0.6239,0.8019,0.9502,0.0703,0.234,0.2843,0.7641,0.906,0.7426,0.1605,0.2937,0.7135,0.7196,0.8522,0.9339,0.2279,0.5568,0.9462,0.5266,0.6768,0.9545,0.1928,0.3936,0.9258,0.6266,0.785,0.9502,0.0662,0.2643,0.2408,0.7381,0.8769,0.7325,0.132,0.3012,0.5495,0.7287,0.8695,0.9407,0.2296,0.7467,0.6981,0.4863,0.6207,0.9587,0.1703,0.4471,0.7154,0.6074,0.7879,0.955
|
| 3 |
+
M1,M0 + poids des petits polygones,1780,0.1951,0.7857,0.2111,0.6599,0.4632,0.8535,0.2007,0.4829,0.611,0.7664,0.7685,0.9163,0.2745,0.806,0.85,0.6127,0.5557,0.9321,0.2344,0.5837,0.8392,0.6788,0.7075,0.9394,0.1951,0.7846,0.2137,0.6599,0.4632,0.8535,0.2012,0.4692,0.6381,0.7755,0.7648,0.9163,0.2751,0.7851,0.8922,0.6145,0.5557,0.9321,0.2349,0.5642,0.8771,0.6834,0.7084,0.9394,0.1707,0.6542,0.2365,0.6715,0.535,0.8535,0.1801,0.434,0.6408,0.6777,0.782,0.9163,0.2697,0.7596,0.8923,0.5754,0.5734,0.9321,0.2336,0.5346,0.9096,0.6361,0.7196,0.9394,0.3231,1.4159,0.1157,0.6165,0.2222,0.8527,0.2767,0.7946,0.5683,0.7651,0.6724,0.9227,0.3779,1.2186,0.7704,0.6127,0.4266,0.9388,0.2697,0.7292,0.7805,0.6851,0.6626,0.9437
|
| 4 |
+
M2,M1 + OHEM (arêtes difficiles),1742,0.1089,0.3797,0.2879,0.7308,0.7897,0.836,0.1584,0.317,0.66,0.7958,0.8251,0.9049,0.267,0.6589,0.9735,0.6465,0.6138,0.93,0.1892,0.3849,0.9197,0.704,0.7458,0.9225,0.109,0.3784,0.2922,0.7381,0.7897,0.836,0.1591,0.3056,0.6859,0.8012,0.8239,0.9049,0.268,0.6417,1.0144,0.6459,0.6128,0.93,0.1901,0.368,0.9563,0.7067,0.743,0.9225,0.097,0.3264,0.3144,0.7554,0.8188,0.836,0.1646,0.3254,0.7093,0.7382,0.819,0.9049,0.2639,0.6298,1.0306,0.6234,0.6207,0.93,0.1983,0.3907,0.9935,0.6805,0.7355,0.9225,0.1174,0.4317,0.2495,0.7482,0.7607,0.8417,0.1608,0.3647,0.6073,0.7962,0.8153,0.9147,0.2862,0.7393,0.9183,0.6626,0.601,0.9284,0.1916,0.4099,0.868,0.7294,0.7486,0.9309
|
training/metrics.json
ADDED
|
@@ -0,0 +1,257 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"cfg": {
|
| 3 |
+
"arch": "unetpp",
|
| 4 |
+
"encoder": "efficientnet-b4",
|
| 5 |
+
"input": "rgb+lab",
|
| 6 |
+
"offsets": [
|
| 7 |
+
[
|
| 8 |
+
1,
|
| 9 |
+
0
|
| 10 |
+
],
|
| 11 |
+
[
|
| 12 |
+
0,
|
| 13 |
+
1
|
| 14 |
+
],
|
| 15 |
+
[
|
| 16 |
+
3,
|
| 17 |
+
0
|
| 18 |
+
],
|
| 19 |
+
[
|
| 20 |
+
0,
|
| 21 |
+
3
|
| 22 |
+
],
|
| 23 |
+
[
|
| 24 |
+
9,
|
| 25 |
+
0
|
| 26 |
+
],
|
| 27 |
+
[
|
| 28 |
+
0,
|
| 29 |
+
9
|
| 30 |
+
],
|
| 31 |
+
[
|
| 32 |
+
9,
|
| 33 |
+
9
|
| 34 |
+
],
|
| 35 |
+
[
|
| 36 |
+
9,
|
| 37 |
+
-9
|
| 38 |
+
],
|
| 39 |
+
[
|
| 40 |
+
27,
|
| 41 |
+
0
|
| 42 |
+
],
|
| 43 |
+
[
|
| 44 |
+
0,
|
| 45 |
+
27
|
| 46 |
+
]
|
| 47 |
+
],
|
| 48 |
+
"n_short": 2,
|
| 49 |
+
"mws_stride": 1,
|
| 50 |
+
"mws_min_size": 40,
|
| 51 |
+
"tile": 512,
|
| 52 |
+
"small_w": false,
|
| 53 |
+
"ohem": false,
|
| 54 |
+
"tta": false
|
| 55 |
+
},
|
| 56 |
+
"steps": 22500,
|
| 57 |
+
"val_score": 0.07367399082233667,
|
| 58 |
+
"test": [
|
| 59 |
+
{
|
| 60 |
+
"groupe": "Africa",
|
| 61 |
+
"tuiles": 70,
|
| 62 |
+
"ARAND": 0.0299,
|
| 63 |
+
"sur_decoupage": 0.1534,
|
| 64 |
+
"fusion": 0.1099,
|
| 65 |
+
"rappel_petites": 0.8958,
|
| 66 |
+
"rappel_grandes": 0.9785,
|
| 67 |
+
"rappel_40_200": 0.8,
|
| 68 |
+
"rappel_200_1000": 0.9475,
|
| 69 |
+
"rappel_tous": 0.9466,
|
| 70 |
+
"surface_correcte": 0.9931,
|
| 71 |
+
"F1_failles": NaN
|
| 72 |
+
},
|
| 73 |
+
{
|
| 74 |
+
"groupe": "Europe",
|
| 75 |
+
"tuiles": 120,
|
| 76 |
+
"ARAND": 0.0986,
|
| 77 |
+
"sur_decoupage": 0.6392,
|
| 78 |
+
"fusion": 0.2721,
|
| 79 |
+
"rappel_petites": 0.8224,
|
| 80 |
+
"rappel_grandes": 0.9041,
|
| 81 |
+
"rappel_40_200": 0.781,
|
| 82 |
+
"rappel_200_1000": 0.8925,
|
| 83 |
+
"rappel_tous": 0.8386,
|
| 84 |
+
"surface_correcte": 0.9202,
|
| 85 |
+
"F1_failles": NaN
|
| 86 |
+
},
|
| 87 |
+
{
|
| 88 |
+
"groupe": "North America",
|
| 89 |
+
"tuiles": 268,
|
| 90 |
+
"ARAND": 0.0912,
|
| 91 |
+
"sur_decoupage": 0.4542,
|
| 92 |
+
"fusion": 0.2775,
|
| 93 |
+
"rappel_petites": 0.7249,
|
| 94 |
+
"rappel_grandes": 0.9054,
|
| 95 |
+
"rappel_40_200": 0.6559,
|
| 96 |
+
"rappel_200_1000": 0.7991,
|
| 97 |
+
"rappel_tous": 0.7677,
|
| 98 |
+
"surface_correcte": 0.9507,
|
| 99 |
+
"F1_failles": 0.9503
|
| 100 |
+
},
|
| 101 |
+
{
|
| 102 |
+
"groupe": "Oceania",
|
| 103 |
+
"tuiles": 150,
|
| 104 |
+
"ARAND": 0.1763,
|
| 105 |
+
"sur_decoupage": 0.9725,
|
| 106 |
+
"fusion": 0.4681,
|
| 107 |
+
"rappel_petites": 0.6385,
|
| 108 |
+
"rappel_grandes": 0.7555,
|
| 109 |
+
"rappel_40_200": 0.6085,
|
| 110 |
+
"rappel_200_1000": 0.6838,
|
| 111 |
+
"rappel_tous": 0.6565,
|
| 112 |
+
"surface_correcte": 0.8352,
|
| 113 |
+
"F1_failles": 0.9868
|
| 114 |
+
},
|
| 115 |
+
{
|
| 116 |
+
"groupe": "GA-Australia / synthetic",
|
| 117 |
+
"tuiles": 150,
|
| 118 |
+
"ARAND": 0.1763,
|
| 119 |
+
"sur_decoupage": 0.9725,
|
| 120 |
+
"fusion": 0.4681,
|
| 121 |
+
"rappel_petites": 0.6385,
|
| 122 |
+
"rappel_grandes": 0.7555,
|
| 123 |
+
"rappel_40_200": 0.6085,
|
| 124 |
+
"rappel_200_1000": 0.6838,
|
| 125 |
+
"rappel_tous": 0.6565,
|
| 126 |
+
"surface_correcte": 0.8352,
|
| 127 |
+
"F1_failles": 0.9868
|
| 128 |
+
},
|
| 129 |
+
{
|
| 130 |
+
"groupe": "GEO-LiM / synthetic",
|
| 131 |
+
"tuiles": 120,
|
| 132 |
+
"ARAND": 0.0986,
|
| 133 |
+
"sur_decoupage": 0.6392,
|
| 134 |
+
"fusion": 0.2721,
|
| 135 |
+
"rappel_petites": 0.8224,
|
| 136 |
+
"rappel_grandes": 0.9041,
|
| 137 |
+
"rappel_40_200": 0.781,
|
| 138 |
+
"rappel_200_1000": 0.8925,
|
| 139 |
+
"rappel_tous": 0.8386,
|
| 140 |
+
"surface_correcte": 0.9202,
|
| 141 |
+
"F1_failles": NaN
|
| 142 |
+
},
|
| 143 |
+
{
|
| 144 |
+
"groupe": "GMNA / real",
|
| 145 |
+
"tuiles": 78,
|
| 146 |
+
"ARAND": 0.0518,
|
| 147 |
+
"sur_decoupage": 0.2378,
|
| 148 |
+
"fusion": 0.2227,
|
| 149 |
+
"rappel_petites": 0.7467,
|
| 150 |
+
"rappel_grandes": 0.9341,
|
| 151 |
+
"rappel_40_200": 0.5038,
|
| 152 |
+
"rappel_200_1000": 0.8611,
|
| 153 |
+
"rappel_tous": 0.8336,
|
| 154 |
+
"surface_correcte": 0.9828,
|
| 155 |
+
"F1_failles": 0.6303
|
| 156 |
+
},
|
| 157 |
+
{
|
| 158 |
+
"groupe": "GMNA / synthetic",
|
| 159 |
+
"tuiles": 60,
|
| 160 |
+
"ARAND": 0.0244,
|
| 161 |
+
"sur_decoupage": 0.1356,
|
| 162 |
+
"fusion": 0.1323,
|
| 163 |
+
"rappel_petites": 0.8918,
|
| 164 |
+
"rappel_grandes": 0.977,
|
| 165 |
+
"rappel_40_200": 0.799,
|
| 166 |
+
"rappel_200_1000": 0.9366,
|
| 167 |
+
"rappel_tous": 0.929,
|
| 168 |
+
"surface_correcte": 0.9911,
|
| 169 |
+
"F1_failles": 0.9717
|
| 170 |
+
},
|
| 171 |
+
{
|
| 172 |
+
"groupe": "UNESCO-Ethiopia / synthetic",
|
| 173 |
+
"tuiles": 70,
|
| 174 |
+
"ARAND": 0.0299,
|
| 175 |
+
"sur_decoupage": 0.1534,
|
| 176 |
+
"fusion": 0.1099,
|
| 177 |
+
"rappel_petites": 0.8958,
|
| 178 |
+
"rappel_grandes": 0.9785,
|
| 179 |
+
"rappel_40_200": 0.8,
|
| 180 |
+
"rappel_200_1000": 0.9475,
|
| 181 |
+
"rappel_tous": 0.9466,
|
| 182 |
+
"surface_correcte": 0.9931,
|
| 183 |
+
"F1_failles": NaN
|
| 184 |
+
},
|
| 185 |
+
{
|
| 186 |
+
"groupe": "USGS-California / synthetic",
|
| 187 |
+
"tuiles": 70,
|
| 188 |
+
"ARAND": 0.0766,
|
| 189 |
+
"sur_decoupage": 0.5323,
|
| 190 |
+
"fusion": 0.347,
|
| 191 |
+
"rappel_petites": 0.7412,
|
| 192 |
+
"rappel_grandes": 0.8937,
|
| 193 |
+
"rappel_40_200": 0.6809,
|
| 194 |
+
"rappel_200_1000": 0.7966,
|
| 195 |
+
"rappel_tous": 0.776,
|
| 196 |
+
"surface_correcte": 0.9462,
|
| 197 |
+
"F1_failles": 0.9708
|
| 198 |
+
},
|
| 199 |
+
{
|
| 200 |
+
"groupe": "USGS-Nevada / synthetic",
|
| 201 |
+
"tuiles": 60,
|
| 202 |
+
"ARAND": 0.2261,
|
| 203 |
+
"sur_decoupage": 0.9629,
|
| 204 |
+
"fusion": 0.4127,
|
| 205 |
+
"rappel_petites": 0.6904,
|
| 206 |
+
"rappel_grandes": 0.8674,
|
| 207 |
+
"rappel_40_200": 0.6424,
|
| 208 |
+
"rappel_200_1000": 0.7605,
|
| 209 |
+
"rappel_tous": 0.7195,
|
| 210 |
+
"surface_correcte": 0.8765,
|
| 211 |
+
"F1_failles": 0.9512
|
| 212 |
+
},
|
| 213 |
+
{
|
| 214 |
+
"groupe": "TEST COMPLET",
|
| 215 |
+
"tuiles": 608,
|
| 216 |
+
"ARAND": 0.1066,
|
| 217 |
+
"sur_decoupage": 0.5839,
|
| 218 |
+
"fusion": 0.3042,
|
| 219 |
+
"rappel_petites": 0.7045,
|
| 220 |
+
"rappel_grandes": 0.8619,
|
| 221 |
+
"rappel_40_200": 0.6578,
|
| 222 |
+
"rappel_200_1000": 0.7671,
|
| 223 |
+
"rappel_tous": 0.7365,
|
| 224 |
+
"surface_correcte": 0.9208,
|
| 225 |
+
"F1_failles": 0.9672
|
| 226 |
+
},
|
| 227 |
+
{
|
| 228 |
+
"groupe": "PIÈGE unités de même couleur",
|
| 229 |
+
"tuiles": 48,
|
| 230 |
+
"ARAND": 0.1508,
|
| 231 |
+
"sur_decoupage": 0.6236,
|
| 232 |
+
"fusion": 0.4517,
|
| 233 |
+
"rappel_petites": 0.6281,
|
| 234 |
+
"rappel_grandes": 0.7687,
|
| 235 |
+
"rappel_40_200": 0.5926,
|
| 236 |
+
"rappel_200_1000": 0.6694,
|
| 237 |
+
"rappel_tous": 0.6606,
|
| 238 |
+
"surface_correcte": 0.8492,
|
| 239 |
+
"F1_failles": 0.9746
|
| 240 |
+
},
|
| 241 |
+
{
|
| 242 |
+
"groupe": "PIÈGE unités à motifs",
|
| 243 |
+
"tuiles": 48,
|
| 244 |
+
"ARAND": 0.0994,
|
| 245 |
+
"sur_decoupage": 0.5187,
|
| 246 |
+
"fusion": 0.2895,
|
| 247 |
+
"rappel_petites": 0.7434,
|
| 248 |
+
"rappel_grandes": 0.8976,
|
| 249 |
+
"rappel_40_200": 0.6931,
|
| 250 |
+
"rappel_200_1000": 0.7994,
|
| 251 |
+
"rappel_tous": 0.7802,
|
| 252 |
+
"surface_correcte": 0.9394,
|
| 253 |
+
"F1_failles": 0.9774
|
| 254 |
+
}
|
| 255 |
+
],
|
| 256 |
+
"train_hours": 5.29
|
| 257 |
+
}
|
training/test_examples.png
ADDED
|
Git LFS Details
|
training/test_metrics.csv
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
groupe,tuiles,ARAND,sur_decoupage,fusion,rappel_petites,rappel_grandes,rappel_40_200,rappel_200_1000,rappel_tous,surface_correcte,F1_failles
|
| 2 |
+
Africa,70,0.0299,0.1534,0.1099,0.8958,0.9785,0.8,0.9475,0.9466,0.9931,
|
| 3 |
+
Europe,120,0.0986,0.6392,0.2721,0.8224,0.9041,0.781,0.8925,0.8386,0.9202,
|
| 4 |
+
North America,268,0.0912,0.4542,0.2775,0.7249,0.9054,0.6559,0.7991,0.7677,0.9507,0.9503
|
| 5 |
+
Oceania,150,0.1763,0.9725,0.4681,0.6385,0.7555,0.6085,0.6838,0.6565,0.8352,0.9868
|
| 6 |
+
GA-Australia / synthetic,150,0.1763,0.9725,0.4681,0.6385,0.7555,0.6085,0.6838,0.6565,0.8352,0.9868
|
| 7 |
+
GEO-LiM / synthetic,120,0.0986,0.6392,0.2721,0.8224,0.9041,0.781,0.8925,0.8386,0.9202,
|
| 8 |
+
GMNA / real,78,0.0518,0.2378,0.2227,0.7467,0.9341,0.5038,0.8611,0.8336,0.9828,0.6303
|
| 9 |
+
GMNA / synthetic,60,0.0244,0.1356,0.1323,0.8918,0.977,0.799,0.9366,0.929,0.9911,0.9717
|
| 10 |
+
UNESCO-Ethiopia / synthetic,70,0.0299,0.1534,0.1099,0.8958,0.9785,0.8,0.9475,0.9466,0.9931,
|
| 11 |
+
USGS-California / synthetic,70,0.0766,0.5323,0.347,0.7412,0.8937,0.6809,0.7966,0.776,0.9462,0.9708
|
| 12 |
+
USGS-Nevada / synthetic,60,0.2261,0.9629,0.4127,0.6904,0.8674,0.6424,0.7605,0.7195,0.8765,0.9512
|
| 13 |
+
TEST COMPLET,608,0.1066,0.5839,0.3042,0.7045,0.8619,0.6578,0.7671,0.7365,0.9208,0.9672
|
| 14 |
+
PIÈGE unités de même couleur,48,0.1508,0.6236,0.4517,0.6281,0.7687,0.5926,0.6694,0.6606,0.8492,0.9746
|
| 15 |
+
PIÈGE unités à motifs,48,0.0994,0.5187,0.2895,0.7434,0.8976,0.6931,0.7994,0.7802,0.9394,0.9774
|
training/train_final.py
ADDED
|
@@ -0,0 +1,471 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# =====================================================================================
|
| 2 |
+
# GEOCARTO – ENTRAÎNEMENT FINAL
|
| 3 |
+
# Modèle retenu par le labo (rounds 1-3) : U-Net++ / EfficientNet-B4, entrée RGB+Lab,
|
| 4 |
+
# augmentations de couleur, affinités (10 décalages) + failles, mutex watershed.
|
| 5 |
+
# Données : dataset GeoCarto complet (6 600 tuiles, 4 continents), split spatial.
|
| 6 |
+
# Sorties (/kaggle/working/geocarto_model) :
|
| 7 |
+
# best.pt (poids + config), best_fp16.pt, history.csv, test_metrics.csv, metrics.json, figures
|
| 8 |
+
# =====================================================================================
|
| 9 |
+
import os, io, sys, json, time, copy
|
| 10 |
+
from pathlib import Path
|
| 11 |
+
import numpy as np, pandas as pd
|
| 12 |
+
import pyarrow.parquet as pq
|
| 13 |
+
from PIL import Image
|
| 14 |
+
from scipy import ndimage as ndi
|
| 15 |
+
from skimage.metrics import adapted_rand_error, variation_of_information
|
| 16 |
+
import torch, torch.nn as nn, torch.nn.functional as F
|
| 17 |
+
from torch.utils.data import Dataset, DataLoader
|
| 18 |
+
import segmentation_models_pytorch as smp
|
| 19 |
+
from numba import njit
|
| 20 |
+
|
| 21 |
+
E = os.environ.get
|
| 22 |
+
STEPS = int(E("GC_STEPS", 30000)) # pas d'entraînement
|
| 23 |
+
BATCH, CROP, LR = int(E("GC_BATCH", 8)), int(E("GC_CROP", 448)), float(E("GC_LR", 3e-4))
|
| 24 |
+
EVAL_EVERY = int(E("GC_EVAL_EVERY", 2500))
|
| 25 |
+
N_VAL = int(E("GC_NVAL", 12)) # tuiles de validation par source (suivi pendant l'entraînement)
|
| 26 |
+
P_REAL = float(E("GC_PREAL", 0.35)) # part de tuiles RÉELLES dans chaque lot (elles ne font que 21 % du dataset)
|
| 27 |
+
MAX_HOURS = float(E("GC_MAX_HOURS", 8.0)) # garde-fou : on arrête l'entraînement avant la limite Kaggle (12 h)
|
| 28 |
+
N_TEST = int(E("GC_NTEST", 0)) # 0 = tout le jeu de test
|
| 29 |
+
OUT = Path(E("GC_OUT", "/kaggle/working/geocarto_model")); OUT.mkdir(parents=True, exist_ok=True)
|
| 30 |
+
DEV = "cuda" if torch.cuda.is_available() else "cpu"
|
| 31 |
+
T_START = time.time()
|
| 32 |
+
print("Matériel :", DEV, torch.cuda.get_device_name(0) if DEV == "cuda" else "", flush=True)
|
| 33 |
+
|
| 34 |
+
CFG = dict(arch="unetpp", encoder=E("GC_ENCODER", "efficientnet-b4"), input="rgb+lab",
|
| 35 |
+
offsets=[(1, 0), (0, 1), (3, 0), (0, 3), (9, 0), (0, 9), (9, 9), (9, -9), (27, 0), (0, 27)],
|
| 36 |
+
n_short=2, mws_stride=int(E("GC_MWS_STRIDE", 1)), mws_min_size=int(E("GC_MINSIZE", 40)), tile=512,
|
| 37 |
+
small_w=E("GC_SMALLW", "0") == "1", ohem=E("GC_OHEM", "0") == "1", tta=E("GC_TTA", "0") == "1")
|
| 38 |
+
OFFSETS, N_SHORT, K = CFG["offsets"], CFG["n_short"], len(CFG["offsets"])
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
# ------------------------------------------------------------------ données
|
| 42 |
+
def find_dir(marker):
|
| 43 |
+
root = Path(E("GC_INPUT", "/kaggle/input"))
|
| 44 |
+
hits = sorted(root.rglob(marker))
|
| 45 |
+
return hits[0].parent if hits else None
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
DATA = Path(E("GC_DATA")) if E("GC_DATA") else find_dir("test-00000.parquet") # dataset complet
|
| 49 |
+
HARD = Path(E("GC_HARD")) if E("GC_HARD") else find_dir("hard_mono-00000.parquet") # jeux pièges (geocarto-mini)
|
| 50 |
+
assert DATA is not None, "dataset GeoCarto introuvable (ajoute la sortie du notebook geocarto-dataset en entrée)"
|
| 51 |
+
print("Données :", DATA, "| pièges :", HARD, flush=True)
|
| 52 |
+
|
| 53 |
+
|
| 54 |
+
def read_split(folder, split):
|
| 55 |
+
"""Garde les PNG compressés en mémoire (2 Go au lieu de 11 Go décodés) ; décodage à la volée."""
|
| 56 |
+
rows = []
|
| 57 |
+
for f in sorted(Path(folder).glob(f"{split}-*.parquet")):
|
| 58 |
+
t = pq.read_table(f, columns=["id", "source", "continent", "kind", "image", "instances", "faults"]).to_pylist()
|
| 59 |
+
rows += [dict(id=r["id"], source=r["source"], continent=r["continent"], kind=r["kind"],
|
| 60 |
+
image=r["image"]["bytes"], inst=r["instances"]["bytes"], faults=r["faults"]["bytes"]) for r in t]
|
| 61 |
+
return rows
|
| 62 |
+
|
| 63 |
+
|
| 64 |
+
def decode(r):
|
| 65 |
+
rgb = np.asarray(Image.open(io.BytesIO(r["image"])).convert("RGB"))
|
| 66 |
+
inst = np.asarray(Image.open(io.BytesIO(r["inst"]))).astype(np.int32)
|
| 67 |
+
flt = (np.asarray(Image.open(io.BytesIO(r["faults"]))) > 0).astype(np.uint8)
|
| 68 |
+
return rgb, inst, flt
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
TRAIN = read_split(DATA, "train")
|
| 72 |
+
VAL_ALL = read_split(DATA, "validation")
|
| 73 |
+
TEST = read_split(DATA, "test")
|
| 74 |
+
HARDSETS = {k: read_split(HARD, k) for k in ("hard_mono", "hard_pattern")} if HARD else {}
|
| 75 |
+
REAL_IDX = [i for i, r in enumerate(TRAIN) if r["kind"] == "real"]
|
| 76 |
+
SYN_IDX = [i for i, r in enumerate(TRAIN) if r["kind"] != "real"]
|
| 77 |
+
|
| 78 |
+
|
| 79 |
+
def spread(rows, n):
|
| 80 |
+
return [rows[i] for i in np.linspace(0, len(rows) - 1, min(n, len(rows))).astype(int)] if rows else []
|
| 81 |
+
|
| 82 |
+
|
| 83 |
+
VAL = {} # suivi : n tuiles par (source, type), réparties régulièrement
|
| 84 |
+
for key in sorted({(r["source"], r["kind"]) for r in VAL_ALL}):
|
| 85 |
+
VAL[key] = spread([r for r in VAL_ALL if (r["source"], r["kind"]) == key], N_VAL * (3 if key[1] == "real" else 1))
|
| 86 |
+
print(f"train {len(TRAIN)} ({len(REAL_IDX)} réelles) | validation suivie {sum(map(len, VAL.values()))} | test {len(TEST)} | "
|
| 87 |
+
+ " | ".join(f"{k} {len(v)}" for k, v in HARDSETS.items()), flush=True)
|
| 88 |
+
|
| 89 |
+
|
| 90 |
+
# ------------------------------------------------------------------ affinités + mutex watershed
|
| 91 |
+
def fill_holes(inst):
|
| 92 |
+
"""Chaque zone sans polygone (mer, lac, halo) devient sa propre r��gion."""
|
| 93 |
+
holes, n = ndi.label(inst == 0)
|
| 94 |
+
if n == 0:
|
| 95 |
+
return inst
|
| 96 |
+
out = inst.copy()
|
| 97 |
+
out[holes > 0] = holes[holes > 0] + inst.max()
|
| 98 |
+
return out
|
| 99 |
+
|
| 100 |
+
|
| 101 |
+
def affinity_targets(inst):
|
| 102 |
+
H, W = inst.shape
|
| 103 |
+
aff, mask = np.zeros((K, H, W), np.float32), np.zeros((K, H, W), np.float32)
|
| 104 |
+
for k, (dy, dx) in enumerate(OFFSETS):
|
| 105 |
+
ys, ye, xs, xe = max(0, -dy), H - max(0, dy), max(0, -dx), W - max(0, dx)
|
| 106 |
+
a, b = inst[ys:ye, xs:xe], inst[ys + dy:ye + dy, xs + dx:xe + dx]
|
| 107 |
+
aff[k, ys:ye, xs:xe] = a == b
|
| 108 |
+
mask[k, ys:ye, xs:xe] = 1 # après fill_holes, tous les pixels ont une région
|
| 109 |
+
return aff, mask
|
| 110 |
+
|
| 111 |
+
|
| 112 |
+
@njit
|
| 113 |
+
def _find(parent, x):
|
| 114 |
+
while parent[x] != x:
|
| 115 |
+
parent[x] = parent[parent[x]]
|
| 116 |
+
x = parent[x]
|
| 117 |
+
return x
|
| 118 |
+
|
| 119 |
+
|
| 120 |
+
@njit
|
| 121 |
+
def _mws_core(n, u, v, rep):
|
| 122 |
+
parent = np.arange(n); size = np.ones(n, np.int64); m = len(u)
|
| 123 |
+
mutex_a = np.empty(m, np.int64); mutex_b = np.empty(m, np.int64); nm = 0
|
| 124 |
+
head = -np.ones(n, np.int64); nxt = np.empty(2 * m, np.int64); val = np.empty(2 * m, np.int64); nl = 0
|
| 125 |
+
for e in range(m):
|
| 126 |
+
ra, rb = _find(parent, u[e]), _find(parent, v[e])
|
| 127 |
+
if ra == rb:
|
| 128 |
+
continue
|
| 129 |
+
if rep[e]:
|
| 130 |
+
val[nl] = nm; nxt[nl] = head[ra]; head[ra] = nl; nl += 1
|
| 131 |
+
val[nl] = nm; nxt[nl] = head[rb]; head[rb] = nl; nl += 1
|
| 132 |
+
mutex_a[nm] = ra; mutex_b[nm] = rb; nm += 1
|
| 133 |
+
else:
|
| 134 |
+
small, other = (ra, rb) if size[ra] <= size[rb] else (rb, ra)
|
| 135 |
+
blocked = False; p = head[small]
|
| 136 |
+
while p != -1:
|
| 137 |
+
k = val[p]; x = _find(parent, mutex_a[k]); y = _find(parent, mutex_b[k])
|
| 138 |
+
if (x == small and y == other) or (x == other and y == small):
|
| 139 |
+
blocked = True; break
|
| 140 |
+
p = nxt[p]
|
| 141 |
+
if blocked:
|
| 142 |
+
continue
|
| 143 |
+
parent[small] = other; size[other] += size[small]; p = head[small]
|
| 144 |
+
if p != -1:
|
| 145 |
+
while nxt[p] != -1:
|
| 146 |
+
p = nxt[p]
|
| 147 |
+
nxt[p] = head[other]; head[other] = head[small]
|
| 148 |
+
out = np.empty(n, np.int64)
|
| 149 |
+
for i in range(n):
|
| 150 |
+
out[i] = _find(parent, i)
|
| 151 |
+
return out
|
| 152 |
+
|
| 153 |
+
|
| 154 |
+
def mutex_watershed(aff, stride=CFG["mws_stride"], min_size=CFG["mws_min_size"]):
|
| 155 |
+
_, H, W = aff.shape
|
| 156 |
+
idx = np.arange(H * W).reshape(H, W)
|
| 157 |
+
us, vs, ws, rep = [], [], [], []
|
| 158 |
+
for k, (dy, dx) in enumerate(OFFSETS):
|
| 159 |
+
a = idx[max(0, -dy):H - max(0, dy), max(0, -dx):W - max(0, dx)]
|
| 160 |
+
b = idx[max(0, dy):H - max(0, -dy), max(0, dx):W - max(0, -dx)]
|
| 161 |
+
w = aff[k, max(0, -dy):H - max(0, dy), max(0, -dx):W - max(0, dx)].copy()
|
| 162 |
+
if k >= N_SHORT:
|
| 163 |
+
a, b, w = a[::stride, ::stride], b[::stride, ::stride], 1 - w[::stride, ::stride]
|
| 164 |
+
us.append(a.ravel()); vs.append(b.ravel()); ws.append(w.ravel()); rep.append(np.full(a.size, k >= N_SHORT))
|
| 165 |
+
u, v, w, rep = map(np.concatenate, (us, vs, ws, rep))
|
| 166 |
+
order = np.argsort(-w, kind="stable")
|
| 167 |
+
lab = _mws_core(H * W, u[order], v[order], rep[order]).reshape(H, W)
|
| 168 |
+
_, lab = np.unique(lab, return_inverse=True); lab = lab.reshape(H, W) + 1
|
| 169 |
+
sizes = np.bincount(lab.ravel()); tiny = sizes < min_size; tiny[0] = False
|
| 170 |
+
if tiny[lab].any():
|
| 171 |
+
_, (iy, ix) = ndi.distance_transform_edt(tiny[lab], return_indices=True)
|
| 172 |
+
lab = lab[iy, ix]; _, lab = np.unique(lab, return_inverse=True); lab = lab.reshape(H, W) + 1
|
| 173 |
+
return lab.astype(np.int32)
|
| 174 |
+
|
| 175 |
+
|
| 176 |
+
# ------------------------------------------------------------------ augmentations
|
| 177 |
+
def color_aug(rgb, rng):
|
| 178 |
+
"""Casse le lien 'couleur = unité' : rotation de teinte, saturation, gains, permutation, gris, contraste, bruit."""
|
| 179 |
+
x = rgb.astype(np.float32) / 255
|
| 180 |
+
if rng.random() < 0.8:
|
| 181 |
+
T = np.array([[0.299, 0.587, 0.114], [0.596, -0.274, -0.322], [0.211, -0.523, 0.312]], np.float32)
|
| 182 |
+
a = rng.uniform(-np.pi, np.pi)
|
| 183 |
+
Rm = np.array([[1, 0, 0], [0, np.cos(a), -np.sin(a)], [0, np.sin(a), np.cos(a)]], np.float32)
|
| 184 |
+
Rm[1:, 1:] *= rng.uniform(0.6, 1.3)
|
| 185 |
+
x = x @ (np.linalg.inv(T) @ Rm @ T).T
|
| 186 |
+
if rng.random() < 0.3:
|
| 187 |
+
x = x[..., rng.permutation(3)]
|
| 188 |
+
x = x * rng.uniform(0.85, 1.15, 3) + rng.uniform(-0.06, 0.06, 3)
|
| 189 |
+
if rng.random() < 0.1:
|
| 190 |
+
x = np.repeat(x.mean(-1, keepdims=True), 3, -1)
|
| 191 |
+
m = x.mean()
|
| 192 |
+
x = (x - m) * rng.uniform(0.75, 1.2) + m
|
| 193 |
+
if rng.random() < 0.3:
|
| 194 |
+
x = x + rng.normal(0, rng.uniform(0.005, 0.03), x.shape)
|
| 195 |
+
return np.clip(x * 255, 0, 255).astype(np.uint8)
|
| 196 |
+
|
| 197 |
+
|
| 198 |
+
class TrainSet(Dataset):
|
| 199 |
+
def __len__(self):
|
| 200 |
+
return STEPS * BATCH
|
| 201 |
+
|
| 202 |
+
def __getitem__(self, i):
|
| 203 |
+
rng = np.random.default_rng([i, 20261112])
|
| 204 |
+
pool = REAL_IDX if (REAL_IDX and rng.random() < P_REAL) else SYN_IDX
|
| 205 |
+
rgb, inst, flt = decode(TRAIN[pool[rng.integers(len(pool))]])
|
| 206 |
+
inst = fill_holes(inst) # sur la tuile entière, avant recadrage
|
| 207 |
+
y, x = rng.integers(0, rgb.shape[0] - CROP + 1), rng.integers(0, rgb.shape[1] - CROP + 1)
|
| 208 |
+
rgb, inst, flt = (a[y:y + CROP, x:x + CROP] for a in (rgb, inst, flt))
|
| 209 |
+
k = int(rng.integers(4))
|
| 210 |
+
rgb, inst, flt = (np.rot90(a, k) for a in (rgb, inst, flt))
|
| 211 |
+
if rng.random() < 0.5:
|
| 212 |
+
rgb, inst, flt = rgb[:, ::-1], inst[:, ::-1], flt[:, ::-1]
|
| 213 |
+
rgb = color_aug(rgb, rng)
|
| 214 |
+
inst = np.ascontiguousarray(inst)
|
| 215 |
+
aff, mask = affinity_targets(inst)
|
| 216 |
+
if CFG["small_w"]: # les arêtes touchant un petit polygone pèsent plus (jusqu'à x8)
|
| 217 |
+
area = np.bincount(inst.ravel())
|
| 218 |
+
w = np.clip(np.sqrt(2000.0 / np.maximum(area[inst], 1)), 1.0, 8.0).astype(np.float32)
|
| 219 |
+
H, W = inst.shape
|
| 220 |
+
for k, (dy, dx) in enumerate(OFFSETS):
|
| 221 |
+
ys, ye, xs, xe = max(0, -dy), H - max(0, dy), max(0, -dx), W - max(0, dx)
|
| 222 |
+
mask[k, ys:ye, xs:xe] = np.maximum(w[ys:ye, xs:xe], w[ys + dy:ye + dy, xs + dx:xe + dx])
|
| 223 |
+
return (torch.from_numpy(np.ascontiguousarray(rgb).transpose(2, 0, 1).copy()), torch.from_numpy(aff),
|
| 224 |
+
torch.from_numpy(mask), torch.from_numpy(np.ascontiguousarray(flt, np.float32))[None])
|
| 225 |
+
|
| 226 |
+
|
| 227 |
+
# ------------------------------------------------------------------ modèle
|
| 228 |
+
def rgb2lab(x):
|
| 229 |
+
"""x : B x 3 x H x W dans [0,1] -> Lab normalisé (L/50-1, a/64, b/64)."""
|
| 230 |
+
lin = torch.where(x <= 0.04045, x / 12.92, ((x + 0.055) / 1.055) ** 2.4)
|
| 231 |
+
M = torch.tensor([[0.4124, 0.3576, 0.1805], [0.2126, 0.7152, 0.0722], [0.0193, 0.1192, 0.9505]], device=x.device)
|
| 232 |
+
xyz = torch.einsum("ij,bjhw->bihw", M, lin) / torch.tensor([0.95047, 1.0, 1.08883], device=x.device).view(1, 3, 1, 1)
|
| 233 |
+
f = torch.where(xyz > 0.008856, xyz.clamp(min=1e-6) ** (1 / 3), 7.787 * xyz + 16 / 116)
|
| 234 |
+
L, a, b = 116 * f[:, 1] - 16, 500 * (f[:, 0] - f[:, 1]), 200 * (f[:, 1] - f[:, 2])
|
| 235 |
+
return torch.stack([L / 50 - 1, a / 64, b / 64], 1)
|
| 236 |
+
|
| 237 |
+
|
| 238 |
+
class GeoCartoNet(nn.Module):
|
| 239 |
+
"""Entrée : image uint8 B x 3 x H x W (H, W multiples de 32). Sortie : K affinités + 1 carte de failles (logits)."""
|
| 240 |
+
|
| 241 |
+
def __init__(self, cfg, pretrained=True):
|
| 242 |
+
super().__init__()
|
| 243 |
+
self.cfg = cfg
|
| 244 |
+
self.register_buffer("mean", torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1))
|
| 245 |
+
self.register_buffer("std", torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1))
|
| 246 |
+
Arch = {"unet": smp.Unet, "unetpp": smp.UnetPlusPlus}[cfg["arch"]]
|
| 247 |
+
self.net = Arch(cfg["encoder"], encoder_weights="imagenet" if pretrained else None, in_channels=6,
|
| 248 |
+
classes=len(cfg["offsets"]) + 1)
|
| 249 |
+
|
| 250 |
+
def forward(self, x8):
|
| 251 |
+
with torch.autocast(x8.device.type, enabled=False): # conversion de couleur en précision complète
|
| 252 |
+
x = x8.float() / 255
|
| 253 |
+
x = torch.cat([(x - self.mean) / self.std, rgb2lab(x)], 1)
|
| 254 |
+
return self.net(x)
|
| 255 |
+
|
| 256 |
+
|
| 257 |
+
def dice(p, t, eps=1.0, dims=None):
|
| 258 |
+
return 1 - (2 * (p * t).sum(dims) + eps) / (p.sum(dims) + t.sum(dims) + eps)
|
| 259 |
+
|
| 260 |
+
|
| 261 |
+
def loss_fn(out, aff, mask, flt):
|
| 262 |
+
bce = F.binary_cross_entropy_with_logits(out[:, :K], aff, reduction="none") * mask
|
| 263 |
+
la = bce.sum() / mask.sum().clamp(min=1)
|
| 264 |
+
if CFG["ohem"]: # + moyenne des 5 % d'arêtes les plus mal prédites
|
| 265 |
+
v = bce.flatten()
|
| 266 |
+
la = la + v.topk(max(1, v.numel() // 20)).values.mean() / mask.mean().clamp(min=1e-6)
|
| 267 |
+
p, t = (1 - torch.sigmoid(out[:, :K])) * mask, (1 - aff) * mask # Dice sur les COUPURES (classe rare)
|
| 268 |
+
la = la + dice(p, t, dims=(0, 2, 3)).mean()
|
| 269 |
+
lf = F.binary_cross_entropy_with_logits(out[:, K:], flt, pos_weight=torch.tensor(10.0, device=out.device)) \
|
| 270 |
+
+ dice(torch.sigmoid(out[:, K:]), flt)
|
| 271 |
+
return la + 0.5 * lf
|
| 272 |
+
|
| 273 |
+
|
| 274 |
+
# ------------------------------------------------------------------ évaluation
|
| 275 |
+
@torch.no_grad()
|
| 276 |
+
def predict(model, rgb):
|
| 277 |
+
x = torch.from_numpy(rgb.transpose(2, 0, 1).copy())[None].to(DEV)
|
| 278 |
+
with torch.autocast(DEV, enabled=DEV == "cuda"):
|
| 279 |
+
o = model(x)
|
| 280 |
+
o = torch.sigmoid(o.float())[0].cpu().numpy()
|
| 281 |
+
return o[:K], o[K]
|
| 282 |
+
|
| 283 |
+
|
| 284 |
+
def predict_tta(model, rgb):
|
| 285 |
+
"""Moyenne sur 4 miroirs ; chaque décalage ramené dans le repère d'origine (testé : exact)."""
|
| 286 |
+
H, W = rgb.shape[:2]
|
| 287 |
+
Y, X = np.mgrid[:H, :W]
|
| 288 |
+
idx = {tuple(o): k for k, o in enumerate(OFFSETS)}
|
| 289 |
+
acc, accf = 0, 0
|
| 290 |
+
for fy in (0, 1):
|
| 291 |
+
for fx in (0, 1):
|
| 292 |
+
im = rgb[::-1] if fy else rgb
|
| 293 |
+
im = im[:, ::-1] if fx else im
|
| 294 |
+
A, pf = predict(model, np.ascontiguousarray(im))
|
| 295 |
+
py, px = (H - 1 - Y) if fy else Y, (W - 1 - X) if fx else X
|
| 296 |
+
out = np.empty_like(A)
|
| 297 |
+
for k, (dy, dx) in enumerate(OFFSETS):
|
| 298 |
+
t = (-dy if fy else dy, -dx if fx else dx)
|
| 299 |
+
if t in idx:
|
| 300 |
+
out[k] = A[idx[t]][py, px]
|
| 301 |
+
else:
|
| 302 |
+
out[k] = A[idx[(-t[0], -t[1])]][np.clip(py + t[0], 0, H - 1), np.clip(px + t[1], 0, W - 1)]
|
| 303 |
+
acc = acc + out; accf = accf + pf[py, px]
|
| 304 |
+
return acc / 4, accf / 4
|
| 305 |
+
|
| 306 |
+
|
| 307 |
+
def tile_metrics(pred, pf, inst, flt):
|
| 308 |
+
gt = fill_holes(inst)
|
| 309 |
+
are = adapted_rand_error(gt, pred)[0]
|
| 310 |
+
split, merge = variation_of_information(gt, pred) # H(pred|vérité) = sur-découpage ; H(vérité|pred) = fusion
|
| 311 |
+
small = [0, 0]; big = [0, 0]
|
| 312 |
+
ids, area = np.unique(inst[inst > 0], return_counts=True)
|
| 313 |
+
if len(ids):
|
| 314 |
+
# IoU de chaque instance avec la région prédite qui la recouvre le plus
|
| 315 |
+
pair = inst.astype(np.int64) * (pred.max() + 1) + pred
|
| 316 |
+
pv, pc = np.unique(pair[inst > 0], return_counts=True)
|
| 317 |
+
gi, pi = pv // (pred.max() + 1), pv % (pred.max() + 1)
|
| 318 |
+
parea = np.bincount(pred.ravel())
|
| 319 |
+
order = np.lexsort((-pc, gi))
|
| 320 |
+
first = np.r_[True, gi[order][1:] != gi[order][:-1]]
|
| 321 |
+
g, inter, pa = gi[order][first], pc[order][first], parea[pi[order][first]]
|
| 322 |
+
a = area[np.searchsorted(ids, g)]
|
| 323 |
+
ok = inter / (a + pa - inter) >= 0.5
|
| 324 |
+
small = [int(ok[a < 1000].sum()), int((a < 1000).sum())]; big = [int(ok[a >= 1000].sum()), int((a >= 1000).sum())]
|
| 325 |
+
mid = [int(ok[(a >= 200) & (a < 1000)].sum()), int(((a >= 200) & (a < 1000)).sum())]
|
| 326 |
+
tiny = [int(ok[a < 200].sum()), int((a < 200).sum())]
|
| 327 |
+
surf = [float((a * ok).sum()), float(a.sum())]
|
| 328 |
+
else:
|
| 329 |
+
mid = tiny = [0, 0]; surf = [0.0, 0.0]
|
| 330 |
+
pb, gb = pf > 0.5, flt > 0
|
| 331 |
+
gd, pd_ = ndi.binary_dilation(gb, iterations=2), ndi.binary_dilation(pb, iterations=2)
|
| 332 |
+
return dict(ARAND=are, split=split, merge=merge, s_ok=small[0], s_n=small[1], b_ok=big[0], b_n=big[1],
|
| 333 |
+
t_ok=tiny[0], t_n=tiny[1], m_ok=mid[0], m_n=mid[1], surf_ok=surf[0], surf_n=surf[1],
|
| 334 |
+
tp=int((pb & gd).sum()), fp=int((pb & ~gd).sum()), fn=int((gb & ~pd_).sum()))
|
| 335 |
+
|
| 336 |
+
|
| 337 |
+
def summarize(rows):
|
| 338 |
+
d = pd.DataFrame(rows)
|
| 339 |
+
f1 = 2 * d.tp.sum() / max(2 * d.tp.sum() + d.fp.sum() + d.fn.sum(), 1) if (d.tp.sum() + d.fn.sum()) > 0 else np.nan
|
| 340 |
+
return dict(tuiles=len(d), ARAND=d.ARAND.mean(), sur_decoupage=d.split.mean(), fusion=d["merge"].mean(),
|
| 341 |
+
rappel_petites=d.s_ok.sum() / max(d.s_n.sum(), 1), rappel_grandes=d.b_ok.sum() / max(d.b_n.sum(), 1),
|
| 342 |
+
rappel_40_200=d.t_ok.sum() / max(d.t_n.sum(), 1), rappel_200_1000=d.m_ok.sum() / max(d.m_n.sum(), 1),
|
| 343 |
+
rappel_tous=(d.s_ok.sum() + d.b_ok.sum()) / max(d.s_n.sum() + d.b_n.sum(), 1),
|
| 344 |
+
surface_correcte=d.surf_ok.sum() / max(d.surf_n.sum(), 1), F1_failles=f1)
|
| 345 |
+
|
| 346 |
+
|
| 347 |
+
def evaluate(model, rows, tta=False):
|
| 348 |
+
model.eval()
|
| 349 |
+
out = []
|
| 350 |
+
for r in rows:
|
| 351 |
+
rgb, inst, flt = decode(r)
|
| 352 |
+
aff, pf = predict_tta(model, rgb) if tta else predict(model, rgb)
|
| 353 |
+
out.append(tile_metrics(mutex_watershed(aff), pf, inst, flt))
|
| 354 |
+
return out
|
| 355 |
+
|
| 356 |
+
|
| 357 |
+
def val_score(model):
|
| 358 |
+
"""Score de suivi : moyenne de l'ARAND réel et de l'ARAND synthétique (toutes sources à poids égal)."""
|
| 359 |
+
res = {k: summarize(evaluate(model, v)) for k, v in VAL.items()}
|
| 360 |
+
real = np.mean([v["ARAND"] for k, v in res.items() if k[1] == "real"])
|
| 361 |
+
syn = np.mean([v["ARAND"] for k, v in res.items() if k[1] != "real"])
|
| 362 |
+
return 0.5 * (real + syn), real, syn, res
|
| 363 |
+
|
| 364 |
+
|
| 365 |
+
# ------------------------------------------------------------------ entraînement
|
| 366 |
+
torch.manual_seed(0)
|
| 367 |
+
model = GeoCartoNet(CFG).to(DEV)
|
| 368 |
+
ema = copy.deepcopy(model).eval() # moyenne glissante des poids (souvent plus stable)
|
| 369 |
+
for p in ema.parameters():
|
| 370 |
+
p.requires_grad_(False)
|
| 371 |
+
EMA_DECAY = 0.999
|
| 372 |
+
dl = DataLoader(TrainSet(), BATCH, shuffle=False, num_workers=min(4, os.cpu_count() or 1), pin_memory=DEV == "cuda",
|
| 373 |
+
drop_last=True)
|
| 374 |
+
opt = torch.optim.AdamW(model.parameters(), LR, weight_decay=1e-4)
|
| 375 |
+
sched = torch.optim.lr_scheduler.OneCycleLR(opt, LR, total_steps=STEPS, pct_start=0.05)
|
| 376 |
+
scaler = torch.amp.GradScaler("cuda", enabled=DEV == "cuda")
|
| 377 |
+
mutex_watershed(np.random.rand(K, 64, 64).astype(np.float32)) # compilation numba
|
| 378 |
+
print(f"Paramètres : {sum(p.numel() for p in model.parameters()) / 1e6:.1f} M | {STEPS} pas, lot {BATCH}, recadrage {CROP}", flush=True)
|
| 379 |
+
|
| 380 |
+
HIST, best, t0, run_loss, n_loss = [], 1e9, time.time(), 0.0, 0
|
| 381 |
+
model.train()
|
| 382 |
+
step = 0
|
| 383 |
+
for step, batch in enumerate(dl, 1):
|
| 384 |
+
x, aff, mask, flt = (t.to(DEV, non_blocking=True) for t in batch)
|
| 385 |
+
with torch.autocast(DEV, enabled=DEV == "cuda"):
|
| 386 |
+
out = model(x)
|
| 387 |
+
loss = loss_fn(out.float(), aff, mask, flt)
|
| 388 |
+
if not torch.isfinite(loss):
|
| 389 |
+
print(f" pas {step} : perte non finie, lot ignoré", flush=True)
|
| 390 |
+
opt.zero_grad(set_to_none=True); sched.step()
|
| 391 |
+
continue
|
| 392 |
+
opt.zero_grad(set_to_none=True)
|
| 393 |
+
scaler.scale(loss).backward()
|
| 394 |
+
scaler.unscale_(opt); torch.nn.utils.clip_grad_norm_(model.parameters(), 5.0)
|
| 395 |
+
scaler.step(opt); scaler.update(); sched.step()
|
| 396 |
+
with torch.no_grad():
|
| 397 |
+
d = min(EMA_DECAY, (1 + step) / (10 + step))
|
| 398 |
+
for pe, pm in zip(ema.parameters(), model.parameters()):
|
| 399 |
+
pe.mul_(d).add_(pm.detach(), alpha=1 - d)
|
| 400 |
+
for be, bm in zip(ema.buffers(), model.buffers()):
|
| 401 |
+
be.copy_(bm)
|
| 402 |
+
run_loss += loss.item(); n_loss += 1
|
| 403 |
+
if step % 250 == 0:
|
| 404 |
+
print(f"pas {step}/{STEPS} | perte {run_loss / n_loss:.4f} | {(time.time() - t0) / 60:.0f} min", flush=True)
|
| 405 |
+
hours = (time.time() - T_START) / 3600
|
| 406 |
+
last = step == STEPS or hours > MAX_HOURS
|
| 407 |
+
if step % EVAL_EVERY == 0 or last:
|
| 408 |
+
row = dict(step=step, loss=run_loss / max(n_loss, 1), heures=round(hours, 2))
|
| 409 |
+
for tag, m in (("modele", model), ("ema", ema)):
|
| 410 |
+
sc, real, syn, _ = val_score(m)
|
| 411 |
+
row.update({f"{tag}_score": sc, f"{tag}_reel": real, f"{tag}_synth": syn})
|
| 412 |
+
if sc < best:
|
| 413 |
+
best = sc
|
| 414 |
+
torch.save({"state_dict": m.state_dict(), "cfg": CFG, "step": step, "val_score": float(sc), "poids": tag}, OUT / "best.pt")
|
| 415 |
+
row["meilleur"] = tag
|
| 416 |
+
model.train()
|
| 417 |
+
HIST.append(row); pd.DataFrame(HIST).to_csv(OUT / "history.csv", index=False)
|
| 418 |
+
print(" VALIDATION", json.dumps({k: (round(v, 4) if isinstance(v, float) else v) for k, v in row.items()}), flush=True)
|
| 419 |
+
run_loss, n_loss = 0.0, 0
|
| 420 |
+
if last:
|
| 421 |
+
if step < STEPS:
|
| 422 |
+
print(f"Arrêt au pas {step} : limite de {MAX_HOURS} h atteinte.", flush=True)
|
| 423 |
+
break
|
| 424 |
+
del dl
|
| 425 |
+
|
| 426 |
+
# ------------------------------------------------------------------ test final (meilleur point de contrôle)
|
| 427 |
+
ck = torch.load(OUT / "best.pt", map_location=DEV)
|
| 428 |
+
model = GeoCartoNet(CFG, pretrained=False).to(DEV)
|
| 429 |
+
model.load_state_dict(ck["state_dict"]); model.eval()
|
| 430 |
+
torch.save({"state_dict": {k: (v.half() if v.is_floating_point() else v) for k, v in ck["state_dict"].items()},
|
| 431 |
+
"cfg": CFG, "step": ck["step"]}, OUT / "best_fp16.pt")
|
| 432 |
+
print(f"\nMeilleur modèle : pas {ck['step']} ({ck['poids']}), score de validation {ck['val_score']:.4f}", flush=True)
|
| 433 |
+
|
| 434 |
+
test_rows = TEST if not N_TEST else spread(TEST, N_TEST)
|
| 435 |
+
res = evaluate(model, test_rows, tta=CFG["tta"])
|
| 436 |
+
T = pd.DataFrame(res).assign(continent=[r["continent"] for r in test_rows], source=[r["source"] for r in test_rows],
|
| 437 |
+
type=[r["kind"] for r in test_rows])
|
| 438 |
+
tables = []
|
| 439 |
+
for keys in (["continent"], ["source", "type"]):
|
| 440 |
+
for name, g in T.groupby(keys):
|
| 441 |
+
name = name if isinstance(name, tuple) else (name,)
|
| 442 |
+
tables.append({"groupe": " / ".join(name), **summarize(g.to_dict("records"))})
|
| 443 |
+
tables.append({"groupe": "TEST COMPLET", **summarize(res)})
|
| 444 |
+
for k, rows in HARDSETS.items():
|
| 445 |
+
rows = rows if not N_TEST else spread(rows, N_TEST)
|
| 446 |
+
tables.append({"groupe": {"hard_mono": "PIÈGE unités de même couleur", "hard_pattern": "PIÈGE unités à motifs"}[k],
|
| 447 |
+
**summarize(evaluate(model, rows, tta=CFG["tta"]))})
|
| 448 |
+
TM = pd.DataFrame(tables).round(4)
|
| 449 |
+
TM.to_csv(OUT / "test_metrics.csv", index=False)
|
| 450 |
+
pd.set_option("display.width", 220); pd.set_option("display.max_columns", 20)
|
| 451 |
+
print("\n=== RÉSULTATS SUR LE JEU DE TEST (jamais vu à l'entraînement) ==="); print(TM.to_string(index=False), flush=True)
|
| 452 |
+
json.dump({"cfg": CFG, "steps": int(ck["step"]), "val_score": float(ck["val_score"]), "test": TM.to_dict("records"),
|
| 453 |
+
"train_hours": round((time.time() - T_START) / 3600, 2)}, open(OUT / "metrics.json", "w"), indent=1, ensure_ascii=False)
|
| 454 |
+
|
| 455 |
+
# ------------------------------------------------------------------ figures
|
| 456 |
+
import matplotlib; matplotlib.use("Agg"); import matplotlib.pyplot as plt
|
| 457 |
+
picks = [("réel", spread([r for r in TEST if r["kind"] == "real"], 4))]
|
| 458 |
+
picks += [(s, spread([r for r in TEST if r["source"] == s and r["kind"] != "real"], 2)) for s in sorted({r["source"] for r in TEST})]
|
| 459 |
+
picks += [(k, spread(v, 3)) for k, v in HARDSETS.items()]
|
| 460 |
+
rows_ = [(n, r) for n, rs in picks for r in rs]
|
| 461 |
+
fig, ax = plt.subplots(3, len(rows_), figsize=(4 * len(rows_), 12))
|
| 462 |
+
for j, (n, r) in enumerate(rows_):
|
| 463 |
+
rgb, inst, flt = decode(r); aff, pf = predict(model, rgb); pred = mutex_watershed(aff)
|
| 464 |
+
ax[0, j].imshow(rgb); ax[0, j].set_title(f"{n}\n{r['source']}", fontsize=9)
|
| 465 |
+
ax[1, j].imshow(fill_holes(inst) % 20, cmap="tab20", interpolation="nearest"); ax[1, j].set_title("vérité", fontsize=9)
|
| 466 |
+
ax[2, j].imshow(pred % 20, cmap="tab20", interpolation="nearest")
|
| 467 |
+
ax[2, j].imshow(np.ma.masked_less(pf, 0.5), cmap="gray_r", interpolation="nearest"); ax[2, j].set_title("modèle", fontsize=9)
|
| 468 |
+
for a in ax.ravel():
|
| 469 |
+
a.axis("off")
|
| 470 |
+
plt.tight_layout(); plt.savefig(OUT / "test_examples.png", dpi=55); plt.close()
|
| 471 |
+
print("Fichiers :", sorted(p.name for p in OUT.iterdir()), f"| durée totale {(time.time() - T_START) / 3600:.2f} h", flush=True)
|
vectorize.py
ADDED
|
@@ -0,0 +1,56 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""GeoCarto – vectorise a georeferenced geological map raster into a GeoPackage (QGIS / ArcGIS).
|
| 2 |
+
|
| 3 |
+
python vectorize.py carte.tif legende.csv sortie.gpkg [--model best_fp16.pt] [--tta]
|
| 4 |
+
|
| 5 |
+
legende.csv : one row per legend entry, columns
|
| 6 |
+
code (required, as printed on the map), nom, age, lithologie (optional),
|
| 7 |
+
and EITHER x0,y0,x1,y1 = pixel box of the legend swatch in carte.tif (best),
|
| 8 |
+
OR couleur_hex = colour of the swatch (e.g. #f2c38a).
|
| 9 |
+
Unit names come only from text read on the map and from this legend.
|
| 10 |
+
"""
|
| 11 |
+
import argparse, os, sys
|
| 12 |
+
import numpy as np, pandas as pd, rasterio
|
| 13 |
+
from skimage.color import rgb2lab
|
| 14 |
+
HERE = os.path.dirname(os.path.abspath(__file__)); sys.path.insert(0, HERE)
|
| 15 |
+
import pipeline as PL
|
| 16 |
+
|
| 17 |
+
|
| 18 |
+
def load_legend(csv, rgb):
|
| 19 |
+
L = pd.read_csv(csv, dtype={"code": str}).fillna("")
|
| 20 |
+
lab = None
|
| 21 |
+
feats, hexes = [], []
|
| 22 |
+
for r in L.itertuples():
|
| 23 |
+
if all(hasattr(r, k) and str(getattr(r, k)) != "" for k in ("x0", "y0", "x1", "y1")):
|
| 24 |
+
x0, y0, x1, y1 = (int(float(getattr(r, k))) for k in ("x0", "y0", "x1", "y1"))
|
| 25 |
+
px = rgb[y0:y1, x0:x1].reshape(-1, 3)
|
| 26 |
+
feats.append(PL.swatch_features(rgb2lab(px[None])[0])); hexes.append("#%02x%02x%02x" % tuple(int(v) for v in np.median(px, 0)))
|
| 27 |
+
elif getattr(r, "couleur_hex", ""):
|
| 28 |
+
h = r.couleur_hex.lstrip("#"); c = np.array([[int(h[i:i + 2], 16) for i in (0, 2, 4)]], np.uint8)
|
| 29 |
+
feats.append(PL.swatch_features(np.repeat(rgb2lab(c[None])[0], 50, 0))); hexes.append("#" + h)
|
| 30 |
+
else:
|
| 31 |
+
raise SystemExit(f"légende : la ligne {r.code} n'a ni case (x0,y0,x1,y1) ni couleur_hex")
|
| 32 |
+
out = pd.DataFrame(index=L.code.values)
|
| 33 |
+
for c in ("nom", "age", "lithologie"):
|
| 34 |
+
out[c] = L[c].values if c in L else ""
|
| 35 |
+
out["nom"] = np.where(out.nom == "", out.index, out.nom)
|
| 36 |
+
out["couleur_hex"] = hexes; out["feat"] = feats
|
| 37 |
+
return out
|
| 38 |
+
|
| 39 |
+
|
| 40 |
+
if __name__ == "__main__":
|
| 41 |
+
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
| 42 |
+
ap.add_argument("raster"); ap.add_argument("legend"); ap.add_argument("out")
|
| 43 |
+
ap.add_argument("--model", default=os.path.join(HERE, "best_fp16.pt"))
|
| 44 |
+
ap.add_argument("--tta", action="store_true", help="moyenne sur 4 symétries (plus lent, pas meilleur en moyenne)")
|
| 45 |
+
a = ap.parse_args()
|
| 46 |
+
import easyocr
|
| 47 |
+
with rasterio.open(a.raster) as s:
|
| 48 |
+
rgb = np.moveaxis(s.read([1, 2, 3]), 0, -1).astype(np.uint8); tr, crs = s.transform, s.crs
|
| 49 |
+
legend = load_legend(a.legend, rgb)
|
| 50 |
+
print(f"carte {rgb.shape[1]}×{rgb.shape[0]} px | légende : {len(legend)} unités")
|
| 51 |
+
model = PL.load_model(a.model)
|
| 52 |
+
reader = easyocr.Reader(["en"], gpu=False, verbose=False)
|
| 53 |
+
P, F, lab, pf = PL.run(rgb, tr, crs, model, legend, reader, os.path.join(HERE, "naming_model.pkl"), a.out,
|
| 54 |
+
tta=a.tta, ranker_pkl=os.path.join(HERE, "legend_ranker.pkl"))
|
| 55 |
+
n = P.code.ne("").sum()
|
| 56 |
+
print(f"\n{len(P)} unités ({n} nommées, {len(P) - n} 'Unité non identifiée' avec suggestions), {len(F)} failles -> {a.out}")
|