PerfectWin commited on
Commit
efdf9ef
·
verified ·
1 Parent(s): 8e6ed15

GeoCarto model v0.2 (research preview): segmentation + naming + GeoPackage export

Browse files
.gitattributes CHANGED
@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ exemple/carte_geologique.gpkg filter=lfs diff=lfs merge=lfs -text
37
+ exemple/rendu_qgis.png filter=lfs diff=lfs merge=lfs -text
38
+ training/test_examples.png filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,101 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: cc-by-sa-4.0
3
+ library_name: pytorch
4
+ pipeline_tag: image-segmentation
5
+ datasets:
6
+ - PerfectWin/geocarto
7
+ tags:
8
+ - geology
9
+ - geological-map
10
+ - vectorization
11
+ - instance-segmentation
12
+ - gis
13
+ - qgis
14
+ ---
15
+
16
+ # GeoCarto model – geological map raster → GIS polygons, faults and unit names (GeoPackage)
17
+
18
+ **Research preview (v0.2).** Turns a scanned / rasterised geological map into a GeoPackage that opens directly in
19
+ QGIS or ArcGIS: one polygon layer of geological units (with code, name, age, lithology when they can be read),
20
+ one line layer of faults, the legend table and default QGIS styles.
21
+
22
+ Trained on [PerfectWin/geocarto](https://huggingface.co/datasets/PerfectWin/geocarto) **v2.0** (4 continents,
23
+ real USGS GMNA scans + realistic synthetic maps).
24
+
25
+ ## How it works
26
+ 1. **Segmentation** – U-Net++ / EfficientNet-B4, input RGB + CIE-Lab (6 channels), predicts 10 pixel affinities
27
+ (offsets 1–27 px) + a fault map; sliding window 512 px, stride 384, Hann blending.
28
+ 2. **Polygons** – mutex watershed on the affinities (full resolution), minimum region 40 px, vectorised with rasterio.
29
+ 3. **Faults** – skeleton of the fault map, traced into lines.
30
+ 4. **Text** – EasyOCR on the whole map.
31
+ 5. **Naming** – names come **only from text printed on the map and from the legend** (no propagation of names between
32
+ polygons that merely look alike). A learned ranker (gradient boosting) scores every (polygon, candidate code) pair
33
+ from: OCR evidence and leader-line features, distance to the legend swatch (relative to the other swatches),
34
+ where else the code is read on the map. A name is written only when the confidence is ≥ 0.8; otherwise the polygon
35
+ is `Unité non identifiée` with the 3 best ranked suggestions in the `suggestions` column.
36
+ 6. **Export** – GeoPackage with layers `unites_geologiques`, `failles`, table `legende`, styles in `layer_styles`.
37
+
38
+ ## Usage
39
+ ```bash
40
+ pip install -r requirements.txt
41
+ python vectorize.py carte.tif legende.csv sortie.gpkg
42
+ ```
43
+ `carte.tif` must be georeferenced (any CRS). `legende.csv` has one row per legend entry: `code` (as printed on the
44
+ map), optional `nom`, `age`, `lithologie`, and either the pixel box of the legend swatch in the raster
45
+ (`x0,y0,x1,y1`, recommended) or its colour `couleur_hex`. See `exemple/legende_exemple.csv`.
46
+ `qgis_test.py` checks a produced GeoPackage with PyQGIS (layers, styles, geometry validity, rendering).
47
+
48
+ Attribute table of `unites_geologiques`: `id, code, nom, age, lithologie, source_nom` (code read on the map / legend
49
+ colour / not identified), `confiance, texte_lu, suggestions, couleur_hex` (legend colour), `couleur_carte` (colour
50
+ measured on the scan), `surface_km2`.
51
+
52
+ ## Results (held-out test split, never seen in training, 608 tiles)
53
+ Recall = share of reference polygons recovered with IoU ≥ 0.5. *Small* < 1000 px, *large* ≥ 1000 px (512 px tiles).
54
+ *Correct area* = share of the map area that falls in a correctly recovered polygon.
55
+
56
+ | group | ARAND ↓ | small recall | large recall | correct area | fault F1 (2 px) |
57
+ |---|---|---|---|---|---|
58
+ | Africa (synthetic) | 0.030 | 89.6 % | 97.9 % | 99.3 % | – |
59
+ | Europe (synthetic) | 0.099 | 82.2 % | 90.4 % | 92.0 % | – |
60
+ | North America (all) | 0.091 | 72.5 % | 90.5 % | 95.1 % | 0.95 |
61
+ | Oceania (synthetic) | 0.176 | 63.9 % | 75.6 % | 83.5 % | 0.99 |
62
+ | **GMNA real scans** | 0.052 | **74.7 %** | **93.4 %** | **98.3 %** | 0.63 |
63
+ | **full test set** | 0.107 | 70.5 % | 86.2 % | 92.1 % | 0.97 |
64
+ | trap set: neighbouring units with the same colour | 0.151 | 62.8 % | 76.9 % | 84.9 % | 0.97 |
65
+ | trap set: patterned units | 0.099 | 74.3 % | 89.8 % | 93.9 % | 0.98 |
66
+
67
+ Full table: `training/test_metrics.csv`; ablation of the training choices: `training/lab_round4_ablation.csv`
68
+ (small-polygon loss re-weighting and OHEM were tested and rejected: they over-segment real maps).
69
+
70
+ ### Naming (real GMNA scans, map sheet never used to train the ranker)
71
+ End-to-end test (segmentation by this model + OCR + naming) on 9 real 2048 × 2048 px windows of the GMNA north sheet;
72
+ legend = all units of the sheet, swatches measured outside the tested window; ranker trained on the south sheet only.
73
+
74
+ | | value |
75
+ |---|---|
76
+ | area that receives a name (confidence ≥ 0.8) | **30.0 %** (per window 4 % – 57 %) |
77
+ | name correct, on the named area | **91.6 %** (per window 76 % – 98 %) |
78
+ | area correctly named, over the whole map | 27.5 % |
79
+
80
+ Cross-sheet benchmark on reference polygons (38 windows, both directions): names given at confidence ≥ 0.8 are
81
+ 94–96 % correct on ~1/3 of the area. The rest of the area is mostly units without a legible label in the window.
82
+ Per-window results: `training/e2e_naming_9_zones.csv`. Example output: `exemple/carte_geologique.gpkg`
83
+ (rendered in QGIS: `exemple/rendu_qgis.png`).
84
+
85
+ ## Limitations
86
+ * Small units: about 1 in 4 small polygons on real scans is still missed or merged.
87
+ * Naming depends on labels being printed and legible; unlabelled units are left as *Unité non identifiée* with
88
+ suggestions rather than guessed. Text recognition is trained for Latin script codes.
89
+ * Real-scan validation is so far only on the USGS Geologic Map of North America; other continents are evaluated on
90
+ realistic synthetic renderings of official vector maps.
91
+ * Fault detection on real scans (F1 0.63) is much weaker than on synthetic maps.
92
+ * Large maps are processed in one piece for the watershed: keep inputs below ~6000 × 6000 px or split them.
93
+
94
+ ## Files
95
+ `best_fp16.pt` weights (step 22 500) · `gc_model.py` network + mutex watershed · `pipeline.py` full pipeline ·
96
+ `naming_model.py`, `leader.py`, `naming_model.pkl` OCR-to-polygon model · `legend_ranker.pkl` naming ranker ·
97
+ `vectorize.py` command line · `qml_*.xml` QGIS styles · `training/` training script, curves and metrics.
98
+ The `.pkl` files are scikit-learn pickles: only load files from this repository.
99
+
100
+ ## Citation
101
+ Please cite the dataset sources listed in the [dataset card](https://huggingface.co/datasets/PerfectWin/geocarto).
best_fp16.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fe12b55b0b460bafe50628b32372c857b3d446b6a5b329a2ad0274b2b0bb5f54
3
+ size 42158475
exemple/carte_geologique.gpkg ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5d49b7700611a18c8d02636eae0bdc6485eb661d5d625424eceae6b8ccc18a20
3
+ size 557056
exemple/legende_exemple.csv ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ code,nom,age,lithologie,couleur_hex
2
+ X²,X²,Early Proterozoic – Early Proterozoic,Sedimentary,#bda8c6
3
+ X²g,X²g,Early Proterozoic – Early Proterozoic,Plutonic undivided granitic rocks,#d96288
4
+ At,At,Archean – Archean,Plutonic tonalite,#efadcf
5
+ Agn,Agn,Archean – Archean,Metamorphic and undivided crystalline orthogneiss,#ebd0df
6
+ Wtn,Wtn,Late Archean – Late Archean,Metamorphic and undivided crystalline mainly tonalite gneiss,#aaddf1
exemple/rendu_qgis.png ADDED

Git LFS Details

  • SHA256: 3f04cbd9d8ee9ee2ebd2a5def850098439ae1ef4fd6298d4e4d820e40f95d99a
  • Pointer size: 131 Bytes
  • Size of remote file: 264 kB
gc_model.py ADDED
@@ -0,0 +1,127 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Définitions du modèle GeoCarto (extraites de train_final.py) pour l'inférence hors Kaggle.
2
+ import numpy as np, torch, torch.nn as nn
3
+ from scipy import ndimage as ndi
4
+ from numba import njit
5
+ import segmentation_models_pytorch as smp
6
+
7
+ CFG = dict(arch="unetpp", encoder="efficientnet-b4", input="rgb+lab",
8
+ offsets=[(1, 0), (0, 1), (3, 0), (0, 3), (9, 0), (0, 9), (9, 9), (9, -9), (27, 0), (0, 27)],
9
+ n_short=2, mws_stride=1, mws_min_size=40, tile=512)
10
+ OFFSETS, N_SHORT, K = CFG["offsets"], CFG["n_short"], len(CFG["offsets"])
11
+
12
+ # ------------------------------------------------------------------ affinités + mutex watershed
13
+ def fill_holes(inst):
14
+ """Chaque zone sans polygone (mer, lac, halo) devient sa propre région."""
15
+ holes, n = ndi.label(inst == 0)
16
+ if n == 0:
17
+ return inst
18
+ out = inst.copy()
19
+ out[holes > 0] = holes[holes > 0] + inst.max()
20
+ return out
21
+
22
+
23
+ def affinity_targets(inst):
24
+ H, W = inst.shape
25
+ aff, mask = np.zeros((K, H, W), np.float32), np.zeros((K, H, W), np.float32)
26
+ for k, (dy, dx) in enumerate(OFFSETS):
27
+ ys, ye, xs, xe = max(0, -dy), H - max(0, dy), max(0, -dx), W - max(0, dx)
28
+ a, b = inst[ys:ye, xs:xe], inst[ys + dy:ye + dy, xs + dx:xe + dx]
29
+ aff[k, ys:ye, xs:xe] = a == b
30
+ mask[k, ys:ye, xs:xe] = 1 # après fill_holes, tous les pixels ont une région
31
+ return aff, mask
32
+
33
+
34
+ @njit
35
+ def _find(parent, x):
36
+ while parent[x] != x:
37
+ parent[x] = parent[parent[x]]
38
+ x = parent[x]
39
+ return x
40
+
41
+
42
+ @njit
43
+ def _mws_core(n, u, v, rep):
44
+ parent = np.arange(n); size = np.ones(n, np.int64); m = len(u)
45
+ mutex_a = np.empty(m, np.int64); mutex_b = np.empty(m, np.int64); nm = 0
46
+ head = -np.ones(n, np.int64); nxt = np.empty(2 * m, np.int64); val = np.empty(2 * m, np.int64); nl = 0
47
+ for e in range(m):
48
+ ra, rb = _find(parent, u[e]), _find(parent, v[e])
49
+ if ra == rb:
50
+ continue
51
+ if rep[e]:
52
+ val[nl] = nm; nxt[nl] = head[ra]; head[ra] = nl; nl += 1
53
+ val[nl] = nm; nxt[nl] = head[rb]; head[rb] = nl; nl += 1
54
+ mutex_a[nm] = ra; mutex_b[nm] = rb; nm += 1
55
+ else:
56
+ small, other = (ra, rb) if size[ra] <= size[rb] else (rb, ra)
57
+ blocked = False; p = head[small]
58
+ while p != -1:
59
+ k = val[p]; x = _find(parent, mutex_a[k]); y = _find(parent, mutex_b[k])
60
+ if (x == small and y == other) or (x == other and y == small):
61
+ blocked = True; break
62
+ p = nxt[p]
63
+ if blocked:
64
+ continue
65
+ parent[small] = other; size[other] += size[small]; p = head[small]
66
+ if p != -1:
67
+ while nxt[p] != -1:
68
+ p = nxt[p]
69
+ nxt[p] = head[other]; head[other] = head[small]
70
+ out = np.empty(n, np.int64)
71
+ for i in range(n):
72
+ out[i] = _find(parent, i)
73
+ return out
74
+
75
+
76
+ def mutex_watershed(aff, stride=CFG["mws_stride"], min_size=CFG["mws_min_size"]):
77
+ _, H, W = aff.shape
78
+ idx = np.arange(H * W).reshape(H, W)
79
+ us, vs, ws, rep = [], [], [], []
80
+ for k, (dy, dx) in enumerate(OFFSETS):
81
+ a = idx[max(0, -dy):H - max(0, dy), max(0, -dx):W - max(0, dx)]
82
+ b = idx[max(0, dy):H - max(0, -dy), max(0, dx):W - max(0, -dx)]
83
+ w = aff[k, max(0, -dy):H - max(0, dy), max(0, -dx):W - max(0, dx)].copy()
84
+ if k >= N_SHORT:
85
+ a, b, w = a[::stride, ::stride], b[::stride, ::stride], 1 - w[::stride, ::stride]
86
+ us.append(a.ravel()); vs.append(b.ravel()); ws.append(w.ravel()); rep.append(np.full(a.size, k >= N_SHORT))
87
+ u, v, w, rep = map(np.concatenate, (us, vs, ws, rep))
88
+ order = np.argsort(-w, kind="stable")
89
+ lab = _mws_core(H * W, u[order], v[order], rep[order]).reshape(H, W)
90
+ _, lab = np.unique(lab, return_inverse=True); lab = lab.reshape(H, W) + 1
91
+ sizes = np.bincount(lab.ravel()); tiny = sizes < min_size; tiny[0] = False
92
+ if tiny[lab].any():
93
+ _, (iy, ix) = ndi.distance_transform_edt(tiny[lab], return_indices=True)
94
+ lab = lab[iy, ix]; _, lab = np.unique(lab, return_inverse=True); lab = lab.reshape(H, W) + 1
95
+ return lab.astype(np.int32)
96
+
97
+
98
+ # ------------------------------------------------------------------ modèle
99
+ def rgb2lab(x):
100
+ """x : B x 3 x H x W dans [0,1] -> Lab normalisé (L/50-1, a/64, b/64)."""
101
+ lin = torch.where(x <= 0.04045, x / 12.92, ((x + 0.055) / 1.055) ** 2.4)
102
+ M = torch.tensor([[0.4124, 0.3576, 0.1805], [0.2126, 0.7152, 0.0722], [0.0193, 0.1192, 0.9505]], device=x.device)
103
+ xyz = torch.einsum("ij,bjhw->bihw", M, lin) / torch.tensor([0.95047, 1.0, 1.08883], device=x.device).view(1, 3, 1, 1)
104
+ f = torch.where(xyz > 0.008856, xyz.clamp(min=1e-6) ** (1 / 3), 7.787 * xyz + 16 / 116)
105
+ L, a, b = 116 * f[:, 1] - 16, 500 * (f[:, 0] - f[:, 1]), 200 * (f[:, 1] - f[:, 2])
106
+ return torch.stack([L / 50 - 1, a / 64, b / 64], 1)
107
+
108
+
109
+ class GeoCartoNet(nn.Module):
110
+ """Entrée : image uint8 B x 3 x H x W (H, W multiples de 32). Sortie : K affinités + 1 carte de failles (logits)."""
111
+
112
+ def __init__(self, cfg, pretrained=True):
113
+ super().__init__()
114
+ self.cfg = cfg
115
+ self.register_buffer("mean", torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1))
116
+ self.register_buffer("std", torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1))
117
+ Arch = {"unet": smp.Unet, "unetpp": smp.UnetPlusPlus}[cfg["arch"]]
118
+ self.net = Arch(cfg["encoder"], encoder_weights="imagenet" if pretrained else None, in_channels=6,
119
+ classes=len(cfg["offsets"]) + 1)
120
+
121
+ def forward(self, x8):
122
+ with torch.autocast(x8.device.type, enabled=False): # conversion de couleur en précision complète
123
+ x = x8.float() / 255
124
+ x = torch.cat([(x - self.mean) / self.std, rgb2lab(x)], 1)
125
+ return self.net(x)
126
+
127
+
leader.py ADDED
@@ -0,0 +1,79 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Détection des traits de rappel : trait sombre fin qui part du bord d'une étiquette, dans n'importe quelle direction.
2
+ import numpy as np
3
+ from scipy import ndimage as ndi
4
+ from skimage.segmentation import find_boundaries
5
+
6
+
7
+ def prepare(reg):
8
+ """Masque des pixels 'trait candidat' : sombres, hors contacts, hors failles. Calculé une fois par zone."""
9
+ if "lead_mask" in reg:
10
+ return
11
+ lum, inst = reg["lum"], reg["inst"]
12
+ bd = find_boundaries(inst, mode="thick")
13
+ nearbd = ndi.binary_dilation(bd, iterations=1)
14
+ flt = reg.get("flt")
15
+ nf = ndi.binary_dilation(flt > 0, iterations=3) if flt is not None else np.zeros_like(bd)
16
+ rgb = reg["rgb"].astype(np.int16)
17
+ sat = rgb.max(-1) - rgb.min(-1)
18
+ dark = (lum < 100) & (sat < 70) # trait noir/gris (pas de texte ou de contour coloré)
19
+ reg["lead_dark"] = dark
20
+ reg["lead_mask"] = dark & ~nearbd & ~nf
21
+
22
+
23
+ def trace(reg, wd, max_len=160, gap=6):
24
+ """Renvoie la liste des extrémités (y, x, longueur) des traits partant de la boîte du mot."""
25
+ prepare(reg)
26
+ M, D, inst = reg["lead_mask"], reg["lead_dark"], reg["inst"]
27
+ N = inst.shape[0]
28
+ x0, y0, x1, y1 = int(max(wd.x0, 0)), int(max(wd.y0, 0)), int(min(wd.x1, N - 1)), int(min(wd.y1, N - 1))
29
+ P = 14
30
+ Y0, X0, Y1, X1 = max(y0 - P, 0), max(x0 - P, 0), min(y1 + P, N - 1), min(x1 + P, N - 1)
31
+ sub = M[Y0:Y1 + 1, X0:X1 + 1].copy()
32
+ # on enlève la boîte de texte (+1 px) : l'encre des lettres n'est pas un trait
33
+ sub[max(y0 - 1 - Y0, 0):y1 + 2 - Y0, max(x0 - 1 - X0, 0):x1 + 2 - X0] = False
34
+ lab, n = ndi.label(sub, structure=np.ones((3, 3)))
35
+ if n == 0:
36
+ return []
37
+ yy, xx = np.nonzero(lab)
38
+ ll = lab[yy, xx]
39
+ gy, gx = yy + Y0, xx + X0
40
+ # distance à la boîte
41
+ db = np.hypot(np.maximum(np.maximum(y0 - gy, gy - y1), 0), np.maximum(np.maximum(x0 - gx, gx - x1), 0))
42
+ out = []
43
+ for c in range(1, n + 1):
44
+ s = ll == c
45
+ if s.sum() < 5 or db[s].min() > 2.5: # doit toucher (presque) la boîte
46
+ continue
47
+ py, px, d = gy[s].astype(float), gx[s].astype(float), db[s]
48
+ # un trait est fin et allongé
49
+ cov = np.cov(np.stack([py, px]))
50
+ ev, evec = np.linalg.eigh(cov)
51
+ v = evec[:, 1]
52
+ if ev[1] < 9: # trop court (< ~10 px)
53
+ continue
54
+ res = np.abs((py - py.mean()) * v[1] - (px - px.mean()) * v[0])
55
+ if np.percentile(res, 90) > 1.3: # pas droit ou trop épais
56
+ continue
57
+ a = np.argmin(d); b = np.argmax(d)
58
+ if d[b] - d[a] < 4: # doit s'éloigner de la boîte
59
+ continue
60
+ if np.dot(v, [py[b] - py[a], px[b] - px[a]]) < 0:
61
+ v = -v
62
+ # suivi dans la direction, au-delà du morceau visible, en tolérant les coupures aux contacts
63
+ y, x = py[b], px[b]; miss = 0; L = float(np.hypot(py[b] - py[a], px[b] - px[a]))
64
+ ey, ex = y, x
65
+ while L < max_len:
66
+ y += v[0]; x += v[1]; L += 1
67
+ iy, ix = int(round(y)), int(round(x))
68
+ if not (0 <= iy < N and 0 <= ix < N):
69
+ break
70
+ if D[max(iy - 1, 0):iy + 2, max(ix - 1, 0):ix + 2].any():
71
+ miss = 0; ey, ex = y, x
72
+ else:
73
+ miss += 1
74
+ if miss > gap:
75
+ break
76
+ # pointe : on avance de 2 px au-delà de la fin (le trait se termine DANS le polygone visé)
77
+ ty, tx = int(np.clip(round(ey + 2 * v[0]), 0, N - 1)), int(np.clip(round(ex + 2 * v[1]), 0, N - 1))
78
+ out.append((ty, tx, float(np.hypot(ty - py[a], tx - px[a]))))
79
+ return out
legend_ranker.pkl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0b99fd9736d2bd63a138a9dd12c7edfe2d46128e9b7454d6c1aeea7166be7a15
3
+ size 550205
naming_model.pkl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c9e2a5367d325a5b4455073597354f815453042a44c604ae82050a5b42ac5046
3
+ size 838928
naming_model.py ADDED
@@ -0,0 +1,203 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # =====================================================================================
2
+ # MODULE DE NOMMAGE : à quel polygone appartient chaque code lu sur la carte ?
3
+ # Module séparé du modèle de segmentation. Pour chaque code lu (OCR), on note chaque polygone
4
+ # voisin avec des indices simples (distance, trait de rappel sombre, place disponible pour le
5
+ # texte, recouvrement...) et un petit modèle appris sur de VRAIES étiquettes GMNA choisit.
6
+ # Apprentissage : zones "south" ; test : zones "north" (jamais vues).
7
+ # =====================================================================================
8
+ import sys, glob, pickle, re, difflib, time
9
+ import numpy as np, pandas as pd
10
+ from scipy import ndimage as ndi
11
+ from sklearn.ensemble import HistGradientBoostingClassifier
12
+
13
+ RADIUS = 110 # rayon de recherche des polygones candidats autour d'une étiquette (px)
14
+ norm = lambda c: re.sub(r"[\*\?\s]", "", str(c))
15
+ import os
16
+ if os.path.exists("naming_cache/vocab.pkl"):
17
+ _codes = pickle.load(open("naming_cache/vocab.pkl", "rb"))
18
+ else: # lecture encore en cours : vocabulaire = codes des zones déjà lues
19
+ _codes = [c for f in glob.glob("naming_cache/*_*.pkl") for c in pickle.load(open(f, "rb"))["codes"]]
20
+ VOCAB = sorted({norm(c) for c in _codes} - {""})
21
+ VSET = set(VOCAB)
22
+
23
+
24
+ def to_code(txt):
25
+ t = re.sub(r"[^A-Za-z0-9€Є³]", "", txt).replace("€", "Є")
26
+ if not t:
27
+ return None, 0.0
28
+ if t in VSET:
29
+ return t, 1.0
30
+ m = difflib.get_close_matches(t, VOCAB, n=1, cutoff=0.75)
31
+ return (m[0], 0.75) if m else (None, 0.0)
32
+
33
+
34
+ FEATS = ["dist_centre", "dist_boite", "sous_centre", "part_boite", "rang_dist", "n_cand", "aire_log", "rayon_int",
35
+ "rayon_sur_texte", "trait_sombre", "trait_lum", "limites_croisees", "long_trait", "h_texte", "l_texte",
36
+ "conf", "qualite_code", "dist_rel", "plus_proche_non_sous", "part_boite_max_autre", "vise_trait", "trait_L"]
37
+
38
+
39
+ def candidates(reg, wd):
40
+ """Lignes de caractéristiques (une par polygone candidat) pour un mot lu."""
41
+ inst, lum = reg["inst"], reg["lum"]
42
+ N = inst.shape[0]
43
+ x0, y0, x1, y1 = int(max(wd.x0, 0)), int(max(wd.y0, 0)), int(min(wd.x1, N - 1)), int(min(wd.y1, N - 1))
44
+ if x1 - x0 < 2 or y1 - y0 < 2:
45
+ return []
46
+ cx, cy = (x0 + x1) / 2, (y0 + y1) / 2
47
+ X0, Y0, X1, Y1 = max(x0 - RADIUS, 0), max(y0 - RADIUS, 0), min(x1 + RADIUS, N - 1), min(y1 + RADIUS, N - 1)
48
+ sub, slum = inst[Y0:Y1 + 1, X0:X1 + 1], lum[Y0:Y1 + 1, X0:X1 + 1]
49
+ yy, xx = np.mgrid[Y0:Y1 + 1, X0:X1 + 1]
50
+ dc = np.hypot(yy - cy, xx - cx) # distance au centre de la boîte
51
+ db = np.hypot(np.maximum(np.maximum(y0 - yy, yy - y1), 0), np.maximum(np.maximum(x0 - xx, xx - x1), 0)) # à la boîte
52
+ ids = np.unique(sub); ids = ids[ids > 0]
53
+ if not len(ids):
54
+ return []
55
+ box = sub[y0 - Y0:y1 - Y0 + 1, x0 - X0:x1 - X0 + 1]
56
+ light = slum[y0 - Y0:y1 - Y0 + 1, x0 - X0:x1 - X0 + 1] >= 120 # hors encre du texte
57
+ nb = max(light.sum(), 1)
58
+ under = inst[int(cy), int(cx)]
59
+ th, tw = y1 - y0, x1 - x0
60
+ dmin_c = ndi.minimum(dc, sub, ids); dmin_b = ndi.minimum(db, sub, ids)
61
+ pos_b = ndi.minimum_position(db, sub, ids)
62
+ edt = ndi.distance_transform_edt(sub > 0) if False else None
63
+ from leader import trace
64
+ tips = trace(reg, wd)
65
+ rows = []
66
+ order = np.argsort(dmin_b)
67
+ rank = np.empty(len(ids)); rank[order] = np.arange(len(ids))
68
+ part = np.array([(box[light] == i).sum() / nb for i in ids])
69
+ for j, i in enumerate(ids):
70
+ m = sub == i
71
+ # rayon intérieur du polygone (dans la fenêtre) : peut-il contenir le texte ?
72
+ rin = ndi.distance_transform_edt(np.pad(m, 1))[1:-1, 1:-1].max()
73
+ # segment droit boîte -> point le plus proche du polygone : un trait de rappel est sombre et droit
74
+ py, px = pos_b[j][0] + Y0, pos_b[j][1] + X0
75
+ qy, qx = min(max(py, y0), y1), min(max(px, x0), x1)
76
+ L = float(np.hypot(py - qy, px - qx))
77
+ if L >= 3:
78
+ n = int(L) + 1
79
+ sy = np.clip(np.round(np.linspace(qy, py, n)).astype(int), 0, N - 1)
80
+ sx = np.clip(np.round(np.linspace(qx, px, n)).astype(int), 0, N - 1)
81
+ ls = np.minimum.reduce([lum[np.clip(sy + a, 0, N - 1), np.clip(sx + b, 0, N - 1)] for a in (-1, 0, 1) for b in (-1, 0, 1)])
82
+ dark, lmean = float((ls[1:-1] < 120).mean()) if n > 2 else 0.0, float(ls.mean())
83
+ cross = int((np.diff(inst[sy, sx]) != 0).sum())
84
+ else:
85
+ dark, lmean, cross = 0.0, 255.0, 0
86
+ others = np.delete(part, j)
87
+ tl = [t[2] for t in tips if inst[t[0], t[1]] == i]
88
+ rows.append(dict(poly=int(i), dist_centre=float(dmin_c[j]), dist_boite=float(dmin_b[j]), sous_centre=float(i == under),
89
+ part_boite=float(part[j]), rang_dist=float(rank[j]), n_cand=float(len(ids)),
90
+ aire_log=float(np.log1p(reg["area"][i])), rayon_int=float(rin), rayon_sur_texte=float(rin / max(th / 2, 1)),
91
+ trait_sombre=dark, trait_lum=lmean, limites_croisees=float(cross), long_trait=L, h_texte=float(th),
92
+ l_texte=float(tw), conf=float(wd.conf), qualite_code=float(wd.q),
93
+ dist_rel=float(dmin_b[j] / max(th, 1)), plus_proche_non_sous=float(rank[j] == (1 if under in ids else 0) and i != under),
94
+ part_boite_max_autre=float(others.max() if len(others) else 0),
95
+ vise_trait=float(len(tl)), trait_L=float(min(tl)) if tl else -1.0))
96
+ return rows
97
+
98
+
99
+ def load_region(f):
100
+ r = pickle.load(open(f, "rb"))
101
+ r["lum"] = (r["rgb"].astype(np.float32) @ np.array([0.299, 0.587, 0.114], np.float32))
102
+ r["area"] = np.bincount(r["inst"].ravel())
103
+ r["true"] = {i + 1: norm(c) for i, c in enumerate(r["codes"])}
104
+ return r
105
+
106
+
107
+ def build(files):
108
+ out = []
109
+ for f in files:
110
+ reg = load_region(f)
111
+ W_ = reg["words"]
112
+ for k, wd in enumerate(W_.itertuples()):
113
+ code, q = to_code(wd.text)
114
+ if code is None:
115
+ continue
116
+ wd = wd._replace(conf=wd.conf) if False else wd
117
+ class _W: pass
118
+ w = _W(); w.__dict__.update(wd._asdict()); w.q = q
119
+ for row in candidates(reg, w):
120
+ row.update(region=f, word=k, code=code, y=float(reg["true"][row["poly"]] == code),
121
+ vrai=reg["true"][row["poly"]], aire=float(reg["area"][row["poly"]]))
122
+ out.append(row)
123
+ return pd.DataFrame(out)
124
+
125
+
126
+ def assign(D, score):
127
+ """Choix final : pour chaque mot le polygone le mieux noté ; un polygone ne garde qu'UN code
128
+ (celui qui a la meilleure note), les mots perdants passent à leur candidat suivant."""
129
+ D = D.assign(s=score)
130
+ choice = {}
131
+ words = {k: g.sort_values("s", ascending=False) for k, g in D.groupby(["region", "word"])}
132
+ ptr = {k: 0 for k in words}
133
+ for _ in range(4):
134
+ for k, g in words.items():
135
+ if ptr[k] < len(g):
136
+ r = g.iloc[ptr[k]]
137
+ choice[k] = (r.region, int(r.poly), r.code, float(r.s))
138
+ else:
139
+ choice.pop(k, None)
140
+ best = {}
141
+ for k, (reg, p, c, s) in choice.items():
142
+ key = (reg, p)
143
+ if key not in best or s > best[key][1]:
144
+ best[key] = (c, s)
145
+ moved = False
146
+ for k, (reg, p, c, s) in list(choice.items()):
147
+ if best[(reg, p)][0] != c and ptr[k] + 1 < len(words[k]) and words[k].iloc[ptr[k] + 1].s > 0.15:
148
+ ptr[k] += 1; moved = True
149
+ if not moved:
150
+ break
151
+ return choice
152
+
153
+
154
+ def report(D, choice, title):
155
+ key = D.set_index(["region", "word", "poly"])
156
+ evaluable = D.groupby(["region", "word"]).y.max() > 0 # le bon code existe parmi les candidats (lecture juste)
157
+ ok = n = 0
158
+ polys = {}
159
+ for k, (reg, p, c, s) in choice.items():
160
+ r = key.loc[(k[0], k[1], p)]
161
+ good = bool(np.atleast_1d(r.y)[0])
162
+ if evaluable[k]:
163
+ n += 1; ok += good
164
+ cur = polys.get((reg, p))
165
+ if cur is None or s > cur[1]:
166
+ polys[(reg, p)] = (good, s, float(np.atleast_1d(r.aire)[0]))
167
+ pg = np.array([v[0] for v in polys.values()]); pa = np.array([v[2] for v in polys.values()])
168
+ print(f"{title:<46} codes bien rattachés : {ok / max(n, 1):5.1%} ({n} codes) | polygones nommés : {len(polys)}, "
169
+ f"noms justes : {pg.mean():5.1%} (en surface : {(pa * pg).sum() / pa.sum():5.1%})")
170
+ return ok / max(n, 1), polys
171
+
172
+
173
+ if __name__ == "__main__":
174
+ t0 = time.time()
175
+ tr_files, te_files = sorted(glob.glob("naming_cache/south_*.pkl")), sorted(glob.glob("naming_cache/north_*.pkl"))
176
+ if len(te_files) < 3: # provisoire : pas encore de zones nord -> 1 zone sud sur 3 en test
177
+ allf = tr_files + te_files
178
+ tr_files, te_files = [f for i, f in enumerate(allf) if i % 3], [f for i, f in enumerate(allf) if i % 3 == 0]
179
+ Dtr, Dte = build(tr_files), build(te_files)
180
+ print(f"apprentissage : {len(tr_files)} zones, {Dtr.groupby(['region', 'word']).ngroups} codes lus, {len(Dtr)} paires | "
181
+ f"test : {len(te_files)} zones, {Dte.groupby(['region', 'word']).ngroups} codes lus ({time.time() - t0:.0f} s)")
182
+ ev = Dtr.groupby(["region", "word"]).y.transform("max") > 0 # on n'apprend que sur les mots correctement lus
183
+ clf = HistGradientBoostingClassifier(max_iter=300, learning_rate=0.06, max_leaf_nodes=24, l2_regularization=1.0,
184
+ class_weight="balanced", random_state=0)
185
+ clf.fit(Dtr.loc[ev, FEATS], Dtr.loc[ev, "y"])
186
+ pickle.dump({"model": clf, "features": FEATS, "radius": RADIUS}, open("naming_model.pkl", "wb"))
187
+ print("\nSUR LES ZONES DE TEST (north, jamais vues) :")
188
+ naive = Dte.sous_centre + 0.001 * Dte.part_boite
189
+ a0, _ = report(Dte, {k: (g.region.iloc[0], int(g.poly.iloc[np.argmax(n_)]), g.code.iloc[0], 1.0)
190
+ for (k, g), n_ in ((kg, naive.loc[kg[1].index].values) for kg in Dte.groupby(["region", "word"]))},
191
+ "Règle naïve (polygone sous l'étiquette)")
192
+ p = clf.predict_proba(Dte[FEATS])[:, 1]
193
+ a1, _ = report(Dte, {k: (g.region.iloc[0], int(g.poly.iloc[np.argmax(p[Dte.index.get_indexer(g.index)])]), g.code.iloc[0],
194
+ float(p[Dte.index.get_indexer(g.index)].max())) for k, g in Dte.groupby(["region", "word"])},
195
+ "Module appris (meilleure note)")
196
+ ch = assign(Dte, p)
197
+ a2, polys = report(Dte, ch, "Module appris + un seul code par polygone")
198
+ # avec seuil de confiance : on ne nomme que si la note est élevée
199
+ for thr in (0.5, 0.7, 0.85):
200
+ report(Dte, {k: v for k, v in ch.items() if v[3] >= thr}, f" ... en ne gardant que les notes >= {thr}")
201
+ from sklearn.inspection import permutation_importance
202
+ imp = permutation_importance(clf, Dte[FEATS], Dte["y"], n_repeats=3, random_state=0, scoring="average_precision")
203
+ print("\nIndices les plus utiles :", ", ".join(f"{FEATS[i]} ({imp.importances_mean[i]:.3f})" for i in np.argsort(-imp.importances_mean)[:8]))
pipeline.py ADDED
@@ -0,0 +1,288 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # =====================================================================================
2
+ # GEOCARTO – PIPELINE COMPLET : raster de carte géologique -> GeoPackage QGIS / ArcGIS
3
+ # 1. modèle (affinités + failles) par fenêtres glissantes 2. mutex watershed -> polygones
4
+ # 3. vectorisation (unités + failles) 4. OCR des textes de la carte
5
+ # 5. nommage : code écrit (module appris) > case de légende (si sûr) > "Unité non identifiée" + 3 suggestions
6
+ # 6. export GeoPackage (couches unites_geologiques, failles, table legende + styles QGIS par défaut)
7
+ # =====================================================================================
8
+ import os, sys, re, sqlite3, pickle, time
9
+ import numpy as np, pandas as pd, torch, geopandas as gpd
10
+ from scipy import ndimage as ndi
11
+ from rasterio.features import shapes
12
+ from rasterio.transform import Affine
13
+ from shapely.geometry import shape, LineString
14
+ from skimage.color import rgb2lab as sk_rgb2lab
15
+ from skimage.morphology import skeletonize
16
+
17
+ HERE = os.path.dirname(os.path.abspath(__file__))
18
+ sys.path.insert(0, HERE)
19
+ import gc_model as M
20
+
21
+
22
+ # ------------------------------------------------------------------ 1. modèle
23
+ def load_model(path):
24
+ ck = torch.load(path, map_location="cpu", weights_only=True)
25
+ cfg = ck["cfg"]
26
+ M.CFG.update(cfg); M.OFFSETS[:] = cfg["offsets"]
27
+ model = M.GeoCartoNet(cfg, pretrained=False)
28
+ model.load_state_dict({k: v.float() for k, v in ck["state_dict"].items()})
29
+ return model.eval()
30
+
31
+
32
+ @torch.no_grad()
33
+ def _run(model, im):
34
+ x = torch.from_numpy(np.ascontiguousarray(im).transpose(2, 0, 1).copy())[None]
35
+ return torch.sigmoid(model(x).float())[0].numpy()
36
+
37
+
38
+ def _tta(model, im, offsets):
39
+ """Moyenne sur 4 miroirs ; chaque décalage est ramené dans le repère d'origine (testé : exact)."""
40
+ H, W = im.shape[:2]
41
+ Y, X = np.mgrid[:H, :W]
42
+ idx = {tuple(o): k for k, o in enumerate(offsets)}
43
+ K = len(offsets)
44
+ acc = 0
45
+ for fy in (0, 1):
46
+ for fx in (0, 1):
47
+ v = im[::-1] if fy else im
48
+ v = v[:, ::-1] if fx else v
49
+ o = _run(model, v)
50
+ A, pf = o[:K], o[K]
51
+ py, px = (H - 1 - Y) if fy else Y, (W - 1 - X) if fx else X
52
+ out = np.empty_like(o)
53
+ for k, (dy, dx) in enumerate(offsets):
54
+ t = (-dy if fy else dy, -dx if fx else dx)
55
+ if t in idx:
56
+ out[k] = A[idx[t]][py, px]
57
+ else:
58
+ out[k] = A[idx[(-t[0], -t[1])]][np.clip(py + t[0], 0, H - 1), np.clip(px + t[1], 0, W - 1)]
59
+ out[K] = pf[py, px]
60
+ acc = acc + out
61
+ return acc / 4
62
+
63
+
64
+ def infer(model, rgb, tile=512, stride=384, tta=False):
65
+ """Fenêtres glissantes avec pondération de Hann -> affinités (K x H x W) + probabilité de faille."""
66
+ H, W = rgb.shape[:2]
67
+ ph, pw = max(tile - H, 0), max(tile - W, 0)
68
+ img = np.pad(rgb, ((0, ph), (0, pw), (0, 0)), mode="edge")
69
+ Hp, Wp = img.shape[:2]
70
+ K = len(M.OFFSETS)
71
+ acc = np.zeros((K + 1, Hp, Wp), np.float32); ws = np.zeros((Hp, Wp), np.float32)
72
+ hw = np.maximum(np.outer(np.hanning(tile), np.hanning(tile)), 1e-3).astype(np.float32)
73
+ pos = lambda n: sorted(set(list(range(0, n - tile + 1, stride)) + [n - tile]))
74
+ for y in pos(Hp):
75
+ for x in pos(Wp):
76
+ im = img[y:y + tile, x:x + tile]
77
+ o = _tta(model, im, M.OFFSETS) if tta else _run(model, im)
78
+ acc[:, y:y + tile, x:x + tile] += o * hw; ws[y:y + tile, x:x + tile] += hw
79
+ o = acc / ws
80
+ return o[:K, :H, :W], o[K, :H, :W]
81
+
82
+
83
+ # ------------------------------------------------------------------ 3. vectorisation
84
+ def vectorize_units(lab, transform, crs):
85
+ geoms, ids = [], []
86
+ for g, v in shapes(lab.astype(np.int32), mask=lab > 0, transform=transform):
87
+ geoms.append(shape(g)); ids.append(int(v))
88
+ P = gpd.GeoDataFrame({"pid": ids}, geometry=geoms, crs=crs).dissolve("pid").reset_index()
89
+ P["geometry"] = P.geometry.simplify(abs(transform.a) * 0.75, preserve_topology=True)
90
+ return P
91
+
92
+
93
+ def vectorize_faults(pf, transform, crs, thr=0.5, min_px=15):
94
+ sk = skeletonize(pf > thr)
95
+ lab, n = ndi.label(sk, structure=np.ones((3, 3)))
96
+ sizes = np.bincount(lab.ravel())
97
+ nb = ndi.convolve(sk.astype(np.uint8), np.ones((3, 3), np.uint8), mode="constant") - 1
98
+ lines = []
99
+ seen = np.zeros_like(sk)
100
+ for c in range(1, n + 1):
101
+ if sizes[c] < min_px:
102
+ continue
103
+ ys, xs = np.nonzero(lab == c)
104
+ pts = set(zip(ys.tolist(), xs.tolist()))
105
+ ends = [p for p in pts if nb[p] == 1] or [next(iter(pts))]
106
+ for st in ends:
107
+ if seen[st]:
108
+ continue
109
+ path, cur = [st], st
110
+ seen[st] = True
111
+ while True:
112
+ nxt = [(cur[0] + a, cur[1] + b) for a in (-1, 0, 1) for b in (-1, 0, 1) if (a or b)
113
+ and (cur[0] + a, cur[1] + b) in pts and not seen[cur[0] + a, cur[1] + b]]
114
+ if not nxt:
115
+ break
116
+ cur = nxt[0]; seen[cur] = True; path.append(cur)
117
+ if len(path) >= min_px:
118
+ lines.append(LineString([transform * (x + 0.5, y + 0.5) for y, x in path]).simplify(abs(transform.a)))
119
+ F = gpd.GeoDataFrame({"type": ["indéterminée"] * len(lines)}, geometry=lines, crs=crs)
120
+ F["longueur_km"] = (F.length / 1000).round(2)
121
+ return F
122
+
123
+
124
+ # ------------------------------------------------------------------ 4. OCR
125
+ def read_text(rgb, reader, B=1024, OV=96):
126
+ words = []
127
+ H, W = rgb.shape[:2]
128
+ for y in range(0, H, B - OV):
129
+ for x in range(0, W, B - OV):
130
+ for box, txt, conf in reader.readtext(rgb[y:y + B, x:x + B], mag_ratio=2.0, text_threshold=0.6, low_text=0.3):
131
+ b = np.array(box) + [x, y]; cx, cy = b.mean(0)
132
+ if (x > 0 and cx < x + OV / 2) or (y > 0 and cy < y + OV / 2):
133
+ continue
134
+ words.append(dict(text=txt, conf=conf, cx=cx, cy=cy, x0=b[:, 0].min(), x1=b[:, 0].max(), y0=b[:, 1].min(), y1=b[:, 1].max()))
135
+ return pd.DataFrame(words, columns=["text", "conf", "cx", "cy", "x0", "x1", "y0", "y1"])
136
+
137
+
138
+ # ------------------------------------------------------------------ 5. nommage
139
+ def swatch_features(lab_pixels):
140
+ """Signature d'une case de légende / d'un polygone : couleur dominante, couleurs claire et foncée du motif, texture."""
141
+ px = lab_pixels[lab_pixels[:, 0] > 45]
142
+ if len(px) < 20:
143
+ px = lab_pixels
144
+ L = px[:, 0]; med = np.percentile(L, 50)
145
+ return np.concatenate([np.median(px, 0), np.median(px[L >= med], 0), np.median(px[L <= med], 0), px.std(0) / 4])
146
+
147
+
148
+ def ocr_candidates(rgb, lab, words, vocab, naming_pkl, flt=None):
149
+ """Toutes les paires (code lu, polygone candidat) avec leur score appris ; et l'affectation un code -> un polygone."""
150
+ import naming_model as NM
151
+ NM.VOCAB = sorted(vocab); NM.VSET = set(NM.VOCAB)
152
+ mdl = pickle.load(open(naming_pkl, "rb"))
153
+ reg = dict(inst=lab, rgb=rgb, lum=rgb.astype(np.float32) @ np.array([0.299, 0.587, 0.114], np.float32),
154
+ area=np.bincount(lab.ravel()), flt=flt)
155
+ rows = []
156
+ for k, wd in enumerate(words.itertuples()):
157
+ code, q = NM.to_code(wd.text)
158
+ if code is None:
159
+ continue
160
+ class _W: pass
161
+ w = _W(); w.__dict__.update(wd._asdict()); w.q = q
162
+ for r in NM.candidates(reg, w):
163
+ r.update(region="carte", word=k, code=code, lu=wd.text); rows.append(r)
164
+ if not rows:
165
+ return pd.DataFrame(columns=["word", "code", "s"]), {}
166
+ D = pd.DataFrame(rows)
167
+ D["s"] = mdl["model"].predict_proba(D[mdl["features"]])[:, 1]
168
+ return D, NM.assign(D, D["s"].values)
169
+
170
+ RANK_FEATS = ["d_norm", "d_rel", "rang", "gap12", "d_ratio", "ocr", "ocr_assigne", "ocr_rel", "lu", "n_lu", "dist_lu", "log_aire"]
171
+
172
+ def rank_pairs(pids, feats, areas, cents, codes, Fm, D, ch, words, K=12):
173
+ """Paires (polygone, code candidat) et leurs indices. Candidats = K cases de légende les plus proches + codes lus par l'OCR.
174
+ Indices : uniquement le texte lu sur la carte et la légende (pas de propagation d'apparence entre polygones)."""
175
+ cidx = {c: i for i, c in enumerate(codes)}
176
+ d = np.linalg.norm(feats[:, None, :] - Fm[None], axis=2)
177
+ E, lu_pos, asg = {}, {}, {}
178
+ if len(D):
179
+ for (p, c), s in D.groupby(["poly", "code"]).s.max().items():
180
+ E[(p, c)] = s
181
+ for w, c in D[["word", "code"]].drop_duplicates().values:
182
+ lu_pos.setdefault(c, []).append((words.iloc[w].cy, words.iloc[w].cx))
183
+ for (_, w), (_, p, c, s) in ch.items():
184
+ asg[(p, c)] = max(asg.get((p, c), 0), s)
185
+ Ep = {}
186
+ for (p, c), s in E.items():
187
+ Ep.setdefault(p, []).append(c)
188
+ lu_pos = {c: np.array(v) for c, v in lu_pos.items()}
189
+ rows = []
190
+ for i, p in enumerate(pids):
191
+ o = np.argsort(d[i]); rank = np.empty(len(codes), int); rank[o] = np.arange(len(codes))
192
+ cand = list(dict.fromkeys(list(codes[o[:K]]) + [c for c in Ep.get(p, []) if c in cidx]))
193
+ emax = max([E[(p, c)] for c in Ep.get(p, [])] + [0]); gap = d[i, o[1]] - d[i, o[0]] if len(o) > 1 else 0
194
+ for c in cand:
195
+ j = cidx[c]; pos = lu_pos.get(c)
196
+ rows.append(dict(poly=p, code=c, d=d[i, j], d_rel=d[i, j] - d[i, o[0]], rang=rank[j], gap12=gap,
197
+ ocr=E.get((p, c), 0), ocr_assigne=asg.get((p, c), 0), ocr_max_poly=emax,
198
+ lu=int(pos is not None), n_lu=0 if pos is None else len(pos),
199
+ dist_lu=5000 if pos is None else np.hypot(pos[:, 0] - cents[i][0], pos[:, 1] - cents[i][1]).min(),
200
+ log_aire=np.log(areas[i])))
201
+ X = pd.DataFrame(rows)
202
+ X["d_ratio"] = X.d_rel / (X.gap12 + 1); X["d_norm"] = X.d / X.d.median(); X["ocr_rel"] = X.ocr - X.ocr_max_poly
203
+ return X
204
+
205
+ def name_polygons(rgb, lab, words, legend, naming_pkl, flt=None, ranker_pkl="legend_ranker.pkl", sure=0.8, check=0.5):
206
+ """legend : DataFrame index=code, colonnes nom, age, lithologie, couleur_hex, feat (signature de la case).
207
+ Retourne par polygone : code, source_nom, confiance, texte_lu, suggestions.
208
+ source_nom : 'code lu sur la carte' / 'légende' (confiance >= sure), '… – à vérifier' (check <= confiance < sure), 'non identifiée'."""
209
+ D, ch = ocr_candidates(rgb, lab, words, legend.index, naming_pkl, flt)
210
+ R = pickle.load(open(ranker_pkl, "rb"))
211
+ labc = sk_rgb2lab(rgb).reshape(-1, 3)
212
+ ids = np.unique(lab[lab > 0])
213
+ order = np.argsort(lab.ravel(), kind="stable"); sl = lab.ravel()[order]
214
+ starts = np.searchsorted(sl, ids); ends = np.searchsorted(sl, ids, side="right")
215
+ feats = np.stack([swatch_features(labc[order[a:b]]) for a, b in zip(starts, ends)])
216
+ cents = ndi.center_of_mass(np.ones_like(lab), lab, ids)
217
+ codes = np.array(legend.index); Fm = np.stack(legend.feat.values)
218
+ X = rank_pairs(ids, feats, ends - starts, cents, codes, Fm, D, ch, words, K=R.get("K", 12))
219
+ X["p"] = R["model"].predict_proba(X[R["features"]])[:, 1]
220
+ X = X.sort_values(["poly", "p"], ascending=[True, False])
221
+ lu_txt = {}
222
+ for (_, w), (_, p, c, s) in ch.items():
223
+ lu_txt[(p, c)] = words.iloc[w].text
224
+ out = {}
225
+ for p, g in X.groupby("poly", sort=False):
226
+ top = g.head(3); c, pr = top.code.iloc[0], float(top.p.iloc[0])
227
+ sug = ", ".join(f"{a} ({b:.0%})" for a, b in zip(top.code, top.p))
228
+ ocr = top.ocr.iloc[0] > 0
229
+ src = "code lu sur la carte" if ocr else "couleur de la légende"
230
+ if pr >= sure:
231
+ out[p] = dict(code=c, source_nom=src, confiance=round(pr, 2), texte_lu=lu_txt.get((p, c), ""), suggestions=sug)
232
+ elif pr >= check: # trop peu fiable sur cartes réelles (31-64 % juste) : pas de nom, mais suggestion classée en tête
233
+ out[p] = dict(code="", source_nom="non identifiée – suggestion probable", confiance=round(pr, 2), texte_lu=lu_txt.get((p, c), ""), suggestions=sug)
234
+ else:
235
+ out[p] = dict(code="", source_nom="non identifiée", confiance=round(pr, 2), texte_lu="", suggestions=sug)
236
+ return pd.DataFrame.from_dict(out, orient="index")
237
+
238
+ # ------------------------------------------------------------------ 6. export
239
+ QML_UNITS = open(os.path.join(HERE, "qml_units.xml")).read() if os.path.exists(os.path.join(HERE, "qml_units.xml")) else None
240
+ QML_FAULTS = open(os.path.join(HERE, "qml_faults.xml")).read() if os.path.exists(os.path.join(HERE, "qml_faults.xml")) else None
241
+
242
+
243
+ def export_gpkg(path, P, F, legend):
244
+ if os.path.exists(path):
245
+ os.remove(path)
246
+ P.to_file(path, layer="unites_geologiques", driver="GPKG")
247
+ if len(F):
248
+ F.to_file(path, layer="failles", driver="GPKG")
249
+ con = sqlite3.connect(path)
250
+ legend.drop(columns=["feat"]).reset_index().rename(columns={"index": "code"}).to_sql("legende", con, index=False, if_exists="replace")
251
+ con.execute("""CREATE TABLE IF NOT EXISTS layer_styles (id INTEGER PRIMARY KEY AUTOINCREMENT, f_table_catalog TEXT(256),
252
+ f_table_schema TEXT(256), f_table_name TEXT(256), f_geometry_column TEXT(256), styleName TEXT(30), styleQML TEXT,
253
+ styleSLD TEXT, useAsDefault BOOLEAN, description TEXT, owner TEXT(30), ui TEXT(30), update_time DATETIME DEFAULT CURRENT_TIMESTAMP)""")
254
+ for tbl, qml in (("unites_geologiques", QML_UNITS), ("failles", QML_FAULTS)):
255
+ if qml and (tbl != "failles" or len(F)):
256
+ con.execute("INSERT INTO layer_styles (f_table_catalog,f_table_schema,f_table_name,f_geometry_column,styleName,styleQML,styleSLD,useAsDefault,description,owner,update_time) "
257
+ "VALUES ('','',?,'geom',?,?,'',1,'style GeoCarto','',strftime('%Y-%m-%dT%H:%M:%fZ','now'))", (tbl, tbl, qml))
258
+ con.execute("INSERT INTO gpkg_contents (table_name, data_type, identifier) SELECT 'layer_styles','attributes','layer_styles' "
259
+ "WHERE NOT EXISTS (SELECT 1 FROM gpkg_contents WHERE table_name='layer_styles')")
260
+ con.execute("INSERT INTO gpkg_contents (table_name, data_type, identifier) SELECT 'legende','attributes','legende' "
261
+ "WHERE NOT EXISTS (SELECT 1 FROM gpkg_contents WHERE table_name='legende')")
262
+ con.commit(); con.close()
263
+
264
+
265
+ def run(rgb, transform, crs, model, legend, reader, naming_pkl, out_gpkg, tta=False, log=print, ranker_pkl="legend_ranker.pkl"):
266
+ t = time.time()
267
+ aff, pf = infer(model, rgb, tta=tta); log(f"modèle : {time.time() - t:.0f} s")
268
+ lab = M.mutex_watershed(aff); log(f"segmentation : {lab.max()} polygones ({time.time() - t:.0f} s)")
269
+ words = read_text(rgb, reader); log(f"OCR : {len(words)} textes ({time.time() - t:.0f} s)")
270
+ N = name_polygons(rgb, lab, words, legend, naming_pkl, flt=(pf > 0.5).astype(np.uint8), ranker_pkl=ranker_pkl); log(f"nommage ({time.time() - t:.0f} s)")
271
+ P = vectorize_units(lab, transform, crs)
272
+ P = P.join(N, on="pid")
273
+ P["code"] = P.code.fillna(""); P["source_nom"] = P.source_nom.fillna("non identifiée")
274
+ lg = legend.reindex(P.code)
275
+ P["nom"] = np.where(P.code != "", (P.code + " – " + lg.lithologie.fillna("").values).str.strip(" –"), "Unité non identifiée")
276
+ P["age"] = lg.age.fillna("").values; P["lithologie"] = lg.lithologie.fillna("").values
277
+ med = [ndi.median(rgb[..., c], lab, P.pid.values) for c in range(3)]
278
+ P["couleur_carte"] = ["#%02x%02x%02x" % tuple(int(v) for v in x) for x in zip(*med)]
279
+ P["couleur_hex"] = np.where(P.code != "", lg.couleur_hex.fillna("").values, P.couleur_carte)
280
+ P["couleur_hex"] = np.where(P.couleur_hex == "", P.couleur_carte, P.couleur_hex)
281
+ P["surface_km2"] = (P.area / 1e6).round(3)
282
+ P["id"] = np.arange(1, len(P) + 1)
283
+ P = P[["id", "code", "nom", "age", "lithologie", "source_nom", "confiance", "texte_lu", "suggestions",
284
+ "couleur_hex", "couleur_carte", "surface_km2", "geometry"]]
285
+ F = vectorize_faults(pf, transform, crs)
286
+ export_gpkg(out_gpkg, P, F, legend.loc[legend.index.isin(P.code)])
287
+ log(f"export : {out_gpkg} | {len(P)} unités, {len(F)} failles ({time.time() - t:.0f} s)")
288
+ return P, F, lab, pf
qgis_test.py ADDED
@@ -0,0 +1,55 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Vrai test QGIS (PyQGIS 3.34, sans interface) d'un GeoPackage produit par pipeline.py
2
+ # QT_QPA_PLATFORM=offscreen python3.12 qgis_test.py <carte.gpkg> <rendu.png>
3
+ import sys
4
+ from qgis.core import (QgsApplication, QgsVectorLayer, QgsProject, QgsMapSettings, QgsMapRendererParallelJob,
5
+ QgsGeometry, QgsWkbTypes)
6
+ from qgis.PyQt.QtCore import QSize
7
+ from qgis.PyQt.QtGui import QColor
8
+
9
+ gpkg, png = sys.argv[1], sys.argv[2]
10
+ app = QgsApplication([], False); app.initQgis()
11
+ ok = True
12
+ def check(cond, msg):
13
+ global ok
14
+ print(("OK " if cond else "ÉCHEC"), msg); ok &= bool(cond)
15
+
16
+ layers = []
17
+ for name, geom in (("unites_geologiques", QgsWkbTypes.PolygonGeometry), ("failles", QgsWkbTypes.LineGeometry)):
18
+ lyr = QgsVectorLayer(f"{gpkg}|layername={name}", name, "ogr")
19
+ check(lyr.isValid(), f"couche {name} chargée")
20
+ if not lyr.isValid():
21
+ continue
22
+ msg, styled = lyr.loadDefaultStyle() # style QML stocké dans la table layer_styles du GeoPackage
23
+ check(styled, f"{name} : style intégré chargé automatiquement ({msg})")
24
+ check(lyr.geometryType() == geom, f"{name} : type de géométrie {QgsWkbTypes.geometryDisplayString(lyr.geometryType())}")
25
+ check(lyr.crs().isValid(), f"{name} : SCR {lyr.crs().authid() or lyr.crs().description()}")
26
+ n = lyr.featureCount(); check(n > 0, f"{name} : {n} entités")
27
+ bad = sum(1 for f in lyr.getFeatures() if not f.geometry().isGeosValid())
28
+ check(bad == 0, f"{name} : géométries invalides = {bad}")
29
+ print(" champs :", [f.name() for f in lyr.fields()])
30
+ r = lyr.renderer()
31
+ print(" rendu :", r.type(), (f"{len(r.categories())} catégories" if hasattr(r, "categories") else ""))
32
+ if name == "unites_geologiques":
33
+ need = {"code", "nom", "age", "lithologie", "source_nom", "confiance", "suggestions"}
34
+ check(need <= {f.name() for f in lyr.fields()}, "colonnes de nommage présentes")
35
+ # filtre attributaire comme le ferait un utilisateur QGIS
36
+ lyr.setSubsetString("\"source_nom\" = 'code lu sur la carte'")
37
+ print(" filtre source_nom='code lu sur la carte' :", lyr.featureCount(), "entités"); lyr.setSubsetString("")
38
+ ex = next(lyr.getFeatures())
39
+ print(" exemple :", {k: ex[k] for k in ("code", "nom", "age", "source_nom", "confiance")})
40
+ layers.append(lyr)
41
+
42
+ leg = QgsVectorLayer(f"{gpkg}|layername=legende", "legende", "ogr")
43
+ check(leg.isValid() and leg.featureCount() > 0, f"table legende : {leg.featureCount() if leg.isValid() else 0} lignes")
44
+
45
+ if layers:
46
+ QgsProject.instance().addMapLayers(layers)
47
+ ms = QgsMapSettings(); ms.setLayers(layers[::-1]); ms.setDestinationCrs(layers[0].crs())
48
+ ext = layers[0].extent(); ms.setExtent(ext); ms.setOutputSize(QSize(1400, int(1400 * ext.height() / ext.width())))
49
+ ms.setBackgroundColor(QColor("white"))
50
+ job = QgsMapRendererParallelJob(ms); job.start(); job.waitForFinished()
51
+ img = job.renderedImage(); check(img.save(png), f"rendu carte -> {png}")
52
+ QgsProject.instance().write(png.rsplit(".", 1)[0] + ".qgz")
53
+ print(" projet QGIS écrit :", png.rsplit(".", 1)[0] + ".qgz")
54
+ print("\nRÉSULTAT QGIS :", "TOUT OK" if ok else "PROBLÈMES")
55
+ app.exitQgis()
qml_faults.xml ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE qgis PUBLIC 'http://mrcc.com/qgis.dtd' 'SYSTEM'>
2
+ <qgis version="3.34.0" styleCategories="Symbology">
3
+ <renderer-v2 type="singleSymbol" symbollevels="0" enableorderby="0" forceraster="0">
4
+ <symbols>
5
+ <symbol type="line" name="0" alpha="1" clip_to_extent="1" force_rhr="0">
6
+ <layer class="SimpleLine" enabled="1" locked="0" pass="0">
7
+ <Option type="Map">
8
+ <Option name="line_color" type="QString" value="0,0,0,255"/>
9
+ <Option name="line_width" type="QString" value="0.6"/>
10
+ <Option name="line_width_unit" type="QString" value="MM"/>
11
+ <Option name="line_style" type="QString" value="solid"/>
12
+ </Option>
13
+ </layer>
14
+ </symbol>
15
+ </symbols>
16
+ </renderer-v2>
17
+ </qgis>
qml_units.xml ADDED
@@ -0,0 +1,41 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE qgis PUBLIC 'http://mrcc.com/qgis.dtd' 'SYSTEM'>
2
+ <qgis version="3.34.0" styleCategories="Symbology|Labeling">
3
+ <renderer-v2 type="singleSymbol" symbollevels="0" enableorderby="0" forceraster="0">
4
+ <symbols>
5
+ <symbol type="fill" name="0" alpha="1" clip_to_extent="1" force_rhr="0">
6
+ <layer class="SimpleFill" enabled="1" locked="0" pass="0">
7
+ <Option type="Map">
8
+ <Option name="color" type="QString" value="200,200,200,255"/>
9
+ <Option name="outline_color" type="QString" value="40,40,40,255"/>
10
+ <Option name="outline_style" type="QString" value="solid"/>
11
+ <Option name="outline_width" type="QString" value="0.15"/>
12
+ <Option name="outline_width_unit" type="QString" value="MM"/>
13
+ <Option name="style" type="QString" value="solid"/>
14
+ </Option>
15
+ <data_defined_properties>
16
+ <Option type="Map">
17
+ <Option name="name" type="QString" value=""/>
18
+ <Option name="properties" type="Map">
19
+ <Option name="fillColor" type="Map">
20
+ <Option name="active" type="bool" value="true"/>
21
+ <Option name="expression" type="QString" value="&quot;couleur_hex&quot;"/>
22
+ <Option name="type" type="int" value="3"/>
23
+ </Option>
24
+ </Option>
25
+ <Option name="type" type="QString" value="collection"/>
26
+ </Option>
27
+ </data_defined_properties>
28
+ </layer>
29
+ </symbol>
30
+ </symbols>
31
+ </renderer-v2>
32
+ <labeling type="simple">
33
+ <settings calloutType="simple">
34
+ <text-style fieldName="code" isExpression="0" fontSize="8" fontSizeUnit="Point" fontFamily="Arial" textColor="20,20,20,255" fontWeight="50">
35
+ <text-buffer bufferDraw="1" bufferSize="0.6" bufferSizeUnits="MM" bufferColor="255,255,255,200"/>
36
+ </text-style>
37
+ <placement placement="1" placementFlags="10" centroidInside="1" fitInPolygonOnly="1"/>
38
+ <rendering scaleVisibility="0" obstacle="1" limitNumLabels="0" displayAll="0"/>
39
+ </settings>
40
+ </labeling>
41
+ </qgis>
requirements.txt ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ torch>=2.2
2
+ segmentation-models-pytorch==0.6.0
3
+ timm==1.0.31
4
+ numba>=0.60
5
+ numpy
6
+ scipy
7
+ pandas
8
+ scikit-learn==1.9.1
9
+ scikit-image>=0.24
10
+ rasterio>=1.4
11
+ geopandas>=1.0
12
+ shapely>=2.0
13
+ easyocr==1.7.2
training/e2e_naming_9_zones.csv ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ zone,modele,surface,nommee,juste,sure,sure_juste
2
+ north_2048_10240,geocarto_model/best_fp16.pt,3599656,1499573,1385929,1499573,1385929
3
+ north_0_14336,geocarto_model/best_fp16.pt,3946304,1366865,1312457,1366865,1312457
4
+ north_4096_8192,geocarto_model/best_fp16.pt,3701097,629318,476448,629318,476448
5
+ north_4096_18432,geocarto_model/best_fp16.pt,4194304,162091,145539,162091,145539
6
+ north_8192_16384,geocarto_model/best_fp16.pt,4012500,835803,665180,835803,665180
7
+ north_6144_16384,geocarto_model/best_fp16.pt,3958564,1188440,1148893,1188440,1148893
8
+ north_8192_10240,geocarto_model/best_fp16.pt,3799155,2138119,1859959,2138119,1859959
9
+ north_2048_16384,geocarto_model/best_fp16.pt,4184402,491224,465825,491224,465825
10
+ north_0_2048,geocarto_model/best_fp16.pt,4074680,2335504,2287133,2335504,2287133
training/history.csv ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ step,loss,heures,modele_score,modele_reel,modele_synth,meilleur,ema_score,ema_reel,ema_synth
2
+ 2500,1.2678692572832106,0.48,0.1156618180162316,0.05801492429830092,0.17330871173416226,modele,0.1257966551104029,0.05539731695128261,0.1961959932695232
3
+ 5000,0.7672936572551727,0.91,0.09529314214143128,0.052139002017007524,0.13844728226585504,modele,0.0973675850057942,0.04910327124788072,0.1456318987637077
4
+ 7500,0.7159945685982704,1.34,0.08752472791366481,0.05359620966079264,0.12145324616653697,modele,0.09248038060147537,0.04966643006802915,0.1352943311349216
5
+ 10000,0.685878133893013,1.76,0.0836330135427693,0.050465125732164565,0.11680090135337402,modele,0.08596079908456222,0.05022387633190165,0.12169772183722279
6
+ 12500,0.6719268432140351,2.18,0.08175700726637768,0.04745522503445026,0.1160587894983051,ema,0.07972059465372879,0.0443447903329452,0.11509639897451236
7
+ 15000,0.6572574779510498,2.61,0.07931626369027464,0.049011944981386724,0.10962058239916257,modele,0.08061698872873928,0.04628204825805805,0.1149519291994205
8
+ 17500,0.6435735144972802,3.03,0.07761612833109743,0.04601976537745372,0.10921249128474114,modele,0.07827108745221906,0.04586425730415124,0.11067791760028689
9
+ 20000,0.6359998931407929,3.45,0.07532928573818343,0.04557381860105503,0.10508475287531183,modele,0.07670002805089449,0.046568494432873635,0.10683156166891535
10
+ 22500,0.626114172899723,3.87,0.07367399082233667,0.043336204269031287,0.10401177737564203,modele,0.07437775992976638,0.044113875308752916,0.10464164455077984
11
+ 25000,0.6217467277765274,4.3,0.07452715778951396,0.0464807556550982,0.10257355992392973,,0.07528071106283755,0.04745712026447532,0.10310430186119979
12
+ 27500,0.6217535385727883,4.72,0.07442850533802978,0.045681225671011565,0.103175785005048,,0.0742625662646455,0.04581297982583516,0.10271215270345585
13
+ 30000,0.6190747852683067,5.14,0.07454904397298479,0.04589639765003511,0.10320169029593447,,0.07448182474760545,0.04589398591477694,0.10306966358043397
training/lab_round4_ablation.csv ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ variante,desc,train_s,réel|ARAND,réel|split,réel|merge,réel|rappel_petites,réel|rappel_grandes,réel|F1_failles,synthétique|ARAND,synthétique|split,synthétique|merge,synthétique|rappel_petites,synthétique|rappel_grandes,synthétique|F1_failles,piège_mono|ARAND,piège_mono|split,piège_mono|merge,piège_mono|rappel_petites,piège_mono|rappel_grandes,piège_mono|F1_failles,piège_motifs|ARAND,piège_motifs|split,piège_motifs|merge,piège_motifs|rappel_petites,piège_motifs|rappel_grandes,piège_motifs|F1_failles,réel@s3m15|ARAND,réel@s3m15|split,réel@s3m15|merge,réel@s3m15|rappel_petites,réel@s3m15|rappel_grandes,réel@s3m15|F1_failles,synthétique@s3m15|ARAND,synthétique@s3m15|split,synthétique@s3m15|merge,synthétique@s3m15|rappel_petites,synthétique@s3m15|rappel_grandes,synthétique@s3m15|F1_failles,piège_mono@s3m15|ARAND,piège_mono@s3m15|split,piège_mono@s3m15|merge,piège_mono@s3m15|rappel_petites,piège_mono@s3m15|rappel_grandes,piège_mono@s3m15|F1_failles,piège_motifs@s3m15|ARAND,piège_motifs@s3m15|split,piège_motifs@s3m15|merge,piège_motifs@s3m15|rappel_petites,piège_motifs@s3m15|rappel_grandes,piège_motifs@s3m15|F1_failles,réel@s1m40|ARAND,réel@s1m40|split,réel@s1m40|merge,réel@s1m40|rappel_petites,réel@s1m40|rappel_grandes,réel@s1m40|F1_failles,synthétique@s1m40|ARAND,synthétique@s1m40|split,synthétique@s1m40|merge,synthétique@s1m40|rappel_petites,synthétique@s1m40|rappel_grandes,synthétique@s1m40|F1_failles,piège_mono@s1m40|ARAND,piège_mono@s1m40|split,piège_mono@s1m40|merge,piège_mono@s1m40|rappel_petites,piège_mono@s1m40|rappel_grandes,piège_mono@s1m40|F1_failles,piège_motifs@s1m40|ARAND,piège_motifs@s1m40|split,piège_motifs@s1m40|merge,piège_motifs@s1m40|rappel_petites,piège_motifs@s1m40|rappel_grandes,piège_motifs@s1m40|F1_failles,réel@sttam40|ARAND,réel@sttam40|split,réel@sttam40|merge,réel@sttam40|rappel_petites,réel@sttam40|rappel_grandes,réel@sttam40|F1_failles,synthétique@sttam40|ARAND,synthétique@sttam40|split,synthétique@sttam40|merge,synthétique@sttam40|rappel_petites,synthétique@sttam40|rappel_grandes,synthétique@sttam40|F1_failles,piège_mono@sttam40|ARAND,piège_mono@sttam40|split,piège_mono@sttam40|merge,piège_mono@sttam40|rappel_petites,piège_mono@sttam40|rappel_grandes,piège_mono@sttam40|F1_failles,piège_motifs@sttam40|ARAND,piège_motifs@sttam40|split,piège_motifs@sttam40|merge,piège_motifs@sttam40|rappel_petites,piège_motifs@sttam40|rappel_grandes,piège_motifs@sttam40|F1_failles
2
+ M0,R3 sur données généralisées (unité minimale de cartographie),1748,0.0672,0.2439,0.2701,0.7482,0.8923,0.7426,0.1463,0.295,0.6285,0.7225,0.8621,0.9339,0.2164,0.6157,0.8184,0.5046,0.6552,0.9545,0.1749,0.405,0.794,0.6166,0.8019,0.9502,0.0672,0.2428,0.2724,0.754,0.8923,0.7426,0.1464,0.2838,0.6452,0.7378,0.8608,0.9339,0.2165,0.6035,0.8388,0.5083,0.6562,0.9545,0.1748,0.3917,0.8149,0.6239,0.8019,0.9502,0.0703,0.234,0.2843,0.7641,0.906,0.7426,0.1605,0.2937,0.7135,0.7196,0.8522,0.9339,0.2279,0.5568,0.9462,0.5266,0.6768,0.9545,0.1928,0.3936,0.9258,0.6266,0.785,0.9502,0.0662,0.2643,0.2408,0.7381,0.8769,0.7325,0.132,0.3012,0.5495,0.7287,0.8695,0.9407,0.2296,0.7467,0.6981,0.4863,0.6207,0.9587,0.1703,0.4471,0.7154,0.6074,0.7879,0.955
3
+ M1,M0 + poids des petits polygones,1780,0.1951,0.7857,0.2111,0.6599,0.4632,0.8535,0.2007,0.4829,0.611,0.7664,0.7685,0.9163,0.2745,0.806,0.85,0.6127,0.5557,0.9321,0.2344,0.5837,0.8392,0.6788,0.7075,0.9394,0.1951,0.7846,0.2137,0.6599,0.4632,0.8535,0.2012,0.4692,0.6381,0.7755,0.7648,0.9163,0.2751,0.7851,0.8922,0.6145,0.5557,0.9321,0.2349,0.5642,0.8771,0.6834,0.7084,0.9394,0.1707,0.6542,0.2365,0.6715,0.535,0.8535,0.1801,0.434,0.6408,0.6777,0.782,0.9163,0.2697,0.7596,0.8923,0.5754,0.5734,0.9321,0.2336,0.5346,0.9096,0.6361,0.7196,0.9394,0.3231,1.4159,0.1157,0.6165,0.2222,0.8527,0.2767,0.7946,0.5683,0.7651,0.6724,0.9227,0.3779,1.2186,0.7704,0.6127,0.4266,0.9388,0.2697,0.7292,0.7805,0.6851,0.6626,0.9437
4
+ M2,M1 + OHEM (arêtes difficiles),1742,0.1089,0.3797,0.2879,0.7308,0.7897,0.836,0.1584,0.317,0.66,0.7958,0.8251,0.9049,0.267,0.6589,0.9735,0.6465,0.6138,0.93,0.1892,0.3849,0.9197,0.704,0.7458,0.9225,0.109,0.3784,0.2922,0.7381,0.7897,0.836,0.1591,0.3056,0.6859,0.8012,0.8239,0.9049,0.268,0.6417,1.0144,0.6459,0.6128,0.93,0.1901,0.368,0.9563,0.7067,0.743,0.9225,0.097,0.3264,0.3144,0.7554,0.8188,0.836,0.1646,0.3254,0.7093,0.7382,0.819,0.9049,0.2639,0.6298,1.0306,0.6234,0.6207,0.93,0.1983,0.3907,0.9935,0.6805,0.7355,0.9225,0.1174,0.4317,0.2495,0.7482,0.7607,0.8417,0.1608,0.3647,0.6073,0.7962,0.8153,0.9147,0.2862,0.7393,0.9183,0.6626,0.601,0.9284,0.1916,0.4099,0.868,0.7294,0.7486,0.9309
training/metrics.json ADDED
@@ -0,0 +1,257 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "cfg": {
3
+ "arch": "unetpp",
4
+ "encoder": "efficientnet-b4",
5
+ "input": "rgb+lab",
6
+ "offsets": [
7
+ [
8
+ 1,
9
+ 0
10
+ ],
11
+ [
12
+ 0,
13
+ 1
14
+ ],
15
+ [
16
+ 3,
17
+ 0
18
+ ],
19
+ [
20
+ 0,
21
+ 3
22
+ ],
23
+ [
24
+ 9,
25
+ 0
26
+ ],
27
+ [
28
+ 0,
29
+ 9
30
+ ],
31
+ [
32
+ 9,
33
+ 9
34
+ ],
35
+ [
36
+ 9,
37
+ -9
38
+ ],
39
+ [
40
+ 27,
41
+ 0
42
+ ],
43
+ [
44
+ 0,
45
+ 27
46
+ ]
47
+ ],
48
+ "n_short": 2,
49
+ "mws_stride": 1,
50
+ "mws_min_size": 40,
51
+ "tile": 512,
52
+ "small_w": false,
53
+ "ohem": false,
54
+ "tta": false
55
+ },
56
+ "steps": 22500,
57
+ "val_score": 0.07367399082233667,
58
+ "test": [
59
+ {
60
+ "groupe": "Africa",
61
+ "tuiles": 70,
62
+ "ARAND": 0.0299,
63
+ "sur_decoupage": 0.1534,
64
+ "fusion": 0.1099,
65
+ "rappel_petites": 0.8958,
66
+ "rappel_grandes": 0.9785,
67
+ "rappel_40_200": 0.8,
68
+ "rappel_200_1000": 0.9475,
69
+ "rappel_tous": 0.9466,
70
+ "surface_correcte": 0.9931,
71
+ "F1_failles": NaN
72
+ },
73
+ {
74
+ "groupe": "Europe",
75
+ "tuiles": 120,
76
+ "ARAND": 0.0986,
77
+ "sur_decoupage": 0.6392,
78
+ "fusion": 0.2721,
79
+ "rappel_petites": 0.8224,
80
+ "rappel_grandes": 0.9041,
81
+ "rappel_40_200": 0.781,
82
+ "rappel_200_1000": 0.8925,
83
+ "rappel_tous": 0.8386,
84
+ "surface_correcte": 0.9202,
85
+ "F1_failles": NaN
86
+ },
87
+ {
88
+ "groupe": "North America",
89
+ "tuiles": 268,
90
+ "ARAND": 0.0912,
91
+ "sur_decoupage": 0.4542,
92
+ "fusion": 0.2775,
93
+ "rappel_petites": 0.7249,
94
+ "rappel_grandes": 0.9054,
95
+ "rappel_40_200": 0.6559,
96
+ "rappel_200_1000": 0.7991,
97
+ "rappel_tous": 0.7677,
98
+ "surface_correcte": 0.9507,
99
+ "F1_failles": 0.9503
100
+ },
101
+ {
102
+ "groupe": "Oceania",
103
+ "tuiles": 150,
104
+ "ARAND": 0.1763,
105
+ "sur_decoupage": 0.9725,
106
+ "fusion": 0.4681,
107
+ "rappel_petites": 0.6385,
108
+ "rappel_grandes": 0.7555,
109
+ "rappel_40_200": 0.6085,
110
+ "rappel_200_1000": 0.6838,
111
+ "rappel_tous": 0.6565,
112
+ "surface_correcte": 0.8352,
113
+ "F1_failles": 0.9868
114
+ },
115
+ {
116
+ "groupe": "GA-Australia / synthetic",
117
+ "tuiles": 150,
118
+ "ARAND": 0.1763,
119
+ "sur_decoupage": 0.9725,
120
+ "fusion": 0.4681,
121
+ "rappel_petites": 0.6385,
122
+ "rappel_grandes": 0.7555,
123
+ "rappel_40_200": 0.6085,
124
+ "rappel_200_1000": 0.6838,
125
+ "rappel_tous": 0.6565,
126
+ "surface_correcte": 0.8352,
127
+ "F1_failles": 0.9868
128
+ },
129
+ {
130
+ "groupe": "GEO-LiM / synthetic",
131
+ "tuiles": 120,
132
+ "ARAND": 0.0986,
133
+ "sur_decoupage": 0.6392,
134
+ "fusion": 0.2721,
135
+ "rappel_petites": 0.8224,
136
+ "rappel_grandes": 0.9041,
137
+ "rappel_40_200": 0.781,
138
+ "rappel_200_1000": 0.8925,
139
+ "rappel_tous": 0.8386,
140
+ "surface_correcte": 0.9202,
141
+ "F1_failles": NaN
142
+ },
143
+ {
144
+ "groupe": "GMNA / real",
145
+ "tuiles": 78,
146
+ "ARAND": 0.0518,
147
+ "sur_decoupage": 0.2378,
148
+ "fusion": 0.2227,
149
+ "rappel_petites": 0.7467,
150
+ "rappel_grandes": 0.9341,
151
+ "rappel_40_200": 0.5038,
152
+ "rappel_200_1000": 0.8611,
153
+ "rappel_tous": 0.8336,
154
+ "surface_correcte": 0.9828,
155
+ "F1_failles": 0.6303
156
+ },
157
+ {
158
+ "groupe": "GMNA / synthetic",
159
+ "tuiles": 60,
160
+ "ARAND": 0.0244,
161
+ "sur_decoupage": 0.1356,
162
+ "fusion": 0.1323,
163
+ "rappel_petites": 0.8918,
164
+ "rappel_grandes": 0.977,
165
+ "rappel_40_200": 0.799,
166
+ "rappel_200_1000": 0.9366,
167
+ "rappel_tous": 0.929,
168
+ "surface_correcte": 0.9911,
169
+ "F1_failles": 0.9717
170
+ },
171
+ {
172
+ "groupe": "UNESCO-Ethiopia / synthetic",
173
+ "tuiles": 70,
174
+ "ARAND": 0.0299,
175
+ "sur_decoupage": 0.1534,
176
+ "fusion": 0.1099,
177
+ "rappel_petites": 0.8958,
178
+ "rappel_grandes": 0.9785,
179
+ "rappel_40_200": 0.8,
180
+ "rappel_200_1000": 0.9475,
181
+ "rappel_tous": 0.9466,
182
+ "surface_correcte": 0.9931,
183
+ "F1_failles": NaN
184
+ },
185
+ {
186
+ "groupe": "USGS-California / synthetic",
187
+ "tuiles": 70,
188
+ "ARAND": 0.0766,
189
+ "sur_decoupage": 0.5323,
190
+ "fusion": 0.347,
191
+ "rappel_petites": 0.7412,
192
+ "rappel_grandes": 0.8937,
193
+ "rappel_40_200": 0.6809,
194
+ "rappel_200_1000": 0.7966,
195
+ "rappel_tous": 0.776,
196
+ "surface_correcte": 0.9462,
197
+ "F1_failles": 0.9708
198
+ },
199
+ {
200
+ "groupe": "USGS-Nevada / synthetic",
201
+ "tuiles": 60,
202
+ "ARAND": 0.2261,
203
+ "sur_decoupage": 0.9629,
204
+ "fusion": 0.4127,
205
+ "rappel_petites": 0.6904,
206
+ "rappel_grandes": 0.8674,
207
+ "rappel_40_200": 0.6424,
208
+ "rappel_200_1000": 0.7605,
209
+ "rappel_tous": 0.7195,
210
+ "surface_correcte": 0.8765,
211
+ "F1_failles": 0.9512
212
+ },
213
+ {
214
+ "groupe": "TEST COMPLET",
215
+ "tuiles": 608,
216
+ "ARAND": 0.1066,
217
+ "sur_decoupage": 0.5839,
218
+ "fusion": 0.3042,
219
+ "rappel_petites": 0.7045,
220
+ "rappel_grandes": 0.8619,
221
+ "rappel_40_200": 0.6578,
222
+ "rappel_200_1000": 0.7671,
223
+ "rappel_tous": 0.7365,
224
+ "surface_correcte": 0.9208,
225
+ "F1_failles": 0.9672
226
+ },
227
+ {
228
+ "groupe": "PIÈGE unités de même couleur",
229
+ "tuiles": 48,
230
+ "ARAND": 0.1508,
231
+ "sur_decoupage": 0.6236,
232
+ "fusion": 0.4517,
233
+ "rappel_petites": 0.6281,
234
+ "rappel_grandes": 0.7687,
235
+ "rappel_40_200": 0.5926,
236
+ "rappel_200_1000": 0.6694,
237
+ "rappel_tous": 0.6606,
238
+ "surface_correcte": 0.8492,
239
+ "F1_failles": 0.9746
240
+ },
241
+ {
242
+ "groupe": "PIÈGE unités à motifs",
243
+ "tuiles": 48,
244
+ "ARAND": 0.0994,
245
+ "sur_decoupage": 0.5187,
246
+ "fusion": 0.2895,
247
+ "rappel_petites": 0.7434,
248
+ "rappel_grandes": 0.8976,
249
+ "rappel_40_200": 0.6931,
250
+ "rappel_200_1000": 0.7994,
251
+ "rappel_tous": 0.7802,
252
+ "surface_correcte": 0.9394,
253
+ "F1_failles": 0.9774
254
+ }
255
+ ],
256
+ "train_hours": 5.29
257
+ }
training/test_examples.png ADDED

Git LFS Details

  • SHA256: 2f930df47fdd7bfd1360b0e5f990c77a892f717c11908e607faa937357989fc4
  • Pointer size: 132 Bytes
  • Size of remote file: 2.03 MB
training/test_metrics.csv ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ groupe,tuiles,ARAND,sur_decoupage,fusion,rappel_petites,rappel_grandes,rappel_40_200,rappel_200_1000,rappel_tous,surface_correcte,F1_failles
2
+ Africa,70,0.0299,0.1534,0.1099,0.8958,0.9785,0.8,0.9475,0.9466,0.9931,
3
+ Europe,120,0.0986,0.6392,0.2721,0.8224,0.9041,0.781,0.8925,0.8386,0.9202,
4
+ North America,268,0.0912,0.4542,0.2775,0.7249,0.9054,0.6559,0.7991,0.7677,0.9507,0.9503
5
+ Oceania,150,0.1763,0.9725,0.4681,0.6385,0.7555,0.6085,0.6838,0.6565,0.8352,0.9868
6
+ GA-Australia / synthetic,150,0.1763,0.9725,0.4681,0.6385,0.7555,0.6085,0.6838,0.6565,0.8352,0.9868
7
+ GEO-LiM / synthetic,120,0.0986,0.6392,0.2721,0.8224,0.9041,0.781,0.8925,0.8386,0.9202,
8
+ GMNA / real,78,0.0518,0.2378,0.2227,0.7467,0.9341,0.5038,0.8611,0.8336,0.9828,0.6303
9
+ GMNA / synthetic,60,0.0244,0.1356,0.1323,0.8918,0.977,0.799,0.9366,0.929,0.9911,0.9717
10
+ UNESCO-Ethiopia / synthetic,70,0.0299,0.1534,0.1099,0.8958,0.9785,0.8,0.9475,0.9466,0.9931,
11
+ USGS-California / synthetic,70,0.0766,0.5323,0.347,0.7412,0.8937,0.6809,0.7966,0.776,0.9462,0.9708
12
+ USGS-Nevada / synthetic,60,0.2261,0.9629,0.4127,0.6904,0.8674,0.6424,0.7605,0.7195,0.8765,0.9512
13
+ TEST COMPLET,608,0.1066,0.5839,0.3042,0.7045,0.8619,0.6578,0.7671,0.7365,0.9208,0.9672
14
+ PIÈGE unités de même couleur,48,0.1508,0.6236,0.4517,0.6281,0.7687,0.5926,0.6694,0.6606,0.8492,0.9746
15
+ PIÈGE unités à motifs,48,0.0994,0.5187,0.2895,0.7434,0.8976,0.6931,0.7994,0.7802,0.9394,0.9774
training/train_final.py ADDED
@@ -0,0 +1,471 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # =====================================================================================
2
+ # GEOCARTO – ENTRAÎNEMENT FINAL
3
+ # Modèle retenu par le labo (rounds 1-3) : U-Net++ / EfficientNet-B4, entrée RGB+Lab,
4
+ # augmentations de couleur, affinités (10 décalages) + failles, mutex watershed.
5
+ # Données : dataset GeoCarto complet (6 600 tuiles, 4 continents), split spatial.
6
+ # Sorties (/kaggle/working/geocarto_model) :
7
+ # best.pt (poids + config), best_fp16.pt, history.csv, test_metrics.csv, metrics.json, figures
8
+ # =====================================================================================
9
+ import os, io, sys, json, time, copy
10
+ from pathlib import Path
11
+ import numpy as np, pandas as pd
12
+ import pyarrow.parquet as pq
13
+ from PIL import Image
14
+ from scipy import ndimage as ndi
15
+ from skimage.metrics import adapted_rand_error, variation_of_information
16
+ import torch, torch.nn as nn, torch.nn.functional as F
17
+ from torch.utils.data import Dataset, DataLoader
18
+ import segmentation_models_pytorch as smp
19
+ from numba import njit
20
+
21
+ E = os.environ.get
22
+ STEPS = int(E("GC_STEPS", 30000)) # pas d'entraînement
23
+ BATCH, CROP, LR = int(E("GC_BATCH", 8)), int(E("GC_CROP", 448)), float(E("GC_LR", 3e-4))
24
+ EVAL_EVERY = int(E("GC_EVAL_EVERY", 2500))
25
+ N_VAL = int(E("GC_NVAL", 12)) # tuiles de validation par source (suivi pendant l'entraînement)
26
+ P_REAL = float(E("GC_PREAL", 0.35)) # part de tuiles RÉELLES dans chaque lot (elles ne font que 21 % du dataset)
27
+ MAX_HOURS = float(E("GC_MAX_HOURS", 8.0)) # garde-fou : on arrête l'entraînement avant la limite Kaggle (12 h)
28
+ N_TEST = int(E("GC_NTEST", 0)) # 0 = tout le jeu de test
29
+ OUT = Path(E("GC_OUT", "/kaggle/working/geocarto_model")); OUT.mkdir(parents=True, exist_ok=True)
30
+ DEV = "cuda" if torch.cuda.is_available() else "cpu"
31
+ T_START = time.time()
32
+ print("Matériel :", DEV, torch.cuda.get_device_name(0) if DEV == "cuda" else "", flush=True)
33
+
34
+ CFG = dict(arch="unetpp", encoder=E("GC_ENCODER", "efficientnet-b4"), input="rgb+lab",
35
+ offsets=[(1, 0), (0, 1), (3, 0), (0, 3), (9, 0), (0, 9), (9, 9), (9, -9), (27, 0), (0, 27)],
36
+ n_short=2, mws_stride=int(E("GC_MWS_STRIDE", 1)), mws_min_size=int(E("GC_MINSIZE", 40)), tile=512,
37
+ small_w=E("GC_SMALLW", "0") == "1", ohem=E("GC_OHEM", "0") == "1", tta=E("GC_TTA", "0") == "1")
38
+ OFFSETS, N_SHORT, K = CFG["offsets"], CFG["n_short"], len(CFG["offsets"])
39
+
40
+
41
+ # ------------------------------------------------------------------ données
42
+ def find_dir(marker):
43
+ root = Path(E("GC_INPUT", "/kaggle/input"))
44
+ hits = sorted(root.rglob(marker))
45
+ return hits[0].parent if hits else None
46
+
47
+
48
+ DATA = Path(E("GC_DATA")) if E("GC_DATA") else find_dir("test-00000.parquet") # dataset complet
49
+ HARD = Path(E("GC_HARD")) if E("GC_HARD") else find_dir("hard_mono-00000.parquet") # jeux pièges (geocarto-mini)
50
+ assert DATA is not None, "dataset GeoCarto introuvable (ajoute la sortie du notebook geocarto-dataset en entrée)"
51
+ print("Données :", DATA, "| pièges :", HARD, flush=True)
52
+
53
+
54
+ def read_split(folder, split):
55
+ """Garde les PNG compressés en mémoire (2 Go au lieu de 11 Go décodés) ; décodage à la volée."""
56
+ rows = []
57
+ for f in sorted(Path(folder).glob(f"{split}-*.parquet")):
58
+ t = pq.read_table(f, columns=["id", "source", "continent", "kind", "image", "instances", "faults"]).to_pylist()
59
+ rows += [dict(id=r["id"], source=r["source"], continent=r["continent"], kind=r["kind"],
60
+ image=r["image"]["bytes"], inst=r["instances"]["bytes"], faults=r["faults"]["bytes"]) for r in t]
61
+ return rows
62
+
63
+
64
+ def decode(r):
65
+ rgb = np.asarray(Image.open(io.BytesIO(r["image"])).convert("RGB"))
66
+ inst = np.asarray(Image.open(io.BytesIO(r["inst"]))).astype(np.int32)
67
+ flt = (np.asarray(Image.open(io.BytesIO(r["faults"]))) > 0).astype(np.uint8)
68
+ return rgb, inst, flt
69
+
70
+
71
+ TRAIN = read_split(DATA, "train")
72
+ VAL_ALL = read_split(DATA, "validation")
73
+ TEST = read_split(DATA, "test")
74
+ HARDSETS = {k: read_split(HARD, k) for k in ("hard_mono", "hard_pattern")} if HARD else {}
75
+ REAL_IDX = [i for i, r in enumerate(TRAIN) if r["kind"] == "real"]
76
+ SYN_IDX = [i for i, r in enumerate(TRAIN) if r["kind"] != "real"]
77
+
78
+
79
+ def spread(rows, n):
80
+ return [rows[i] for i in np.linspace(0, len(rows) - 1, min(n, len(rows))).astype(int)] if rows else []
81
+
82
+
83
+ VAL = {} # suivi : n tuiles par (source, type), réparties régulièrement
84
+ for key in sorted({(r["source"], r["kind"]) for r in VAL_ALL}):
85
+ VAL[key] = spread([r for r in VAL_ALL if (r["source"], r["kind"]) == key], N_VAL * (3 if key[1] == "real" else 1))
86
+ print(f"train {len(TRAIN)} ({len(REAL_IDX)} réelles) | validation suivie {sum(map(len, VAL.values()))} | test {len(TEST)} | "
87
+ + " | ".join(f"{k} {len(v)}" for k, v in HARDSETS.items()), flush=True)
88
+
89
+
90
+ # ------------------------------------------------------------------ affinités + mutex watershed
91
+ def fill_holes(inst):
92
+ """Chaque zone sans polygone (mer, lac, halo) devient sa propre r��gion."""
93
+ holes, n = ndi.label(inst == 0)
94
+ if n == 0:
95
+ return inst
96
+ out = inst.copy()
97
+ out[holes > 0] = holes[holes > 0] + inst.max()
98
+ return out
99
+
100
+
101
+ def affinity_targets(inst):
102
+ H, W = inst.shape
103
+ aff, mask = np.zeros((K, H, W), np.float32), np.zeros((K, H, W), np.float32)
104
+ for k, (dy, dx) in enumerate(OFFSETS):
105
+ ys, ye, xs, xe = max(0, -dy), H - max(0, dy), max(0, -dx), W - max(0, dx)
106
+ a, b = inst[ys:ye, xs:xe], inst[ys + dy:ye + dy, xs + dx:xe + dx]
107
+ aff[k, ys:ye, xs:xe] = a == b
108
+ mask[k, ys:ye, xs:xe] = 1 # après fill_holes, tous les pixels ont une région
109
+ return aff, mask
110
+
111
+
112
+ @njit
113
+ def _find(parent, x):
114
+ while parent[x] != x:
115
+ parent[x] = parent[parent[x]]
116
+ x = parent[x]
117
+ return x
118
+
119
+
120
+ @njit
121
+ def _mws_core(n, u, v, rep):
122
+ parent = np.arange(n); size = np.ones(n, np.int64); m = len(u)
123
+ mutex_a = np.empty(m, np.int64); mutex_b = np.empty(m, np.int64); nm = 0
124
+ head = -np.ones(n, np.int64); nxt = np.empty(2 * m, np.int64); val = np.empty(2 * m, np.int64); nl = 0
125
+ for e in range(m):
126
+ ra, rb = _find(parent, u[e]), _find(parent, v[e])
127
+ if ra == rb:
128
+ continue
129
+ if rep[e]:
130
+ val[nl] = nm; nxt[nl] = head[ra]; head[ra] = nl; nl += 1
131
+ val[nl] = nm; nxt[nl] = head[rb]; head[rb] = nl; nl += 1
132
+ mutex_a[nm] = ra; mutex_b[nm] = rb; nm += 1
133
+ else:
134
+ small, other = (ra, rb) if size[ra] <= size[rb] else (rb, ra)
135
+ blocked = False; p = head[small]
136
+ while p != -1:
137
+ k = val[p]; x = _find(parent, mutex_a[k]); y = _find(parent, mutex_b[k])
138
+ if (x == small and y == other) or (x == other and y == small):
139
+ blocked = True; break
140
+ p = nxt[p]
141
+ if blocked:
142
+ continue
143
+ parent[small] = other; size[other] += size[small]; p = head[small]
144
+ if p != -1:
145
+ while nxt[p] != -1:
146
+ p = nxt[p]
147
+ nxt[p] = head[other]; head[other] = head[small]
148
+ out = np.empty(n, np.int64)
149
+ for i in range(n):
150
+ out[i] = _find(parent, i)
151
+ return out
152
+
153
+
154
+ def mutex_watershed(aff, stride=CFG["mws_stride"], min_size=CFG["mws_min_size"]):
155
+ _, H, W = aff.shape
156
+ idx = np.arange(H * W).reshape(H, W)
157
+ us, vs, ws, rep = [], [], [], []
158
+ for k, (dy, dx) in enumerate(OFFSETS):
159
+ a = idx[max(0, -dy):H - max(0, dy), max(0, -dx):W - max(0, dx)]
160
+ b = idx[max(0, dy):H - max(0, -dy), max(0, dx):W - max(0, -dx)]
161
+ w = aff[k, max(0, -dy):H - max(0, dy), max(0, -dx):W - max(0, dx)].copy()
162
+ if k >= N_SHORT:
163
+ a, b, w = a[::stride, ::stride], b[::stride, ::stride], 1 - w[::stride, ::stride]
164
+ us.append(a.ravel()); vs.append(b.ravel()); ws.append(w.ravel()); rep.append(np.full(a.size, k >= N_SHORT))
165
+ u, v, w, rep = map(np.concatenate, (us, vs, ws, rep))
166
+ order = np.argsort(-w, kind="stable")
167
+ lab = _mws_core(H * W, u[order], v[order], rep[order]).reshape(H, W)
168
+ _, lab = np.unique(lab, return_inverse=True); lab = lab.reshape(H, W) + 1
169
+ sizes = np.bincount(lab.ravel()); tiny = sizes < min_size; tiny[0] = False
170
+ if tiny[lab].any():
171
+ _, (iy, ix) = ndi.distance_transform_edt(tiny[lab], return_indices=True)
172
+ lab = lab[iy, ix]; _, lab = np.unique(lab, return_inverse=True); lab = lab.reshape(H, W) + 1
173
+ return lab.astype(np.int32)
174
+
175
+
176
+ # ------------------------------------------------------------------ augmentations
177
+ def color_aug(rgb, rng):
178
+ """Casse le lien 'couleur = unité' : rotation de teinte, saturation, gains, permutation, gris, contraste, bruit."""
179
+ x = rgb.astype(np.float32) / 255
180
+ if rng.random() < 0.8:
181
+ T = np.array([[0.299, 0.587, 0.114], [0.596, -0.274, -0.322], [0.211, -0.523, 0.312]], np.float32)
182
+ a = rng.uniform(-np.pi, np.pi)
183
+ Rm = np.array([[1, 0, 0], [0, np.cos(a), -np.sin(a)], [0, np.sin(a), np.cos(a)]], np.float32)
184
+ Rm[1:, 1:] *= rng.uniform(0.6, 1.3)
185
+ x = x @ (np.linalg.inv(T) @ Rm @ T).T
186
+ if rng.random() < 0.3:
187
+ x = x[..., rng.permutation(3)]
188
+ x = x * rng.uniform(0.85, 1.15, 3) + rng.uniform(-0.06, 0.06, 3)
189
+ if rng.random() < 0.1:
190
+ x = np.repeat(x.mean(-1, keepdims=True), 3, -1)
191
+ m = x.mean()
192
+ x = (x - m) * rng.uniform(0.75, 1.2) + m
193
+ if rng.random() < 0.3:
194
+ x = x + rng.normal(0, rng.uniform(0.005, 0.03), x.shape)
195
+ return np.clip(x * 255, 0, 255).astype(np.uint8)
196
+
197
+
198
+ class TrainSet(Dataset):
199
+ def __len__(self):
200
+ return STEPS * BATCH
201
+
202
+ def __getitem__(self, i):
203
+ rng = np.random.default_rng([i, 20261112])
204
+ pool = REAL_IDX if (REAL_IDX and rng.random() < P_REAL) else SYN_IDX
205
+ rgb, inst, flt = decode(TRAIN[pool[rng.integers(len(pool))]])
206
+ inst = fill_holes(inst) # sur la tuile entière, avant recadrage
207
+ y, x = rng.integers(0, rgb.shape[0] - CROP + 1), rng.integers(0, rgb.shape[1] - CROP + 1)
208
+ rgb, inst, flt = (a[y:y + CROP, x:x + CROP] for a in (rgb, inst, flt))
209
+ k = int(rng.integers(4))
210
+ rgb, inst, flt = (np.rot90(a, k) for a in (rgb, inst, flt))
211
+ if rng.random() < 0.5:
212
+ rgb, inst, flt = rgb[:, ::-1], inst[:, ::-1], flt[:, ::-1]
213
+ rgb = color_aug(rgb, rng)
214
+ inst = np.ascontiguousarray(inst)
215
+ aff, mask = affinity_targets(inst)
216
+ if CFG["small_w"]: # les arêtes touchant un petit polygone pèsent plus (jusqu'à x8)
217
+ area = np.bincount(inst.ravel())
218
+ w = np.clip(np.sqrt(2000.0 / np.maximum(area[inst], 1)), 1.0, 8.0).astype(np.float32)
219
+ H, W = inst.shape
220
+ for k, (dy, dx) in enumerate(OFFSETS):
221
+ ys, ye, xs, xe = max(0, -dy), H - max(0, dy), max(0, -dx), W - max(0, dx)
222
+ mask[k, ys:ye, xs:xe] = np.maximum(w[ys:ye, xs:xe], w[ys + dy:ye + dy, xs + dx:xe + dx])
223
+ return (torch.from_numpy(np.ascontiguousarray(rgb).transpose(2, 0, 1).copy()), torch.from_numpy(aff),
224
+ torch.from_numpy(mask), torch.from_numpy(np.ascontiguousarray(flt, np.float32))[None])
225
+
226
+
227
+ # ------------------------------------------------------------------ modèle
228
+ def rgb2lab(x):
229
+ """x : B x 3 x H x W dans [0,1] -> Lab normalisé (L/50-1, a/64, b/64)."""
230
+ lin = torch.where(x <= 0.04045, x / 12.92, ((x + 0.055) / 1.055) ** 2.4)
231
+ M = torch.tensor([[0.4124, 0.3576, 0.1805], [0.2126, 0.7152, 0.0722], [0.0193, 0.1192, 0.9505]], device=x.device)
232
+ xyz = torch.einsum("ij,bjhw->bihw", M, lin) / torch.tensor([0.95047, 1.0, 1.08883], device=x.device).view(1, 3, 1, 1)
233
+ f = torch.where(xyz > 0.008856, xyz.clamp(min=1e-6) ** (1 / 3), 7.787 * xyz + 16 / 116)
234
+ L, a, b = 116 * f[:, 1] - 16, 500 * (f[:, 0] - f[:, 1]), 200 * (f[:, 1] - f[:, 2])
235
+ return torch.stack([L / 50 - 1, a / 64, b / 64], 1)
236
+
237
+
238
+ class GeoCartoNet(nn.Module):
239
+ """Entrée : image uint8 B x 3 x H x W (H, W multiples de 32). Sortie : K affinités + 1 carte de failles (logits)."""
240
+
241
+ def __init__(self, cfg, pretrained=True):
242
+ super().__init__()
243
+ self.cfg = cfg
244
+ self.register_buffer("mean", torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1))
245
+ self.register_buffer("std", torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1))
246
+ Arch = {"unet": smp.Unet, "unetpp": smp.UnetPlusPlus}[cfg["arch"]]
247
+ self.net = Arch(cfg["encoder"], encoder_weights="imagenet" if pretrained else None, in_channels=6,
248
+ classes=len(cfg["offsets"]) + 1)
249
+
250
+ def forward(self, x8):
251
+ with torch.autocast(x8.device.type, enabled=False): # conversion de couleur en précision complète
252
+ x = x8.float() / 255
253
+ x = torch.cat([(x - self.mean) / self.std, rgb2lab(x)], 1)
254
+ return self.net(x)
255
+
256
+
257
+ def dice(p, t, eps=1.0, dims=None):
258
+ return 1 - (2 * (p * t).sum(dims) + eps) / (p.sum(dims) + t.sum(dims) + eps)
259
+
260
+
261
+ def loss_fn(out, aff, mask, flt):
262
+ bce = F.binary_cross_entropy_with_logits(out[:, :K], aff, reduction="none") * mask
263
+ la = bce.sum() / mask.sum().clamp(min=1)
264
+ if CFG["ohem"]: # + moyenne des 5 % d'arêtes les plus mal prédites
265
+ v = bce.flatten()
266
+ la = la + v.topk(max(1, v.numel() // 20)).values.mean() / mask.mean().clamp(min=1e-6)
267
+ p, t = (1 - torch.sigmoid(out[:, :K])) * mask, (1 - aff) * mask # Dice sur les COUPURES (classe rare)
268
+ la = la + dice(p, t, dims=(0, 2, 3)).mean()
269
+ lf = F.binary_cross_entropy_with_logits(out[:, K:], flt, pos_weight=torch.tensor(10.0, device=out.device)) \
270
+ + dice(torch.sigmoid(out[:, K:]), flt)
271
+ return la + 0.5 * lf
272
+
273
+
274
+ # ------------------------------------------------------------------ évaluation
275
+ @torch.no_grad()
276
+ def predict(model, rgb):
277
+ x = torch.from_numpy(rgb.transpose(2, 0, 1).copy())[None].to(DEV)
278
+ with torch.autocast(DEV, enabled=DEV == "cuda"):
279
+ o = model(x)
280
+ o = torch.sigmoid(o.float())[0].cpu().numpy()
281
+ return o[:K], o[K]
282
+
283
+
284
+ def predict_tta(model, rgb):
285
+ """Moyenne sur 4 miroirs ; chaque décalage ramené dans le repère d'origine (testé : exact)."""
286
+ H, W = rgb.shape[:2]
287
+ Y, X = np.mgrid[:H, :W]
288
+ idx = {tuple(o): k for k, o in enumerate(OFFSETS)}
289
+ acc, accf = 0, 0
290
+ for fy in (0, 1):
291
+ for fx in (0, 1):
292
+ im = rgb[::-1] if fy else rgb
293
+ im = im[:, ::-1] if fx else im
294
+ A, pf = predict(model, np.ascontiguousarray(im))
295
+ py, px = (H - 1 - Y) if fy else Y, (W - 1 - X) if fx else X
296
+ out = np.empty_like(A)
297
+ for k, (dy, dx) in enumerate(OFFSETS):
298
+ t = (-dy if fy else dy, -dx if fx else dx)
299
+ if t in idx:
300
+ out[k] = A[idx[t]][py, px]
301
+ else:
302
+ out[k] = A[idx[(-t[0], -t[1])]][np.clip(py + t[0], 0, H - 1), np.clip(px + t[1], 0, W - 1)]
303
+ acc = acc + out; accf = accf + pf[py, px]
304
+ return acc / 4, accf / 4
305
+
306
+
307
+ def tile_metrics(pred, pf, inst, flt):
308
+ gt = fill_holes(inst)
309
+ are = adapted_rand_error(gt, pred)[0]
310
+ split, merge = variation_of_information(gt, pred) # H(pred|vérité) = sur-découpage ; H(vérité|pred) = fusion
311
+ small = [0, 0]; big = [0, 0]
312
+ ids, area = np.unique(inst[inst > 0], return_counts=True)
313
+ if len(ids):
314
+ # IoU de chaque instance avec la région prédite qui la recouvre le plus
315
+ pair = inst.astype(np.int64) * (pred.max() + 1) + pred
316
+ pv, pc = np.unique(pair[inst > 0], return_counts=True)
317
+ gi, pi = pv // (pred.max() + 1), pv % (pred.max() + 1)
318
+ parea = np.bincount(pred.ravel())
319
+ order = np.lexsort((-pc, gi))
320
+ first = np.r_[True, gi[order][1:] != gi[order][:-1]]
321
+ g, inter, pa = gi[order][first], pc[order][first], parea[pi[order][first]]
322
+ a = area[np.searchsorted(ids, g)]
323
+ ok = inter / (a + pa - inter) >= 0.5
324
+ small = [int(ok[a < 1000].sum()), int((a < 1000).sum())]; big = [int(ok[a >= 1000].sum()), int((a >= 1000).sum())]
325
+ mid = [int(ok[(a >= 200) & (a < 1000)].sum()), int(((a >= 200) & (a < 1000)).sum())]
326
+ tiny = [int(ok[a < 200].sum()), int((a < 200).sum())]
327
+ surf = [float((a * ok).sum()), float(a.sum())]
328
+ else:
329
+ mid = tiny = [0, 0]; surf = [0.0, 0.0]
330
+ pb, gb = pf > 0.5, flt > 0
331
+ gd, pd_ = ndi.binary_dilation(gb, iterations=2), ndi.binary_dilation(pb, iterations=2)
332
+ return dict(ARAND=are, split=split, merge=merge, s_ok=small[0], s_n=small[1], b_ok=big[0], b_n=big[1],
333
+ t_ok=tiny[0], t_n=tiny[1], m_ok=mid[0], m_n=mid[1], surf_ok=surf[0], surf_n=surf[1],
334
+ tp=int((pb & gd).sum()), fp=int((pb & ~gd).sum()), fn=int((gb & ~pd_).sum()))
335
+
336
+
337
+ def summarize(rows):
338
+ d = pd.DataFrame(rows)
339
+ f1 = 2 * d.tp.sum() / max(2 * d.tp.sum() + d.fp.sum() + d.fn.sum(), 1) if (d.tp.sum() + d.fn.sum()) > 0 else np.nan
340
+ return dict(tuiles=len(d), ARAND=d.ARAND.mean(), sur_decoupage=d.split.mean(), fusion=d["merge"].mean(),
341
+ rappel_petites=d.s_ok.sum() / max(d.s_n.sum(), 1), rappel_grandes=d.b_ok.sum() / max(d.b_n.sum(), 1),
342
+ rappel_40_200=d.t_ok.sum() / max(d.t_n.sum(), 1), rappel_200_1000=d.m_ok.sum() / max(d.m_n.sum(), 1),
343
+ rappel_tous=(d.s_ok.sum() + d.b_ok.sum()) / max(d.s_n.sum() + d.b_n.sum(), 1),
344
+ surface_correcte=d.surf_ok.sum() / max(d.surf_n.sum(), 1), F1_failles=f1)
345
+
346
+
347
+ def evaluate(model, rows, tta=False):
348
+ model.eval()
349
+ out = []
350
+ for r in rows:
351
+ rgb, inst, flt = decode(r)
352
+ aff, pf = predict_tta(model, rgb) if tta else predict(model, rgb)
353
+ out.append(tile_metrics(mutex_watershed(aff), pf, inst, flt))
354
+ return out
355
+
356
+
357
+ def val_score(model):
358
+ """Score de suivi : moyenne de l'ARAND réel et de l'ARAND synthétique (toutes sources à poids égal)."""
359
+ res = {k: summarize(evaluate(model, v)) for k, v in VAL.items()}
360
+ real = np.mean([v["ARAND"] for k, v in res.items() if k[1] == "real"])
361
+ syn = np.mean([v["ARAND"] for k, v in res.items() if k[1] != "real"])
362
+ return 0.5 * (real + syn), real, syn, res
363
+
364
+
365
+ # ------------------------------------------------------------------ entraînement
366
+ torch.manual_seed(0)
367
+ model = GeoCartoNet(CFG).to(DEV)
368
+ ema = copy.deepcopy(model).eval() # moyenne glissante des poids (souvent plus stable)
369
+ for p in ema.parameters():
370
+ p.requires_grad_(False)
371
+ EMA_DECAY = 0.999
372
+ dl = DataLoader(TrainSet(), BATCH, shuffle=False, num_workers=min(4, os.cpu_count() or 1), pin_memory=DEV == "cuda",
373
+ drop_last=True)
374
+ opt = torch.optim.AdamW(model.parameters(), LR, weight_decay=1e-4)
375
+ sched = torch.optim.lr_scheduler.OneCycleLR(opt, LR, total_steps=STEPS, pct_start=0.05)
376
+ scaler = torch.amp.GradScaler("cuda", enabled=DEV == "cuda")
377
+ mutex_watershed(np.random.rand(K, 64, 64).astype(np.float32)) # compilation numba
378
+ print(f"Paramètres : {sum(p.numel() for p in model.parameters()) / 1e6:.1f} M | {STEPS} pas, lot {BATCH}, recadrage {CROP}", flush=True)
379
+
380
+ HIST, best, t0, run_loss, n_loss = [], 1e9, time.time(), 0.0, 0
381
+ model.train()
382
+ step = 0
383
+ for step, batch in enumerate(dl, 1):
384
+ x, aff, mask, flt = (t.to(DEV, non_blocking=True) for t in batch)
385
+ with torch.autocast(DEV, enabled=DEV == "cuda"):
386
+ out = model(x)
387
+ loss = loss_fn(out.float(), aff, mask, flt)
388
+ if not torch.isfinite(loss):
389
+ print(f" pas {step} : perte non finie, lot ignoré", flush=True)
390
+ opt.zero_grad(set_to_none=True); sched.step()
391
+ continue
392
+ opt.zero_grad(set_to_none=True)
393
+ scaler.scale(loss).backward()
394
+ scaler.unscale_(opt); torch.nn.utils.clip_grad_norm_(model.parameters(), 5.0)
395
+ scaler.step(opt); scaler.update(); sched.step()
396
+ with torch.no_grad():
397
+ d = min(EMA_DECAY, (1 + step) / (10 + step))
398
+ for pe, pm in zip(ema.parameters(), model.parameters()):
399
+ pe.mul_(d).add_(pm.detach(), alpha=1 - d)
400
+ for be, bm in zip(ema.buffers(), model.buffers()):
401
+ be.copy_(bm)
402
+ run_loss += loss.item(); n_loss += 1
403
+ if step % 250 == 0:
404
+ print(f"pas {step}/{STEPS} | perte {run_loss / n_loss:.4f} | {(time.time() - t0) / 60:.0f} min", flush=True)
405
+ hours = (time.time() - T_START) / 3600
406
+ last = step == STEPS or hours > MAX_HOURS
407
+ if step % EVAL_EVERY == 0 or last:
408
+ row = dict(step=step, loss=run_loss / max(n_loss, 1), heures=round(hours, 2))
409
+ for tag, m in (("modele", model), ("ema", ema)):
410
+ sc, real, syn, _ = val_score(m)
411
+ row.update({f"{tag}_score": sc, f"{tag}_reel": real, f"{tag}_synth": syn})
412
+ if sc < best:
413
+ best = sc
414
+ torch.save({"state_dict": m.state_dict(), "cfg": CFG, "step": step, "val_score": float(sc), "poids": tag}, OUT / "best.pt")
415
+ row["meilleur"] = tag
416
+ model.train()
417
+ HIST.append(row); pd.DataFrame(HIST).to_csv(OUT / "history.csv", index=False)
418
+ print(" VALIDATION", json.dumps({k: (round(v, 4) if isinstance(v, float) else v) for k, v in row.items()}), flush=True)
419
+ run_loss, n_loss = 0.0, 0
420
+ if last:
421
+ if step < STEPS:
422
+ print(f"Arrêt au pas {step} : limite de {MAX_HOURS} h atteinte.", flush=True)
423
+ break
424
+ del dl
425
+
426
+ # ------------------------------------------------------------------ test final (meilleur point de contrôle)
427
+ ck = torch.load(OUT / "best.pt", map_location=DEV)
428
+ model = GeoCartoNet(CFG, pretrained=False).to(DEV)
429
+ model.load_state_dict(ck["state_dict"]); model.eval()
430
+ torch.save({"state_dict": {k: (v.half() if v.is_floating_point() else v) for k, v in ck["state_dict"].items()},
431
+ "cfg": CFG, "step": ck["step"]}, OUT / "best_fp16.pt")
432
+ print(f"\nMeilleur modèle : pas {ck['step']} ({ck['poids']}), score de validation {ck['val_score']:.4f}", flush=True)
433
+
434
+ test_rows = TEST if not N_TEST else spread(TEST, N_TEST)
435
+ res = evaluate(model, test_rows, tta=CFG["tta"])
436
+ T = pd.DataFrame(res).assign(continent=[r["continent"] for r in test_rows], source=[r["source"] for r in test_rows],
437
+ type=[r["kind"] for r in test_rows])
438
+ tables = []
439
+ for keys in (["continent"], ["source", "type"]):
440
+ for name, g in T.groupby(keys):
441
+ name = name if isinstance(name, tuple) else (name,)
442
+ tables.append({"groupe": " / ".join(name), **summarize(g.to_dict("records"))})
443
+ tables.append({"groupe": "TEST COMPLET", **summarize(res)})
444
+ for k, rows in HARDSETS.items():
445
+ rows = rows if not N_TEST else spread(rows, N_TEST)
446
+ tables.append({"groupe": {"hard_mono": "PIÈGE unités de même couleur", "hard_pattern": "PIÈGE unités à motifs"}[k],
447
+ **summarize(evaluate(model, rows, tta=CFG["tta"]))})
448
+ TM = pd.DataFrame(tables).round(4)
449
+ TM.to_csv(OUT / "test_metrics.csv", index=False)
450
+ pd.set_option("display.width", 220); pd.set_option("display.max_columns", 20)
451
+ print("\n=== RÉSULTATS SUR LE JEU DE TEST (jamais vu à l'entraînement) ==="); print(TM.to_string(index=False), flush=True)
452
+ json.dump({"cfg": CFG, "steps": int(ck["step"]), "val_score": float(ck["val_score"]), "test": TM.to_dict("records"),
453
+ "train_hours": round((time.time() - T_START) / 3600, 2)}, open(OUT / "metrics.json", "w"), indent=1, ensure_ascii=False)
454
+
455
+ # ------------------------------------------------------------------ figures
456
+ import matplotlib; matplotlib.use("Agg"); import matplotlib.pyplot as plt
457
+ picks = [("réel", spread([r for r in TEST if r["kind"] == "real"], 4))]
458
+ picks += [(s, spread([r for r in TEST if r["source"] == s and r["kind"] != "real"], 2)) for s in sorted({r["source"] for r in TEST})]
459
+ picks += [(k, spread(v, 3)) for k, v in HARDSETS.items()]
460
+ rows_ = [(n, r) for n, rs in picks for r in rs]
461
+ fig, ax = plt.subplots(3, len(rows_), figsize=(4 * len(rows_), 12))
462
+ for j, (n, r) in enumerate(rows_):
463
+ rgb, inst, flt = decode(r); aff, pf = predict(model, rgb); pred = mutex_watershed(aff)
464
+ ax[0, j].imshow(rgb); ax[0, j].set_title(f"{n}\n{r['source']}", fontsize=9)
465
+ ax[1, j].imshow(fill_holes(inst) % 20, cmap="tab20", interpolation="nearest"); ax[1, j].set_title("vérité", fontsize=9)
466
+ ax[2, j].imshow(pred % 20, cmap="tab20", interpolation="nearest")
467
+ ax[2, j].imshow(np.ma.masked_less(pf, 0.5), cmap="gray_r", interpolation="nearest"); ax[2, j].set_title("modèle", fontsize=9)
468
+ for a in ax.ravel():
469
+ a.axis("off")
470
+ plt.tight_layout(); plt.savefig(OUT / "test_examples.png", dpi=55); plt.close()
471
+ print("Fichiers :", sorted(p.name for p in OUT.iterdir()), f"| durée totale {(time.time() - T_START) / 3600:.2f} h", flush=True)
vectorize.py ADDED
@@ -0,0 +1,56 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """GeoCarto – vectorise a georeferenced geological map raster into a GeoPackage (QGIS / ArcGIS).
2
+
3
+ python vectorize.py carte.tif legende.csv sortie.gpkg [--model best_fp16.pt] [--tta]
4
+
5
+ legende.csv : one row per legend entry, columns
6
+ code (required, as printed on the map), nom, age, lithologie (optional),
7
+ and EITHER x0,y0,x1,y1 = pixel box of the legend swatch in carte.tif (best),
8
+ OR couleur_hex = colour of the swatch (e.g. #f2c38a).
9
+ Unit names come only from text read on the map and from this legend.
10
+ """
11
+ import argparse, os, sys
12
+ import numpy as np, pandas as pd, rasterio
13
+ from skimage.color import rgb2lab
14
+ HERE = os.path.dirname(os.path.abspath(__file__)); sys.path.insert(0, HERE)
15
+ import pipeline as PL
16
+
17
+
18
+ def load_legend(csv, rgb):
19
+ L = pd.read_csv(csv, dtype={"code": str}).fillna("")
20
+ lab = None
21
+ feats, hexes = [], []
22
+ for r in L.itertuples():
23
+ if all(hasattr(r, k) and str(getattr(r, k)) != "" for k in ("x0", "y0", "x1", "y1")):
24
+ x0, y0, x1, y1 = (int(float(getattr(r, k))) for k in ("x0", "y0", "x1", "y1"))
25
+ px = rgb[y0:y1, x0:x1].reshape(-1, 3)
26
+ feats.append(PL.swatch_features(rgb2lab(px[None])[0])); hexes.append("#%02x%02x%02x" % tuple(int(v) for v in np.median(px, 0)))
27
+ elif getattr(r, "couleur_hex", ""):
28
+ h = r.couleur_hex.lstrip("#"); c = np.array([[int(h[i:i + 2], 16) for i in (0, 2, 4)]], np.uint8)
29
+ feats.append(PL.swatch_features(np.repeat(rgb2lab(c[None])[0], 50, 0))); hexes.append("#" + h)
30
+ else:
31
+ raise SystemExit(f"légende : la ligne {r.code} n'a ni case (x0,y0,x1,y1) ni couleur_hex")
32
+ out = pd.DataFrame(index=L.code.values)
33
+ for c in ("nom", "age", "lithologie"):
34
+ out[c] = L[c].values if c in L else ""
35
+ out["nom"] = np.where(out.nom == "", out.index, out.nom)
36
+ out["couleur_hex"] = hexes; out["feat"] = feats
37
+ return out
38
+
39
+
40
+ if __name__ == "__main__":
41
+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
42
+ ap.add_argument("raster"); ap.add_argument("legend"); ap.add_argument("out")
43
+ ap.add_argument("--model", default=os.path.join(HERE, "best_fp16.pt"))
44
+ ap.add_argument("--tta", action="store_true", help="moyenne sur 4 symétries (plus lent, pas meilleur en moyenne)")
45
+ a = ap.parse_args()
46
+ import easyocr
47
+ with rasterio.open(a.raster) as s:
48
+ rgb = np.moveaxis(s.read([1, 2, 3]), 0, -1).astype(np.uint8); tr, crs = s.transform, s.crs
49
+ legend = load_legend(a.legend, rgb)
50
+ print(f"carte {rgb.shape[1]}×{rgb.shape[0]} px | légende : {len(legend)} unités")
51
+ model = PL.load_model(a.model)
52
+ reader = easyocr.Reader(["en"], gpu=False, verbose=False)
53
+ P, F, lab, pf = PL.run(rgb, tr, crs, model, legend, reader, os.path.join(HERE, "naming_model.pkl"), a.out,
54
+ tta=a.tta, ranker_pkl=os.path.join(HERE, "legend_ranker.pkl"))
55
+ n = P.code.ne("").sum()
56
+ print(f"\n{len(P)} unités ({n} nommées, {len(P) - n} 'Unité non identifiée' avec suggestions), {len(F)} failles -> {a.out}")