PerfectWin commited on
Commit
1c68a3f
·
verified ·
1 Parent(s): efdf9ef

v0.5: GeoRas2Vec model files, figures and card (code moved to GitHub)

Browse files
.gitattributes CHANGED
@@ -36,3 +36,9 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
36
  exemple/carte_geologique.gpkg filter=lfs diff=lfs merge=lfs -text
37
  exemple/rendu_qgis.png filter=lfs diff=lfs merge=lfs -text
38
  training/test_examples.png filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
36
  exemple/carte_geologique.gpkg filter=lfs diff=lfs merge=lfs -text
37
  exemple/rendu_qgis.png filter=lfs diff=lfs merge=lfs -text
38
  training/test_examples.png filter=lfs diff=lfs merge=lfs -text
39
+ figures/results_real_scans.png filter=lfs diff=lfs merge=lfs -text
40
+ figures/results_continents.png filter=lfs diff=lfs merge=lfs -text
41
+ figures/naming_region.png filter=lfs diff=lfs merge=lfs -text
42
+ figures/naming_zoom.png filter=lfs diff=lfs merge=lfs -text
43
+ figures/method_geometry.png filter=lfs diff=lfs merge=lfs -text
44
+ figures/method_naming.png filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -3,7 +3,7 @@ license: cc-by-sa-4.0
3
  library_name: pytorch
4
  pipeline_tag: image-segmentation
5
  datasets:
6
- - PerfectWin/geocarto
7
  tags:
8
  - geology
9
  - geological-map
@@ -13,89 +13,85 @@ tags:
13
  - qgis
14
  ---
15
 
16
- # GeoCarto model – geological map raster → GIS polygons, faults and unit names (GeoPackage)
17
 
18
- **Research preview (v0.2).** Turns a scanned / rasterised geological map into a GeoPackage that opens directly in
19
- QGIS or ArcGIS: one polygon layer of geological units (with code, name, age, lithology when they can be read),
20
- one line layer of faults, the legend table and default QGIS styles.
21
 
22
- Trained on [PerfectWin/geocarto](https://huggingface.co/datasets/PerfectWin/geocarto) **v2.0** (4 continents,
23
- real USGS GMNA scans + realistic synthetic maps).
24
 
25
- ## How it works
26
- 1. **Segmentation** – U-Net++ / EfficientNet-B4, input RGB + CIE-Lab (6 channels), predicts 10 pixel affinities
27
- (offsets 1–27 px) + a fault map; sliding window 512 px, stride 384, Hann blending.
28
- 2. **Polygons** – mutex watershed on the affinities (full resolution), minimum region 40 px, vectorised with rasterio.
29
- 3. **Faults** – skeleton of the fault map, traced into lines.
30
- 4. **Text** – EasyOCR on the whole map.
31
- 5. **Naming** – names come **only from text printed on the map and from the legend** (no propagation of names between
32
- polygons that merely look alike). A learned ranker (gradient boosting) scores every (polygon, candidate code) pair
33
- from: OCR evidence and leader-line features, distance to the legend swatch (relative to the other swatches),
34
- where else the code is read on the map. A name is written only when the confidence is ≥ 0.8; otherwise the polygon
35
- is `Unité non identifiée` with the 3 best ranked suggestions in the `suggestions` column.
36
- 6. **Export** – GeoPackage with layers `unites_geologiques`, `failles`, table `legende`, styles in `layer_styles`.
 
 
 
 
 
 
 
 
 
 
37
 
38
  ## Usage
39
  ```bash
 
40
  pip install -r requirements.txt
41
- python vectorize.py carte.tif legende.csv sortie.gpkg
42
  ```
43
- `carte.tif` must be georeferenced (any CRS). `legende.csv` has one row per legend entry: `code` (as printed on the
44
- map), optional `nom`, `age`, `lithologie`, and either the pixel box of the legend swatch in the raster
45
- (`x0,y0,x1,y1`, recommended) or its colour `couleur_hex`. See `exemple/legende_exemple.csv`.
46
- `qgis_test.py` checks a produced GeoPackage with PyQGIS (layers, styles, geometry validity, rendering).
47
-
48
- Attribute table of `unites_geologiques`: `id, code, nom, age, lithologie, source_nom` (code read on the map / legend
49
- colour / not identified), `confiance, texte_lu, suggestions, couleur_hex` (legend colour), `couleur_carte` (colour
50
- measured on the scan), `surface_km2`.
51
-
52
- ## Results (held-out test split, never seen in training, 608 tiles)
53
- Recall = share of reference polygons recovered with IoU ≥ 0.5. *Small* < 1000 px, *large* ≥ 1000 px (512 px tiles).
54
- *Correct area* = share of the map area that falls in a correctly recovered polygon.
55
-
56
- | group | ARAND ↓ | small recall | large recall | correct area | fault F1 (2 px) |
57
- |---|---|---|---|---|---|
58
- | Africa (synthetic) | 0.030 | 89.6 % | 97.9 % | 99.3 % | – |
59
- | Europe (synthetic) | 0.099 | 82.2 % | 90.4 % | 92.0 % | – |
60
- | North America (all) | 0.091 | 72.5 % | 90.5 % | 95.1 % | 0.95 |
61
- | Oceania (synthetic) | 0.176 | 63.9 % | 75.6 % | 83.5 % | 0.99 |
62
- | **GMNA real scans** | 0.052 | **74.7 %** | **93.4 %** | **98.3 %** | 0.63 |
63
- | **full test set** | 0.107 | 70.5 % | 86.2 % | 92.1 % | 0.97 |
64
- | trap set: neighbouring units with the same colour | 0.151 | 62.8 % | 76.9 % | 84.9 % | 0.97 |
65
- | trap set: patterned units | 0.099 | 74.3 % | 89.8 % | 93.9 % | 0.98 |
66
-
67
- Full table: `training/test_metrics.csv`; ablation of the training choices: `training/lab_round4_ablation.csv`
68
- (small-polygon loss re-weighting and OHEM were tested and rejected: they over-segment real maps).
69
-
70
- ### Naming (real GMNA scans, map sheet never used to train the ranker)
71
- End-to-end test (segmentation by this model + OCR + naming) on 9 real 2048 × 2048 px windows of the GMNA north sheet;
72
- legend = all units of the sheet, swatches measured outside the tested window; ranker trained on the south sheet only.
73
-
74
- | | value |
75
- |---|---|
76
- | area that receives a name (confidence ≥ 0.8) | **30.0 %** (per window 4 % – 57 %) |
77
- | name correct, on the named area | **91.6 %** (per window 76 % – 98 %) |
78
- | area correctly named, over the whole map | 27.5 % |
79
 
80
- Cross-sheet benchmark on reference polygons (38 windows, both directions): names given at confidence ≥ 0.8 are
81
- 94–96 % correct on ~1/3 of the area. The rest of the area is mostly units without a legible label in the window.
82
- Per-window results: `training/e2e_naming_9_zones.csv`. Example output: `exemple/carte_geologique.gpkg`
83
- (rendered in QGIS: `exemple/rendu_qgis.png`).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
84
 
85
  ## Limitations
86
- * Small units: about 1 in 4 small polygons on real scans is still missed or merged.
87
- * Naming depends on labels being printed and legible; unlabelled units are left as *Unité non identifiée* with
88
- suggestions rather than guessed. Text recognition is trained for Latin script codes.
89
- * Real-scan validation is so far only on the USGS Geologic Map of North America; other continents are evaluated on
90
- realistic synthetic renderings of official vector maps.
91
- * Fault detection on real scans (F1 0.63) is much weaker than on synthetic maps.
92
- * Large maps are processed in one piece for the watershed: keep inputs below ~6000 × 6000 px or split them.
93
 
94
- ## Files
95
- `best_fp16.pt` weights (step 22 500) · `gc_model.py` network + mutex watershed · `pipeline.py` full pipeline ·
96
- `naming_model.py`, `leader.py`, `naming_model.pkl` OCR-to-polygon model · `legend_ranker.pkl` naming ranker ·
97
- `vectorize.py` command line · `qml_*.xml` QGIS styles · `training/` training script, curves and metrics.
98
- The `.pkl` files are scikit-learn pickles: only load files from this repository.
 
 
99
 
100
  ## Citation
101
- Please cite the dataset sources listed in the [dataset card](https://huggingface.co/datasets/PerfectWin/geocarto).
 
 
3
  library_name: pytorch
4
  pipeline_tag: image-segmentation
5
  datasets:
6
+ - PerfectWin/georas2vec
7
  tags:
8
  - geology
9
  - geological-map
 
13
  - qgis
14
  ---
15
 
16
+ # GeoRas2Vec model – scanned geological map → GIS polygons, faults and unit names
17
 
18
+ **Research preview (v0.5).** **Code, installation and documentation: [GeoRas2Vec on GitHub](https://github.com/PerfectWin7777/georas2vec).**
 
 
19
 
20
+ GeoRas2Vec turns a scanned or rasterised geological map into a GeoPackage that opens directly in QGIS or ArcGIS. The GeoPackage holds three things:
 
21
 
22
+ - a polygon layer of geological units, with code, name, age and lithology when they can be read;
23
+ - a line layer of faults;
24
+ - the legend table, with default QGIS styles.
25
+
26
+ Unit names come **only from text printed on the map and from its legend**.
27
+
28
+ This repository holds the model files. The command-line tool downloads them automatically.
29
+
30
+ ![Real scanned maps](https://huggingface.co/PerfectWin/georas2vec-model/resolve/main/figures/results_real_scans.png)
31
+
32
+ ## Files
33
+ | file | role |
34
+ |---|---|
35
+ | `best_fp16.pt` | segmentation network (U-Net++ / EfficientNet-B4, RGB + Lab): 10 pixel affinities + fault map |
36
+ | `naming_model.pkl` | scores (text read on the map, polygon) pairs |
37
+ | `legend_ranker.pkl` | ranks the candidate codes of each polygon (text evidence, leader lines, legend swatch) |
38
+ | `prop_model.pkl` | decides when a code read on the map can be propagated to polygons of the same colour |
39
+ | `water_model.pkl` | recognises water / outside-the-map regions |
40
+ | `exemple/` | example output GeoPackage, legend CSV, QGIS render |
41
+ | `training/` | training curves and metrics of the network |
42
+
43
+ The `.pkl` files are scikit-learn pickles: only load files from this repository.
44
 
45
  ## Usage
46
  ```bash
47
+ git clone https://github.com/PerfectWin7777/georas2vec.git && cd georas2vec
48
  pip install -r requirements.txt
49
+ python scripts/vectorize.py my_map.tif my_legend.csv my_map.gpkg
50
  ```
51
+ See the [GitHub README](https://github.com/PerfectWin7777/georas2vec#quick-start) for the legend CSV format.
52
+
53
+ ## Results (never seen during training)
54
+ ![Official test split](https://huggingface.co/PerfectWin/georas2vec-model/resolve/main/figures/results_test_scores.png)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
55
 
56
+ | official test split | large units | small units | map area in the correct polygon |
57
+ |---|---|---|---|
58
+ | real scans (78 tiles, USGS GMNA 300 dpi) | **94.5 %** | **77.0 %** | **98.5 %** |
59
+ | synthetic maps, 4 continents (60 tiles) | 85.2 % | 73.6 % | 90.4 % |
60
+
61
+ Faults: line-level F1 at 2 px is **0.80** on the real-scan test split and **0.85** on a held-out region.
62
+
63
+ ![Four continents](https://huggingface.co/PerfectWin/georas2vec-model/resolve/main/figures/results_continents.png)
64
+
65
+ ### Names
66
+ End to end on a held-out 6144 × 6144 px region of the GMNA north sheet. Polygons and names are both predicted.
67
+ The ranker and the propagation model were trained without this region.
68
+
69
+ | | area named | names correct | area correctly named |
70
+ |---|---|---|---|
71
+ | names from text only | 65.9 % | 94.1 % | 62.0 % |
72
+ | **+ colour propagation** | **73.9 %** | **94.2 %** | **69.6 %** |
73
+
74
+ ![Names](https://huggingface.co/PerfectWin/georas2vec-model/resolve/main/figures/naming_region.png)
75
+
76
+ ## How it works
77
+ ![Geometry](https://huggingface.co/PerfectWin/georas2vec-model/resolve/main/figures/method_geometry.png)
78
+ ![Names](https://huggingface.co/PerfectWin/georas2vec-model/resolve/main/figures/method_naming.png)
79
 
80
  ## Limitations
81
+ * A unit whose code is never printed (or never readable) stays *Unité non identifiée*, with 3 suggestions.
82
+ * Sibling units with almost identical colours (e.g. `lK` / `K`) cause most propagation errors.
83
+ * Real-scan validation is so far on the USGS Geologic Map of North America. The other continents are evaluated
84
+ on realistic renderings of official vector maps.
85
+ * On large maps, unnamed areas can show straight cuts at the 2048 px block edges.
 
 
86
 
87
+ ## Versions
88
+ | version | change |
89
+ |---|---|
90
+ | **v0.5** | two-scale mutex watershed; fault lines split at junctions; topological contours (no gap, no overlap); water detection; OCR restricted to the legend alphabet; decoding of age prefixes, Ʀ, Є and superscripts; colour propagation over the whole map; block processing of large maps |
91
+ | v0.2 | first public release (`revision="v0.2"`) |
92
+
93
+ The repository was renamed from `PerfectWin/geocarto-model` to `PerfectWin/georas2vec-model` (old links redirect).
94
 
95
  ## Citation
96
+ Winter Tony, *GeoRas2Vec: from scanned geological maps to vector units, faults and names*, 2026 –
97
+ [GitHub](https://github.com/PerfectWin7777/georas2vec). Please also cite the data sources listed in the [dataset card](https://huggingface.co/datasets/PerfectWin/georas2vec).
exemple/carte_geologique.gpkg CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:5d49b7700611a18c8d02636eae0bdc6485eb661d5d625424eceae6b8ccc18a20
3
- size 557056
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:52d72d064a6607c334abc8ecaaf8d8d192b195e04ee1dc845334d92c2f350bf1
3
+ size 2760704
exemple/rendu_qgis.png CHANGED

Git LFS Details

  • SHA256: 3f04cbd9d8ee9ee2ebd2a5def850098439ae1ef4fd6298d4e4d820e40f95d99a
  • Pointer size: 131 Bytes
  • Size of remote file: 264 kB

Git LFS Details

  • SHA256: 13ae22bd6e1008aee33c75561b209299eab35050ccbfa234c16d4a48c537605f
  • Pointer size: 131 Bytes
  • Size of remote file: 532 kB
figures/method_geometry.png ADDED

Git LFS Details

  • SHA256: 564e6c47d55607960665d3b24d7f9388e62a9c287f9c20548ab0eda946c16c07
  • Pointer size: 132 Bytes
  • Size of remote file: 1.5 MB
figures/method_naming.png ADDED

Git LFS Details

  • SHA256: 86d635b7881f9863269b304096d7a193ff912199c5ac82e9ec45acd9b71d32b3
  • Pointer size: 132 Bytes
  • Size of remote file: 1.64 MB
figures/naming_region.png ADDED

Git LFS Details

  • SHA256: 0825d2766d7c84d48764ed42ad107572a100693634eade532ba5e980e51dc9f9
  • Pointer size: 132 Bytes
  • Size of remote file: 1.74 MB
figures/naming_zoom.png ADDED

Git LFS Details

  • SHA256: 902b4f894a6973902b36bd214b42af8a037d8d39ffac81a1a401b260c9d1827f
  • Pointer size: 132 Bytes
  • Size of remote file: 3.24 MB
figures/results_continents.png ADDED

Git LFS Details

  • SHA256: c96a6ee85d2d55940197bc18617927164eb733f6b27127c9bfe5e6bac0624678
  • Pointer size: 132 Bytes
  • Size of remote file: 2.83 MB
figures/results_real_scans.png ADDED

Git LFS Details

  • SHA256: abbc2c6e4acf22d1eb3fda346b8bfcfc4cf3d7f2618343f54bdb1c053693c0ff
  • Pointer size: 132 Bytes
  • Size of remote file: 1.76 MB
figures/results_test_scores.png ADDED
gc_model.py DELETED
@@ -1,127 +0,0 @@
1
- # Définitions du modèle GeoCarto (extraites de train_final.py) pour l'inférence hors Kaggle.
2
- import numpy as np, torch, torch.nn as nn
3
- from scipy import ndimage as ndi
4
- from numba import njit
5
- import segmentation_models_pytorch as smp
6
-
7
- CFG = dict(arch="unetpp", encoder="efficientnet-b4", input="rgb+lab",
8
- offsets=[(1, 0), (0, 1), (3, 0), (0, 3), (9, 0), (0, 9), (9, 9), (9, -9), (27, 0), (0, 27)],
9
- n_short=2, mws_stride=1, mws_min_size=40, tile=512)
10
- OFFSETS, N_SHORT, K = CFG["offsets"], CFG["n_short"], len(CFG["offsets"])
11
-
12
- # ------------------------------------------------------------------ affinités + mutex watershed
13
- def fill_holes(inst):
14
- """Chaque zone sans polygone (mer, lac, halo) devient sa propre région."""
15
- holes, n = ndi.label(inst == 0)
16
- if n == 0:
17
- return inst
18
- out = inst.copy()
19
- out[holes > 0] = holes[holes > 0] + inst.max()
20
- return out
21
-
22
-
23
- def affinity_targets(inst):
24
- H, W = inst.shape
25
- aff, mask = np.zeros((K, H, W), np.float32), np.zeros((K, H, W), np.float32)
26
- for k, (dy, dx) in enumerate(OFFSETS):
27
- ys, ye, xs, xe = max(0, -dy), H - max(0, dy), max(0, -dx), W - max(0, dx)
28
- a, b = inst[ys:ye, xs:xe], inst[ys + dy:ye + dy, xs + dx:xe + dx]
29
- aff[k, ys:ye, xs:xe] = a == b
30
- mask[k, ys:ye, xs:xe] = 1 # après fill_holes, tous les pixels ont une région
31
- return aff, mask
32
-
33
-
34
- @njit
35
- def _find(parent, x):
36
- while parent[x] != x:
37
- parent[x] = parent[parent[x]]
38
- x = parent[x]
39
- return x
40
-
41
-
42
- @njit
43
- def _mws_core(n, u, v, rep):
44
- parent = np.arange(n); size = np.ones(n, np.int64); m = len(u)
45
- mutex_a = np.empty(m, np.int64); mutex_b = np.empty(m, np.int64); nm = 0
46
- head = -np.ones(n, np.int64); nxt = np.empty(2 * m, np.int64); val = np.empty(2 * m, np.int64); nl = 0
47
- for e in range(m):
48
- ra, rb = _find(parent, u[e]), _find(parent, v[e])
49
- if ra == rb:
50
- continue
51
- if rep[e]:
52
- val[nl] = nm; nxt[nl] = head[ra]; head[ra] = nl; nl += 1
53
- val[nl] = nm; nxt[nl] = head[rb]; head[rb] = nl; nl += 1
54
- mutex_a[nm] = ra; mutex_b[nm] = rb; nm += 1
55
- else:
56
- small, other = (ra, rb) if size[ra] <= size[rb] else (rb, ra)
57
- blocked = False; p = head[small]
58
- while p != -1:
59
- k = val[p]; x = _find(parent, mutex_a[k]); y = _find(parent, mutex_b[k])
60
- if (x == small and y == other) or (x == other and y == small):
61
- blocked = True; break
62
- p = nxt[p]
63
- if blocked:
64
- continue
65
- parent[small] = other; size[other] += size[small]; p = head[small]
66
- if p != -1:
67
- while nxt[p] != -1:
68
- p = nxt[p]
69
- nxt[p] = head[other]; head[other] = head[small]
70
- out = np.empty(n, np.int64)
71
- for i in range(n):
72
- out[i] = _find(parent, i)
73
- return out
74
-
75
-
76
- def mutex_watershed(aff, stride=CFG["mws_stride"], min_size=CFG["mws_min_size"]):
77
- _, H, W = aff.shape
78
- idx = np.arange(H * W).reshape(H, W)
79
- us, vs, ws, rep = [], [], [], []
80
- for k, (dy, dx) in enumerate(OFFSETS):
81
- a = idx[max(0, -dy):H - max(0, dy), max(0, -dx):W - max(0, dx)]
82
- b = idx[max(0, dy):H - max(0, -dy), max(0, dx):W - max(0, -dx)]
83
- w = aff[k, max(0, -dy):H - max(0, dy), max(0, -dx):W - max(0, dx)].copy()
84
- if k >= N_SHORT:
85
- a, b, w = a[::stride, ::stride], b[::stride, ::stride], 1 - w[::stride, ::stride]
86
- us.append(a.ravel()); vs.append(b.ravel()); ws.append(w.ravel()); rep.append(np.full(a.size, k >= N_SHORT))
87
- u, v, w, rep = map(np.concatenate, (us, vs, ws, rep))
88
- order = np.argsort(-w, kind="stable")
89
- lab = _mws_core(H * W, u[order], v[order], rep[order]).reshape(H, W)
90
- _, lab = np.unique(lab, return_inverse=True); lab = lab.reshape(H, W) + 1
91
- sizes = np.bincount(lab.ravel()); tiny = sizes < min_size; tiny[0] = False
92
- if tiny[lab].any():
93
- _, (iy, ix) = ndi.distance_transform_edt(tiny[lab], return_indices=True)
94
- lab = lab[iy, ix]; _, lab = np.unique(lab, return_inverse=True); lab = lab.reshape(H, W) + 1
95
- return lab.astype(np.int32)
96
-
97
-
98
- # ------------------------------------------------------------------ modèle
99
- def rgb2lab(x):
100
- """x : B x 3 x H x W dans [0,1] -> Lab normalisé (L/50-1, a/64, b/64)."""
101
- lin = torch.where(x <= 0.04045, x / 12.92, ((x + 0.055) / 1.055) ** 2.4)
102
- M = torch.tensor([[0.4124, 0.3576, 0.1805], [0.2126, 0.7152, 0.0722], [0.0193, 0.1192, 0.9505]], device=x.device)
103
- xyz = torch.einsum("ij,bjhw->bihw", M, lin) / torch.tensor([0.95047, 1.0, 1.08883], device=x.device).view(1, 3, 1, 1)
104
- f = torch.where(xyz > 0.008856, xyz.clamp(min=1e-6) ** (1 / 3), 7.787 * xyz + 16 / 116)
105
- L, a, b = 116 * f[:, 1] - 16, 500 * (f[:, 0] - f[:, 1]), 200 * (f[:, 1] - f[:, 2])
106
- return torch.stack([L / 50 - 1, a / 64, b / 64], 1)
107
-
108
-
109
- class GeoCartoNet(nn.Module):
110
- """Entrée : image uint8 B x 3 x H x W (H, W multiples de 32). Sortie : K affinités + 1 carte de failles (logits)."""
111
-
112
- def __init__(self, cfg, pretrained=True):
113
- super().__init__()
114
- self.cfg = cfg
115
- self.register_buffer("mean", torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1))
116
- self.register_buffer("std", torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1))
117
- Arch = {"unet": smp.Unet, "unetpp": smp.UnetPlusPlus}[cfg["arch"]]
118
- self.net = Arch(cfg["encoder"], encoder_weights="imagenet" if pretrained else None, in_channels=6,
119
- classes=len(cfg["offsets"]) + 1)
120
-
121
- def forward(self, x8):
122
- with torch.autocast(x8.device.type, enabled=False): # conversion de couleur en précision complète
123
- x = x8.float() / 255
124
- x = torch.cat([(x - self.mean) / self.std, rgb2lab(x)], 1)
125
- return self.net(x)
126
-
127
-
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
leader.py DELETED
@@ -1,79 +0,0 @@
1
- # Détection des traits de rappel : trait sombre fin qui part du bord d'une étiquette, dans n'importe quelle direction.
2
- import numpy as np
3
- from scipy import ndimage as ndi
4
- from skimage.segmentation import find_boundaries
5
-
6
-
7
- def prepare(reg):
8
- """Masque des pixels 'trait candidat' : sombres, hors contacts, hors failles. Calculé une fois par zone."""
9
- if "lead_mask" in reg:
10
- return
11
- lum, inst = reg["lum"], reg["inst"]
12
- bd = find_boundaries(inst, mode="thick")
13
- nearbd = ndi.binary_dilation(bd, iterations=1)
14
- flt = reg.get("flt")
15
- nf = ndi.binary_dilation(flt > 0, iterations=3) if flt is not None else np.zeros_like(bd)
16
- rgb = reg["rgb"].astype(np.int16)
17
- sat = rgb.max(-1) - rgb.min(-1)
18
- dark = (lum < 100) & (sat < 70) # trait noir/gris (pas de texte ou de contour coloré)
19
- reg["lead_dark"] = dark
20
- reg["lead_mask"] = dark & ~nearbd & ~nf
21
-
22
-
23
- def trace(reg, wd, max_len=160, gap=6):
24
- """Renvoie la liste des extrémités (y, x, longueur) des traits partant de la boîte du mot."""
25
- prepare(reg)
26
- M, D, inst = reg["lead_mask"], reg["lead_dark"], reg["inst"]
27
- N = inst.shape[0]
28
- x0, y0, x1, y1 = int(max(wd.x0, 0)), int(max(wd.y0, 0)), int(min(wd.x1, N - 1)), int(min(wd.y1, N - 1))
29
- P = 14
30
- Y0, X0, Y1, X1 = max(y0 - P, 0), max(x0 - P, 0), min(y1 + P, N - 1), min(x1 + P, N - 1)
31
- sub = M[Y0:Y1 + 1, X0:X1 + 1].copy()
32
- # on enlève la boîte de texte (+1 px) : l'encre des lettres n'est pas un trait
33
- sub[max(y0 - 1 - Y0, 0):y1 + 2 - Y0, max(x0 - 1 - X0, 0):x1 + 2 - X0] = False
34
- lab, n = ndi.label(sub, structure=np.ones((3, 3)))
35
- if n == 0:
36
- return []
37
- yy, xx = np.nonzero(lab)
38
- ll = lab[yy, xx]
39
- gy, gx = yy + Y0, xx + X0
40
- # distance à la boîte
41
- db = np.hypot(np.maximum(np.maximum(y0 - gy, gy - y1), 0), np.maximum(np.maximum(x0 - gx, gx - x1), 0))
42
- out = []
43
- for c in range(1, n + 1):
44
- s = ll == c
45
- if s.sum() < 5 or db[s].min() > 2.5: # doit toucher (presque) la boîte
46
- continue
47
- py, px, d = gy[s].astype(float), gx[s].astype(float), db[s]
48
- # un trait est fin et allongé
49
- cov = np.cov(np.stack([py, px]))
50
- ev, evec = np.linalg.eigh(cov)
51
- v = evec[:, 1]
52
- if ev[1] < 9: # trop court (< ~10 px)
53
- continue
54
- res = np.abs((py - py.mean()) * v[1] - (px - px.mean()) * v[0])
55
- if np.percentile(res, 90) > 1.3: # pas droit ou trop épais
56
- continue
57
- a = np.argmin(d); b = np.argmax(d)
58
- if d[b] - d[a] < 4: # doit s'éloigner de la boîte
59
- continue
60
- if np.dot(v, [py[b] - py[a], px[b] - px[a]]) < 0:
61
- v = -v
62
- # suivi dans la direction, au-delà du morceau visible, en tolérant les coupures aux contacts
63
- y, x = py[b], px[b]; miss = 0; L = float(np.hypot(py[b] - py[a], px[b] - px[a]))
64
- ey, ex = y, x
65
- while L < max_len:
66
- y += v[0]; x += v[1]; L += 1
67
- iy, ix = int(round(y)), int(round(x))
68
- if not (0 <= iy < N and 0 <= ix < N):
69
- break
70
- if D[max(iy - 1, 0):iy + 2, max(ix - 1, 0):ix + 2].any():
71
- miss = 0; ey, ex = y, x
72
- else:
73
- miss += 1
74
- if miss > gap:
75
- break
76
- # pointe : on avance de 2 px au-delà de la fin (le trait se termine DANS le polygone visé)
77
- ty, tx = int(np.clip(round(ey + 2 * v[0]), 0, N - 1)), int(np.clip(round(ex + 2 * v[1]), 0, N - 1))
78
- out.append((ty, tx, float(np.hypot(ty - py[a], tx - px[a]))))
79
- return out
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
legend_ranker.pkl CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:0b99fd9736d2bd63a138a9dd12c7edfe2d46128e9b7454d6c1aeea7166be7a15
3
- size 550205
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:34cf13b45af05cc52e897f5bc5203f181c430e3b59306e4d02e2f59320acb069
3
+ size 550493
naming_model.py DELETED
@@ -1,203 +0,0 @@
1
- # =====================================================================================
2
- # MODULE DE NOMMAGE : à quel polygone appartient chaque code lu sur la carte ?
3
- # Module séparé du modèle de segmentation. Pour chaque code lu (OCR), on note chaque polygone
4
- # voisin avec des indices simples (distance, trait de rappel sombre, place disponible pour le
5
- # texte, recouvrement...) et un petit modèle appris sur de VRAIES étiquettes GMNA choisit.
6
- # Apprentissage : zones "south" ; test : zones "north" (jamais vues).
7
- # =====================================================================================
8
- import sys, glob, pickle, re, difflib, time
9
- import numpy as np, pandas as pd
10
- from scipy import ndimage as ndi
11
- from sklearn.ensemble import HistGradientBoostingClassifier
12
-
13
- RADIUS = 110 # rayon de recherche des polygones candidats autour d'une étiquette (px)
14
- norm = lambda c: re.sub(r"[\*\?\s]", "", str(c))
15
- import os
16
- if os.path.exists("naming_cache/vocab.pkl"):
17
- _codes = pickle.load(open("naming_cache/vocab.pkl", "rb"))
18
- else: # lecture encore en cours : vocabulaire = codes des zones déjà lues
19
- _codes = [c for f in glob.glob("naming_cache/*_*.pkl") for c in pickle.load(open(f, "rb"))["codes"]]
20
- VOCAB = sorted({norm(c) for c in _codes} - {""})
21
- VSET = set(VOCAB)
22
-
23
-
24
- def to_code(txt):
25
- t = re.sub(r"[^A-Za-z0-9€Є³]", "", txt).replace("€", "Є")
26
- if not t:
27
- return None, 0.0
28
- if t in VSET:
29
- return t, 1.0
30
- m = difflib.get_close_matches(t, VOCAB, n=1, cutoff=0.75)
31
- return (m[0], 0.75) if m else (None, 0.0)
32
-
33
-
34
- FEATS = ["dist_centre", "dist_boite", "sous_centre", "part_boite", "rang_dist", "n_cand", "aire_log", "rayon_int",
35
- "rayon_sur_texte", "trait_sombre", "trait_lum", "limites_croisees", "long_trait", "h_texte", "l_texte",
36
- "conf", "qualite_code", "dist_rel", "plus_proche_non_sous", "part_boite_max_autre", "vise_trait", "trait_L"]
37
-
38
-
39
- def candidates(reg, wd):
40
- """Lignes de caractéristiques (une par polygone candidat) pour un mot lu."""
41
- inst, lum = reg["inst"], reg["lum"]
42
- N = inst.shape[0]
43
- x0, y0, x1, y1 = int(max(wd.x0, 0)), int(max(wd.y0, 0)), int(min(wd.x1, N - 1)), int(min(wd.y1, N - 1))
44
- if x1 - x0 < 2 or y1 - y0 < 2:
45
- return []
46
- cx, cy = (x0 + x1) / 2, (y0 + y1) / 2
47
- X0, Y0, X1, Y1 = max(x0 - RADIUS, 0), max(y0 - RADIUS, 0), min(x1 + RADIUS, N - 1), min(y1 + RADIUS, N - 1)
48
- sub, slum = inst[Y0:Y1 + 1, X0:X1 + 1], lum[Y0:Y1 + 1, X0:X1 + 1]
49
- yy, xx = np.mgrid[Y0:Y1 + 1, X0:X1 + 1]
50
- dc = np.hypot(yy - cy, xx - cx) # distance au centre de la boîte
51
- db = np.hypot(np.maximum(np.maximum(y0 - yy, yy - y1), 0), np.maximum(np.maximum(x0 - xx, xx - x1), 0)) # à la boîte
52
- ids = np.unique(sub); ids = ids[ids > 0]
53
- if not len(ids):
54
- return []
55
- box = sub[y0 - Y0:y1 - Y0 + 1, x0 - X0:x1 - X0 + 1]
56
- light = slum[y0 - Y0:y1 - Y0 + 1, x0 - X0:x1 - X0 + 1] >= 120 # hors encre du texte
57
- nb = max(light.sum(), 1)
58
- under = inst[int(cy), int(cx)]
59
- th, tw = y1 - y0, x1 - x0
60
- dmin_c = ndi.minimum(dc, sub, ids); dmin_b = ndi.minimum(db, sub, ids)
61
- pos_b = ndi.minimum_position(db, sub, ids)
62
- edt = ndi.distance_transform_edt(sub > 0) if False else None
63
- from leader import trace
64
- tips = trace(reg, wd)
65
- rows = []
66
- order = np.argsort(dmin_b)
67
- rank = np.empty(len(ids)); rank[order] = np.arange(len(ids))
68
- part = np.array([(box[light] == i).sum() / nb for i in ids])
69
- for j, i in enumerate(ids):
70
- m = sub == i
71
- # rayon intérieur du polygone (dans la fenêtre) : peut-il contenir le texte ?
72
- rin = ndi.distance_transform_edt(np.pad(m, 1))[1:-1, 1:-1].max()
73
- # segment droit boîte -> point le plus proche du polygone : un trait de rappel est sombre et droit
74
- py, px = pos_b[j][0] + Y0, pos_b[j][1] + X0
75
- qy, qx = min(max(py, y0), y1), min(max(px, x0), x1)
76
- L = float(np.hypot(py - qy, px - qx))
77
- if L >= 3:
78
- n = int(L) + 1
79
- sy = np.clip(np.round(np.linspace(qy, py, n)).astype(int), 0, N - 1)
80
- sx = np.clip(np.round(np.linspace(qx, px, n)).astype(int), 0, N - 1)
81
- ls = np.minimum.reduce([lum[np.clip(sy + a, 0, N - 1), np.clip(sx + b, 0, N - 1)] for a in (-1, 0, 1) for b in (-1, 0, 1)])
82
- dark, lmean = float((ls[1:-1] < 120).mean()) if n > 2 else 0.0, float(ls.mean())
83
- cross = int((np.diff(inst[sy, sx]) != 0).sum())
84
- else:
85
- dark, lmean, cross = 0.0, 255.0, 0
86
- others = np.delete(part, j)
87
- tl = [t[2] for t in tips if inst[t[0], t[1]] == i]
88
- rows.append(dict(poly=int(i), dist_centre=float(dmin_c[j]), dist_boite=float(dmin_b[j]), sous_centre=float(i == under),
89
- part_boite=float(part[j]), rang_dist=float(rank[j]), n_cand=float(len(ids)),
90
- aire_log=float(np.log1p(reg["area"][i])), rayon_int=float(rin), rayon_sur_texte=float(rin / max(th / 2, 1)),
91
- trait_sombre=dark, trait_lum=lmean, limites_croisees=float(cross), long_trait=L, h_texte=float(th),
92
- l_texte=float(tw), conf=float(wd.conf), qualite_code=float(wd.q),
93
- dist_rel=float(dmin_b[j] / max(th, 1)), plus_proche_non_sous=float(rank[j] == (1 if under in ids else 0) and i != under),
94
- part_boite_max_autre=float(others.max() if len(others) else 0),
95
- vise_trait=float(len(tl)), trait_L=float(min(tl)) if tl else -1.0))
96
- return rows
97
-
98
-
99
- def load_region(f):
100
- r = pickle.load(open(f, "rb"))
101
- r["lum"] = (r["rgb"].astype(np.float32) @ np.array([0.299, 0.587, 0.114], np.float32))
102
- r["area"] = np.bincount(r["inst"].ravel())
103
- r["true"] = {i + 1: norm(c) for i, c in enumerate(r["codes"])}
104
- return r
105
-
106
-
107
- def build(files):
108
- out = []
109
- for f in files:
110
- reg = load_region(f)
111
- W_ = reg["words"]
112
- for k, wd in enumerate(W_.itertuples()):
113
- code, q = to_code(wd.text)
114
- if code is None:
115
- continue
116
- wd = wd._replace(conf=wd.conf) if False else wd
117
- class _W: pass
118
- w = _W(); w.__dict__.update(wd._asdict()); w.q = q
119
- for row in candidates(reg, w):
120
- row.update(region=f, word=k, code=code, y=float(reg["true"][row["poly"]] == code),
121
- vrai=reg["true"][row["poly"]], aire=float(reg["area"][row["poly"]]))
122
- out.append(row)
123
- return pd.DataFrame(out)
124
-
125
-
126
- def assign(D, score):
127
- """Choix final : pour chaque mot le polygone le mieux noté ; un polygone ne garde qu'UN code
128
- (celui qui a la meilleure note), les mots perdants passent à leur candidat suivant."""
129
- D = D.assign(s=score)
130
- choice = {}
131
- words = {k: g.sort_values("s", ascending=False) for k, g in D.groupby(["region", "word"])}
132
- ptr = {k: 0 for k in words}
133
- for _ in range(4):
134
- for k, g in words.items():
135
- if ptr[k] < len(g):
136
- r = g.iloc[ptr[k]]
137
- choice[k] = (r.region, int(r.poly), r.code, float(r.s))
138
- else:
139
- choice.pop(k, None)
140
- best = {}
141
- for k, (reg, p, c, s) in choice.items():
142
- key = (reg, p)
143
- if key not in best or s > best[key][1]:
144
- best[key] = (c, s)
145
- moved = False
146
- for k, (reg, p, c, s) in list(choice.items()):
147
- if best[(reg, p)][0] != c and ptr[k] + 1 < len(words[k]) and words[k].iloc[ptr[k] + 1].s > 0.15:
148
- ptr[k] += 1; moved = True
149
- if not moved:
150
- break
151
- return choice
152
-
153
-
154
- def report(D, choice, title):
155
- key = D.set_index(["region", "word", "poly"])
156
- evaluable = D.groupby(["region", "word"]).y.max() > 0 # le bon code existe parmi les candidats (lecture juste)
157
- ok = n = 0
158
- polys = {}
159
- for k, (reg, p, c, s) in choice.items():
160
- r = key.loc[(k[0], k[1], p)]
161
- good = bool(np.atleast_1d(r.y)[0])
162
- if evaluable[k]:
163
- n += 1; ok += good
164
- cur = polys.get((reg, p))
165
- if cur is None or s > cur[1]:
166
- polys[(reg, p)] = (good, s, float(np.atleast_1d(r.aire)[0]))
167
- pg = np.array([v[0] for v in polys.values()]); pa = np.array([v[2] for v in polys.values()])
168
- print(f"{title:<46} codes bien rattachés : {ok / max(n, 1):5.1%} ({n} codes) | polygones nommés : {len(polys)}, "
169
- f"noms justes : {pg.mean():5.1%} (en surface : {(pa * pg).sum() / pa.sum():5.1%})")
170
- return ok / max(n, 1), polys
171
-
172
-
173
- if __name__ == "__main__":
174
- t0 = time.time()
175
- tr_files, te_files = sorted(glob.glob("naming_cache/south_*.pkl")), sorted(glob.glob("naming_cache/north_*.pkl"))
176
- if len(te_files) < 3: # provisoire : pas encore de zones nord -> 1 zone sud sur 3 en test
177
- allf = tr_files + te_files
178
- tr_files, te_files = [f for i, f in enumerate(allf) if i % 3], [f for i, f in enumerate(allf) if i % 3 == 0]
179
- Dtr, Dte = build(tr_files), build(te_files)
180
- print(f"apprentissage : {len(tr_files)} zones, {Dtr.groupby(['region', 'word']).ngroups} codes lus, {len(Dtr)} paires | "
181
- f"test : {len(te_files)} zones, {Dte.groupby(['region', 'word']).ngroups} codes lus ({time.time() - t0:.0f} s)")
182
- ev = Dtr.groupby(["region", "word"]).y.transform("max") > 0 # on n'apprend que sur les mots correctement lus
183
- clf = HistGradientBoostingClassifier(max_iter=300, learning_rate=0.06, max_leaf_nodes=24, l2_regularization=1.0,
184
- class_weight="balanced", random_state=0)
185
- clf.fit(Dtr.loc[ev, FEATS], Dtr.loc[ev, "y"])
186
- pickle.dump({"model": clf, "features": FEATS, "radius": RADIUS}, open("naming_model.pkl", "wb"))
187
- print("\nSUR LES ZONES DE TEST (north, jamais vues) :")
188
- naive = Dte.sous_centre + 0.001 * Dte.part_boite
189
- a0, _ = report(Dte, {k: (g.region.iloc[0], int(g.poly.iloc[np.argmax(n_)]), g.code.iloc[0], 1.0)
190
- for (k, g), n_ in ((kg, naive.loc[kg[1].index].values) for kg in Dte.groupby(["region", "word"]))},
191
- "Règle naïve (polygone sous l'étiquette)")
192
- p = clf.predict_proba(Dte[FEATS])[:, 1]
193
- a1, _ = report(Dte, {k: (g.region.iloc[0], int(g.poly.iloc[np.argmax(p[Dte.index.get_indexer(g.index)])]), g.code.iloc[0],
194
- float(p[Dte.index.get_indexer(g.index)].max())) for k, g in Dte.groupby(["region", "word"])},
195
- "Module appris (meilleure note)")
196
- ch = assign(Dte, p)
197
- a2, polys = report(Dte, ch, "Module appris + un seul code par polygone")
198
- # avec seuil de confiance : on ne nomme que si la note est élevée
199
- for thr in (0.5, 0.7, 0.85):
200
- report(Dte, {k: v for k, v in ch.items() if v[3] >= thr}, f" ... en ne gardant que les notes >= {thr}")
201
- from sklearn.inspection import permutation_importance
202
- imp = permutation_importance(clf, Dte[FEATS], Dte["y"], n_repeats=3, random_state=0, scoring="average_precision")
203
- print("\nIndices les plus utiles :", ", ".join(f"{FEATS[i]} ({imp.importances_mean[i]:.3f})" for i in np.argsort(-imp.importances_mean)[:8]))
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
pipeline.py DELETED
@@ -1,288 +0,0 @@
1
- # =====================================================================================
2
- # GEOCARTO – PIPELINE COMPLET : raster de carte géologique -> GeoPackage QGIS / ArcGIS
3
- # 1. modèle (affinités + failles) par fenêtres glissantes 2. mutex watershed -> polygones
4
- # 3. vectorisation (unités + failles) 4. OCR des textes de la carte
5
- # 5. nommage : code écrit (module appris) > case de légende (si sûr) > "Unité non identifiée" + 3 suggestions
6
- # 6. export GeoPackage (couches unites_geologiques, failles, table legende + styles QGIS par défaut)
7
- # =====================================================================================
8
- import os, sys, re, sqlite3, pickle, time
9
- import numpy as np, pandas as pd, torch, geopandas as gpd
10
- from scipy import ndimage as ndi
11
- from rasterio.features import shapes
12
- from rasterio.transform import Affine
13
- from shapely.geometry import shape, LineString
14
- from skimage.color import rgb2lab as sk_rgb2lab
15
- from skimage.morphology import skeletonize
16
-
17
- HERE = os.path.dirname(os.path.abspath(__file__))
18
- sys.path.insert(0, HERE)
19
- import gc_model as M
20
-
21
-
22
- # ------------------------------------------------------------------ 1. modèle
23
- def load_model(path):
24
- ck = torch.load(path, map_location="cpu", weights_only=True)
25
- cfg = ck["cfg"]
26
- M.CFG.update(cfg); M.OFFSETS[:] = cfg["offsets"]
27
- model = M.GeoCartoNet(cfg, pretrained=False)
28
- model.load_state_dict({k: v.float() for k, v in ck["state_dict"].items()})
29
- return model.eval()
30
-
31
-
32
- @torch.no_grad()
33
- def _run(model, im):
34
- x = torch.from_numpy(np.ascontiguousarray(im).transpose(2, 0, 1).copy())[None]
35
- return torch.sigmoid(model(x).float())[0].numpy()
36
-
37
-
38
- def _tta(model, im, offsets):
39
- """Moyenne sur 4 miroirs ; chaque décalage est ramené dans le repère d'origine (testé : exact)."""
40
- H, W = im.shape[:2]
41
- Y, X = np.mgrid[:H, :W]
42
- idx = {tuple(o): k for k, o in enumerate(offsets)}
43
- K = len(offsets)
44
- acc = 0
45
- for fy in (0, 1):
46
- for fx in (0, 1):
47
- v = im[::-1] if fy else im
48
- v = v[:, ::-1] if fx else v
49
- o = _run(model, v)
50
- A, pf = o[:K], o[K]
51
- py, px = (H - 1 - Y) if fy else Y, (W - 1 - X) if fx else X
52
- out = np.empty_like(o)
53
- for k, (dy, dx) in enumerate(offsets):
54
- t = (-dy if fy else dy, -dx if fx else dx)
55
- if t in idx:
56
- out[k] = A[idx[t]][py, px]
57
- else:
58
- out[k] = A[idx[(-t[0], -t[1])]][np.clip(py + t[0], 0, H - 1), np.clip(px + t[1], 0, W - 1)]
59
- out[K] = pf[py, px]
60
- acc = acc + out
61
- return acc / 4
62
-
63
-
64
- def infer(model, rgb, tile=512, stride=384, tta=False):
65
- """Fenêtres glissantes avec pondération de Hann -> affinités (K x H x W) + probabilité de faille."""
66
- H, W = rgb.shape[:2]
67
- ph, pw = max(tile - H, 0), max(tile - W, 0)
68
- img = np.pad(rgb, ((0, ph), (0, pw), (0, 0)), mode="edge")
69
- Hp, Wp = img.shape[:2]
70
- K = len(M.OFFSETS)
71
- acc = np.zeros((K + 1, Hp, Wp), np.float32); ws = np.zeros((Hp, Wp), np.float32)
72
- hw = np.maximum(np.outer(np.hanning(tile), np.hanning(tile)), 1e-3).astype(np.float32)
73
- pos = lambda n: sorted(set(list(range(0, n - tile + 1, stride)) + [n - tile]))
74
- for y in pos(Hp):
75
- for x in pos(Wp):
76
- im = img[y:y + tile, x:x + tile]
77
- o = _tta(model, im, M.OFFSETS) if tta else _run(model, im)
78
- acc[:, y:y + tile, x:x + tile] += o * hw; ws[y:y + tile, x:x + tile] += hw
79
- o = acc / ws
80
- return o[:K, :H, :W], o[K, :H, :W]
81
-
82
-
83
- # ------------------------------------------------------------------ 3. vectorisation
84
- def vectorize_units(lab, transform, crs):
85
- geoms, ids = [], []
86
- for g, v in shapes(lab.astype(np.int32), mask=lab > 0, transform=transform):
87
- geoms.append(shape(g)); ids.append(int(v))
88
- P = gpd.GeoDataFrame({"pid": ids}, geometry=geoms, crs=crs).dissolve("pid").reset_index()
89
- P["geometry"] = P.geometry.simplify(abs(transform.a) * 0.75, preserve_topology=True)
90
- return P
91
-
92
-
93
- def vectorize_faults(pf, transform, crs, thr=0.5, min_px=15):
94
- sk = skeletonize(pf > thr)
95
- lab, n = ndi.label(sk, structure=np.ones((3, 3)))
96
- sizes = np.bincount(lab.ravel())
97
- nb = ndi.convolve(sk.astype(np.uint8), np.ones((3, 3), np.uint8), mode="constant") - 1
98
- lines = []
99
- seen = np.zeros_like(sk)
100
- for c in range(1, n + 1):
101
- if sizes[c] < min_px:
102
- continue
103
- ys, xs = np.nonzero(lab == c)
104
- pts = set(zip(ys.tolist(), xs.tolist()))
105
- ends = [p for p in pts if nb[p] == 1] or [next(iter(pts))]
106
- for st in ends:
107
- if seen[st]:
108
- continue
109
- path, cur = [st], st
110
- seen[st] = True
111
- while True:
112
- nxt = [(cur[0] + a, cur[1] + b) for a in (-1, 0, 1) for b in (-1, 0, 1) if (a or b)
113
- and (cur[0] + a, cur[1] + b) in pts and not seen[cur[0] + a, cur[1] + b]]
114
- if not nxt:
115
- break
116
- cur = nxt[0]; seen[cur] = True; path.append(cur)
117
- if len(path) >= min_px:
118
- lines.append(LineString([transform * (x + 0.5, y + 0.5) for y, x in path]).simplify(abs(transform.a)))
119
- F = gpd.GeoDataFrame({"type": ["indéterminée"] * len(lines)}, geometry=lines, crs=crs)
120
- F["longueur_km"] = (F.length / 1000).round(2)
121
- return F
122
-
123
-
124
- # ------------------------------------------------------------------ 4. OCR
125
- def read_text(rgb, reader, B=1024, OV=96):
126
- words = []
127
- H, W = rgb.shape[:2]
128
- for y in range(0, H, B - OV):
129
- for x in range(0, W, B - OV):
130
- for box, txt, conf in reader.readtext(rgb[y:y + B, x:x + B], mag_ratio=2.0, text_threshold=0.6, low_text=0.3):
131
- b = np.array(box) + [x, y]; cx, cy = b.mean(0)
132
- if (x > 0 and cx < x + OV / 2) or (y > 0 and cy < y + OV / 2):
133
- continue
134
- words.append(dict(text=txt, conf=conf, cx=cx, cy=cy, x0=b[:, 0].min(), x1=b[:, 0].max(), y0=b[:, 1].min(), y1=b[:, 1].max()))
135
- return pd.DataFrame(words, columns=["text", "conf", "cx", "cy", "x0", "x1", "y0", "y1"])
136
-
137
-
138
- # ------------------------------------------------------------------ 5. nommage
139
- def swatch_features(lab_pixels):
140
- """Signature d'une case de légende / d'un polygone : couleur dominante, couleurs claire et foncée du motif, texture."""
141
- px = lab_pixels[lab_pixels[:, 0] > 45]
142
- if len(px) < 20:
143
- px = lab_pixels
144
- L = px[:, 0]; med = np.percentile(L, 50)
145
- return np.concatenate([np.median(px, 0), np.median(px[L >= med], 0), np.median(px[L <= med], 0), px.std(0) / 4])
146
-
147
-
148
- def ocr_candidates(rgb, lab, words, vocab, naming_pkl, flt=None):
149
- """Toutes les paires (code lu, polygone candidat) avec leur score appris ; et l'affectation un code -> un polygone."""
150
- import naming_model as NM
151
- NM.VOCAB = sorted(vocab); NM.VSET = set(NM.VOCAB)
152
- mdl = pickle.load(open(naming_pkl, "rb"))
153
- reg = dict(inst=lab, rgb=rgb, lum=rgb.astype(np.float32) @ np.array([0.299, 0.587, 0.114], np.float32),
154
- area=np.bincount(lab.ravel()), flt=flt)
155
- rows = []
156
- for k, wd in enumerate(words.itertuples()):
157
- code, q = NM.to_code(wd.text)
158
- if code is None:
159
- continue
160
- class _W: pass
161
- w = _W(); w.__dict__.update(wd._asdict()); w.q = q
162
- for r in NM.candidates(reg, w):
163
- r.update(region="carte", word=k, code=code, lu=wd.text); rows.append(r)
164
- if not rows:
165
- return pd.DataFrame(columns=["word", "code", "s"]), {}
166
- D = pd.DataFrame(rows)
167
- D["s"] = mdl["model"].predict_proba(D[mdl["features"]])[:, 1]
168
- return D, NM.assign(D, D["s"].values)
169
-
170
- RANK_FEATS = ["d_norm", "d_rel", "rang", "gap12", "d_ratio", "ocr", "ocr_assigne", "ocr_rel", "lu", "n_lu", "dist_lu", "log_aire"]
171
-
172
- def rank_pairs(pids, feats, areas, cents, codes, Fm, D, ch, words, K=12):
173
- """Paires (polygone, code candidat) et leurs indices. Candidats = K cases de légende les plus proches + codes lus par l'OCR.
174
- Indices : uniquement le texte lu sur la carte et la légende (pas de propagation d'apparence entre polygones)."""
175
- cidx = {c: i for i, c in enumerate(codes)}
176
- d = np.linalg.norm(feats[:, None, :] - Fm[None], axis=2)
177
- E, lu_pos, asg = {}, {}, {}
178
- if len(D):
179
- for (p, c), s in D.groupby(["poly", "code"]).s.max().items():
180
- E[(p, c)] = s
181
- for w, c in D[["word", "code"]].drop_duplicates().values:
182
- lu_pos.setdefault(c, []).append((words.iloc[w].cy, words.iloc[w].cx))
183
- for (_, w), (_, p, c, s) in ch.items():
184
- asg[(p, c)] = max(asg.get((p, c), 0), s)
185
- Ep = {}
186
- for (p, c), s in E.items():
187
- Ep.setdefault(p, []).append(c)
188
- lu_pos = {c: np.array(v) for c, v in lu_pos.items()}
189
- rows = []
190
- for i, p in enumerate(pids):
191
- o = np.argsort(d[i]); rank = np.empty(len(codes), int); rank[o] = np.arange(len(codes))
192
- cand = list(dict.fromkeys(list(codes[o[:K]]) + [c for c in Ep.get(p, []) if c in cidx]))
193
- emax = max([E[(p, c)] for c in Ep.get(p, [])] + [0]); gap = d[i, o[1]] - d[i, o[0]] if len(o) > 1 else 0
194
- for c in cand:
195
- j = cidx[c]; pos = lu_pos.get(c)
196
- rows.append(dict(poly=p, code=c, d=d[i, j], d_rel=d[i, j] - d[i, o[0]], rang=rank[j], gap12=gap,
197
- ocr=E.get((p, c), 0), ocr_assigne=asg.get((p, c), 0), ocr_max_poly=emax,
198
- lu=int(pos is not None), n_lu=0 if pos is None else len(pos),
199
- dist_lu=5000 if pos is None else np.hypot(pos[:, 0] - cents[i][0], pos[:, 1] - cents[i][1]).min(),
200
- log_aire=np.log(areas[i])))
201
- X = pd.DataFrame(rows)
202
- X["d_ratio"] = X.d_rel / (X.gap12 + 1); X["d_norm"] = X.d / X.d.median(); X["ocr_rel"] = X.ocr - X.ocr_max_poly
203
- return X
204
-
205
- def name_polygons(rgb, lab, words, legend, naming_pkl, flt=None, ranker_pkl="legend_ranker.pkl", sure=0.8, check=0.5):
206
- """legend : DataFrame index=code, colonnes nom, age, lithologie, couleur_hex, feat (signature de la case).
207
- Retourne par polygone : code, source_nom, confiance, texte_lu, suggestions.
208
- source_nom : 'code lu sur la carte' / 'légende' (confiance >= sure), '… – à vérifier' (check <= confiance < sure), 'non identifiée'."""
209
- D, ch = ocr_candidates(rgb, lab, words, legend.index, naming_pkl, flt)
210
- R = pickle.load(open(ranker_pkl, "rb"))
211
- labc = sk_rgb2lab(rgb).reshape(-1, 3)
212
- ids = np.unique(lab[lab > 0])
213
- order = np.argsort(lab.ravel(), kind="stable"); sl = lab.ravel()[order]
214
- starts = np.searchsorted(sl, ids); ends = np.searchsorted(sl, ids, side="right")
215
- feats = np.stack([swatch_features(labc[order[a:b]]) for a, b in zip(starts, ends)])
216
- cents = ndi.center_of_mass(np.ones_like(lab), lab, ids)
217
- codes = np.array(legend.index); Fm = np.stack(legend.feat.values)
218
- X = rank_pairs(ids, feats, ends - starts, cents, codes, Fm, D, ch, words, K=R.get("K", 12))
219
- X["p"] = R["model"].predict_proba(X[R["features"]])[:, 1]
220
- X = X.sort_values(["poly", "p"], ascending=[True, False])
221
- lu_txt = {}
222
- for (_, w), (_, p, c, s) in ch.items():
223
- lu_txt[(p, c)] = words.iloc[w].text
224
- out = {}
225
- for p, g in X.groupby("poly", sort=False):
226
- top = g.head(3); c, pr = top.code.iloc[0], float(top.p.iloc[0])
227
- sug = ", ".join(f"{a} ({b:.0%})" for a, b in zip(top.code, top.p))
228
- ocr = top.ocr.iloc[0] > 0
229
- src = "code lu sur la carte" if ocr else "couleur de la légende"
230
- if pr >= sure:
231
- out[p] = dict(code=c, source_nom=src, confiance=round(pr, 2), texte_lu=lu_txt.get((p, c), ""), suggestions=sug)
232
- elif pr >= check: # trop peu fiable sur cartes réelles (31-64 % juste) : pas de nom, mais suggestion classée en tête
233
- out[p] = dict(code="", source_nom="non identifiée – suggestion probable", confiance=round(pr, 2), texte_lu=lu_txt.get((p, c), ""), suggestions=sug)
234
- else:
235
- out[p] = dict(code="", source_nom="non identifiée", confiance=round(pr, 2), texte_lu="", suggestions=sug)
236
- return pd.DataFrame.from_dict(out, orient="index")
237
-
238
- # ------------------------------------------------------------------ 6. export
239
- QML_UNITS = open(os.path.join(HERE, "qml_units.xml")).read() if os.path.exists(os.path.join(HERE, "qml_units.xml")) else None
240
- QML_FAULTS = open(os.path.join(HERE, "qml_faults.xml")).read() if os.path.exists(os.path.join(HERE, "qml_faults.xml")) else None
241
-
242
-
243
- def export_gpkg(path, P, F, legend):
244
- if os.path.exists(path):
245
- os.remove(path)
246
- P.to_file(path, layer="unites_geologiques", driver="GPKG")
247
- if len(F):
248
- F.to_file(path, layer="failles", driver="GPKG")
249
- con = sqlite3.connect(path)
250
- legend.drop(columns=["feat"]).reset_index().rename(columns={"index": "code"}).to_sql("legende", con, index=False, if_exists="replace")
251
- con.execute("""CREATE TABLE IF NOT EXISTS layer_styles (id INTEGER PRIMARY KEY AUTOINCREMENT, f_table_catalog TEXT(256),
252
- f_table_schema TEXT(256), f_table_name TEXT(256), f_geometry_column TEXT(256), styleName TEXT(30), styleQML TEXT,
253
- styleSLD TEXT, useAsDefault BOOLEAN, description TEXT, owner TEXT(30), ui TEXT(30), update_time DATETIME DEFAULT CURRENT_TIMESTAMP)""")
254
- for tbl, qml in (("unites_geologiques", QML_UNITS), ("failles", QML_FAULTS)):
255
- if qml and (tbl != "failles" or len(F)):
256
- con.execute("INSERT INTO layer_styles (f_table_catalog,f_table_schema,f_table_name,f_geometry_column,styleName,styleQML,styleSLD,useAsDefault,description,owner,update_time) "
257
- "VALUES ('','',?,'geom',?,?,'',1,'style GeoCarto','',strftime('%Y-%m-%dT%H:%M:%fZ','now'))", (tbl, tbl, qml))
258
- con.execute("INSERT INTO gpkg_contents (table_name, data_type, identifier) SELECT 'layer_styles','attributes','layer_styles' "
259
- "WHERE NOT EXISTS (SELECT 1 FROM gpkg_contents WHERE table_name='layer_styles')")
260
- con.execute("INSERT INTO gpkg_contents (table_name, data_type, identifier) SELECT 'legende','attributes','legende' "
261
- "WHERE NOT EXISTS (SELECT 1 FROM gpkg_contents WHERE table_name='legende')")
262
- con.commit(); con.close()
263
-
264
-
265
- def run(rgb, transform, crs, model, legend, reader, naming_pkl, out_gpkg, tta=False, log=print, ranker_pkl="legend_ranker.pkl"):
266
- t = time.time()
267
- aff, pf = infer(model, rgb, tta=tta); log(f"modèle : {time.time() - t:.0f} s")
268
- lab = M.mutex_watershed(aff); log(f"segmentation : {lab.max()} polygones ({time.time() - t:.0f} s)")
269
- words = read_text(rgb, reader); log(f"OCR : {len(words)} textes ({time.time() - t:.0f} s)")
270
- N = name_polygons(rgb, lab, words, legend, naming_pkl, flt=(pf > 0.5).astype(np.uint8), ranker_pkl=ranker_pkl); log(f"nommage ({time.time() - t:.0f} s)")
271
- P = vectorize_units(lab, transform, crs)
272
- P = P.join(N, on="pid")
273
- P["code"] = P.code.fillna(""); P["source_nom"] = P.source_nom.fillna("non identifiée")
274
- lg = legend.reindex(P.code)
275
- P["nom"] = np.where(P.code != "", (P.code + " – " + lg.lithologie.fillna("").values).str.strip(" –"), "Unité non identifiée")
276
- P["age"] = lg.age.fillna("").values; P["lithologie"] = lg.lithologie.fillna("").values
277
- med = [ndi.median(rgb[..., c], lab, P.pid.values) for c in range(3)]
278
- P["couleur_carte"] = ["#%02x%02x%02x" % tuple(int(v) for v in x) for x in zip(*med)]
279
- P["couleur_hex"] = np.where(P.code != "", lg.couleur_hex.fillna("").values, P.couleur_carte)
280
- P["couleur_hex"] = np.where(P.couleur_hex == "", P.couleur_carte, P.couleur_hex)
281
- P["surface_km2"] = (P.area / 1e6).round(3)
282
- P["id"] = np.arange(1, len(P) + 1)
283
- P = P[["id", "code", "nom", "age", "lithologie", "source_nom", "confiance", "texte_lu", "suggestions",
284
- "couleur_hex", "couleur_carte", "surface_km2", "geometry"]]
285
- F = vectorize_faults(pf, transform, crs)
286
- export_gpkg(out_gpkg, P, F, legend.loc[legend.index.isin(P.code)])
287
- log(f"export : {out_gpkg} | {len(P)} unités, {len(F)} failles ({time.time() - t:.0f} s)")
288
- return P, F, lab, pf
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
prop_model.pkl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ca118ed5248650dd98bc93403c6d57c84806d0f8bb7c0cdd9cf3b51f580df5f6
3
+ size 554843
qgis_test.py DELETED
@@ -1,55 +0,0 @@
1
- # Vrai test QGIS (PyQGIS 3.34, sans interface) d'un GeoPackage produit par pipeline.py
2
- # QT_QPA_PLATFORM=offscreen python3.12 qgis_test.py <carte.gpkg> <rendu.png>
3
- import sys
4
- from qgis.core import (QgsApplication, QgsVectorLayer, QgsProject, QgsMapSettings, QgsMapRendererParallelJob,
5
- QgsGeometry, QgsWkbTypes)
6
- from qgis.PyQt.QtCore import QSize
7
- from qgis.PyQt.QtGui import QColor
8
-
9
- gpkg, png = sys.argv[1], sys.argv[2]
10
- app = QgsApplication([], False); app.initQgis()
11
- ok = True
12
- def check(cond, msg):
13
- global ok
14
- print(("OK " if cond else "ÉCHEC"), msg); ok &= bool(cond)
15
-
16
- layers = []
17
- for name, geom in (("unites_geologiques", QgsWkbTypes.PolygonGeometry), ("failles", QgsWkbTypes.LineGeometry)):
18
- lyr = QgsVectorLayer(f"{gpkg}|layername={name}", name, "ogr")
19
- check(lyr.isValid(), f"couche {name} chargée")
20
- if not lyr.isValid():
21
- continue
22
- msg, styled = lyr.loadDefaultStyle() # style QML stocké dans la table layer_styles du GeoPackage
23
- check(styled, f"{name} : style intégré chargé automatiquement ({msg})")
24
- check(lyr.geometryType() == geom, f"{name} : type de géométrie {QgsWkbTypes.geometryDisplayString(lyr.geometryType())}")
25
- check(lyr.crs().isValid(), f"{name} : SCR {lyr.crs().authid() or lyr.crs().description()}")
26
- n = lyr.featureCount(); check(n > 0, f"{name} : {n} entités")
27
- bad = sum(1 for f in lyr.getFeatures() if not f.geometry().isGeosValid())
28
- check(bad == 0, f"{name} : géométries invalides = {bad}")
29
- print(" champs :", [f.name() for f in lyr.fields()])
30
- r = lyr.renderer()
31
- print(" rendu :", r.type(), (f"{len(r.categories())} catégories" if hasattr(r, "categories") else ""))
32
- if name == "unites_geologiques":
33
- need = {"code", "nom", "age", "lithologie", "source_nom", "confiance", "suggestions"}
34
- check(need <= {f.name() for f in lyr.fields()}, "colonnes de nommage présentes")
35
- # filtre attributaire comme le ferait un utilisateur QGIS
36
- lyr.setSubsetString("\"source_nom\" = 'code lu sur la carte'")
37
- print(" filtre source_nom='code lu sur la carte' :", lyr.featureCount(), "entités"); lyr.setSubsetString("")
38
- ex = next(lyr.getFeatures())
39
- print(" exemple :", {k: ex[k] for k in ("code", "nom", "age", "source_nom", "confiance")})
40
- layers.append(lyr)
41
-
42
- leg = QgsVectorLayer(f"{gpkg}|layername=legende", "legende", "ogr")
43
- check(leg.isValid() and leg.featureCount() > 0, f"table legende : {leg.featureCount() if leg.isValid() else 0} lignes")
44
-
45
- if layers:
46
- QgsProject.instance().addMapLayers(layers)
47
- ms = QgsMapSettings(); ms.setLayers(layers[::-1]); ms.setDestinationCrs(layers[0].crs())
48
- ext = layers[0].extent(); ms.setExtent(ext); ms.setOutputSize(QSize(1400, int(1400 * ext.height() / ext.width())))
49
- ms.setBackgroundColor(QColor("white"))
50
- job = QgsMapRendererParallelJob(ms); job.start(); job.waitForFinished()
51
- img = job.renderedImage(); check(img.save(png), f"rendu carte -> {png}")
52
- QgsProject.instance().write(png.rsplit(".", 1)[0] + ".qgz")
53
- print(" projet QGIS écrit :", png.rsplit(".", 1)[0] + ".qgz")
54
- print("\nRÉSULTAT QGIS :", "TOUT OK" if ok else "PROBLÈMES")
55
- app.exitQgis()
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
qml_faults.xml DELETED
@@ -1,17 +0,0 @@
1
- <!DOCTYPE qgis PUBLIC 'http://mrcc.com/qgis.dtd' 'SYSTEM'>
2
- <qgis version="3.34.0" styleCategories="Symbology">
3
- <renderer-v2 type="singleSymbol" symbollevels="0" enableorderby="0" forceraster="0">
4
- <symbols>
5
- <symbol type="line" name="0" alpha="1" clip_to_extent="1" force_rhr="0">
6
- <layer class="SimpleLine" enabled="1" locked="0" pass="0">
7
- <Option type="Map">
8
- <Option name="line_color" type="QString" value="0,0,0,255"/>
9
- <Option name="line_width" type="QString" value="0.6"/>
10
- <Option name="line_width_unit" type="QString" value="MM"/>
11
- <Option name="line_style" type="QString" value="solid"/>
12
- </Option>
13
- </layer>
14
- </symbol>
15
- </symbols>
16
- </renderer-v2>
17
- </qgis>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
qml_units.xml DELETED
@@ -1,41 +0,0 @@
1
- <!DOCTYPE qgis PUBLIC 'http://mrcc.com/qgis.dtd' 'SYSTEM'>
2
- <qgis version="3.34.0" styleCategories="Symbology|Labeling">
3
- <renderer-v2 type="singleSymbol" symbollevels="0" enableorderby="0" forceraster="0">
4
- <symbols>
5
- <symbol type="fill" name="0" alpha="1" clip_to_extent="1" force_rhr="0">
6
- <layer class="SimpleFill" enabled="1" locked="0" pass="0">
7
- <Option type="Map">
8
- <Option name="color" type="QString" value="200,200,200,255"/>
9
- <Option name="outline_color" type="QString" value="40,40,40,255"/>
10
- <Option name="outline_style" type="QString" value="solid"/>
11
- <Option name="outline_width" type="QString" value="0.15"/>
12
- <Option name="outline_width_unit" type="QString" value="MM"/>
13
- <Option name="style" type="QString" value="solid"/>
14
- </Option>
15
- <data_defined_properties>
16
- <Option type="Map">
17
- <Option name="name" type="QString" value=""/>
18
- <Option name="properties" type="Map">
19
- <Option name="fillColor" type="Map">
20
- <Option name="active" type="bool" value="true"/>
21
- <Option name="expression" type="QString" value="&quot;couleur_hex&quot;"/>
22
- <Option name="type" type="int" value="3"/>
23
- </Option>
24
- </Option>
25
- <Option name="type" type="QString" value="collection"/>
26
- </Option>
27
- </data_defined_properties>
28
- </layer>
29
- </symbol>
30
- </symbols>
31
- </renderer-v2>
32
- <labeling type="simple">
33
- <settings calloutType="simple">
34
- <text-style fieldName="code" isExpression="0" fontSize="8" fontSizeUnit="Point" fontFamily="Arial" textColor="20,20,20,255" fontWeight="50">
35
- <text-buffer bufferDraw="1" bufferSize="0.6" bufferSizeUnits="MM" bufferColor="255,255,255,200"/>
36
- </text-style>
37
- <placement placement="1" placementFlags="10" centroidInside="1" fitInPolygonOnly="1"/>
38
- <rendering scaleVisibility="0" obstacle="1" limitNumLabels="0" displayAll="0"/>
39
- </settings>
40
- </labeling>
41
- </qgis>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
requirements.txt DELETED
@@ -1,13 +0,0 @@
1
- torch>=2.2
2
- segmentation-models-pytorch==0.6.0
3
- timm==1.0.31
4
- numba>=0.60
5
- numpy
6
- scipy
7
- pandas
8
- scikit-learn==1.9.1
9
- scikit-image>=0.24
10
- rasterio>=1.4
11
- geopandas>=1.0
12
- shapely>=2.0
13
- easyocr==1.7.2
 
 
 
 
 
 
 
 
 
 
 
 
 
 
training/train_final.py DELETED
@@ -1,471 +0,0 @@
1
- # =====================================================================================
2
- # GEOCARTO – ENTRAÎNEMENT FINAL
3
- # Modèle retenu par le labo (rounds 1-3) : U-Net++ / EfficientNet-B4, entrée RGB+Lab,
4
- # augmentations de couleur, affinités (10 décalages) + failles, mutex watershed.
5
- # Données : dataset GeoCarto complet (6 600 tuiles, 4 continents), split spatial.
6
- # Sorties (/kaggle/working/geocarto_model) :
7
- # best.pt (poids + config), best_fp16.pt, history.csv, test_metrics.csv, metrics.json, figures
8
- # =====================================================================================
9
- import os, io, sys, json, time, copy
10
- from pathlib import Path
11
- import numpy as np, pandas as pd
12
- import pyarrow.parquet as pq
13
- from PIL import Image
14
- from scipy import ndimage as ndi
15
- from skimage.metrics import adapted_rand_error, variation_of_information
16
- import torch, torch.nn as nn, torch.nn.functional as F
17
- from torch.utils.data import Dataset, DataLoader
18
- import segmentation_models_pytorch as smp
19
- from numba import njit
20
-
21
- E = os.environ.get
22
- STEPS = int(E("GC_STEPS", 30000)) # pas d'entraînement
23
- BATCH, CROP, LR = int(E("GC_BATCH", 8)), int(E("GC_CROP", 448)), float(E("GC_LR", 3e-4))
24
- EVAL_EVERY = int(E("GC_EVAL_EVERY", 2500))
25
- N_VAL = int(E("GC_NVAL", 12)) # tuiles de validation par source (suivi pendant l'entraînement)
26
- P_REAL = float(E("GC_PREAL", 0.35)) # part de tuiles RÉELLES dans chaque lot (elles ne font que 21 % du dataset)
27
- MAX_HOURS = float(E("GC_MAX_HOURS", 8.0)) # garde-fou : on arrête l'entraînement avant la limite Kaggle (12 h)
28
- N_TEST = int(E("GC_NTEST", 0)) # 0 = tout le jeu de test
29
- OUT = Path(E("GC_OUT", "/kaggle/working/geocarto_model")); OUT.mkdir(parents=True, exist_ok=True)
30
- DEV = "cuda" if torch.cuda.is_available() else "cpu"
31
- T_START = time.time()
32
- print("Matériel :", DEV, torch.cuda.get_device_name(0) if DEV == "cuda" else "", flush=True)
33
-
34
- CFG = dict(arch="unetpp", encoder=E("GC_ENCODER", "efficientnet-b4"), input="rgb+lab",
35
- offsets=[(1, 0), (0, 1), (3, 0), (0, 3), (9, 0), (0, 9), (9, 9), (9, -9), (27, 0), (0, 27)],
36
- n_short=2, mws_stride=int(E("GC_MWS_STRIDE", 1)), mws_min_size=int(E("GC_MINSIZE", 40)), tile=512,
37
- small_w=E("GC_SMALLW", "0") == "1", ohem=E("GC_OHEM", "0") == "1", tta=E("GC_TTA", "0") == "1")
38
- OFFSETS, N_SHORT, K = CFG["offsets"], CFG["n_short"], len(CFG["offsets"])
39
-
40
-
41
- # ------------------------------------------------------------------ données
42
- def find_dir(marker):
43
- root = Path(E("GC_INPUT", "/kaggle/input"))
44
- hits = sorted(root.rglob(marker))
45
- return hits[0].parent if hits else None
46
-
47
-
48
- DATA = Path(E("GC_DATA")) if E("GC_DATA") else find_dir("test-00000.parquet") # dataset complet
49
- HARD = Path(E("GC_HARD")) if E("GC_HARD") else find_dir("hard_mono-00000.parquet") # jeux pièges (geocarto-mini)
50
- assert DATA is not None, "dataset GeoCarto introuvable (ajoute la sortie du notebook geocarto-dataset en entrée)"
51
- print("Données :", DATA, "| pièges :", HARD, flush=True)
52
-
53
-
54
- def read_split(folder, split):
55
- """Garde les PNG compressés en mémoire (2 Go au lieu de 11 Go décodés) ; décodage à la volée."""
56
- rows = []
57
- for f in sorted(Path(folder).glob(f"{split}-*.parquet")):
58
- t = pq.read_table(f, columns=["id", "source", "continent", "kind", "image", "instances", "faults"]).to_pylist()
59
- rows += [dict(id=r["id"], source=r["source"], continent=r["continent"], kind=r["kind"],
60
- image=r["image"]["bytes"], inst=r["instances"]["bytes"], faults=r["faults"]["bytes"]) for r in t]
61
- return rows
62
-
63
-
64
- def decode(r):
65
- rgb = np.asarray(Image.open(io.BytesIO(r["image"])).convert("RGB"))
66
- inst = np.asarray(Image.open(io.BytesIO(r["inst"]))).astype(np.int32)
67
- flt = (np.asarray(Image.open(io.BytesIO(r["faults"]))) > 0).astype(np.uint8)
68
- return rgb, inst, flt
69
-
70
-
71
- TRAIN = read_split(DATA, "train")
72
- VAL_ALL = read_split(DATA, "validation")
73
- TEST = read_split(DATA, "test")
74
- HARDSETS = {k: read_split(HARD, k) for k in ("hard_mono", "hard_pattern")} if HARD else {}
75
- REAL_IDX = [i for i, r in enumerate(TRAIN) if r["kind"] == "real"]
76
- SYN_IDX = [i for i, r in enumerate(TRAIN) if r["kind"] != "real"]
77
-
78
-
79
- def spread(rows, n):
80
- return [rows[i] for i in np.linspace(0, len(rows) - 1, min(n, len(rows))).astype(int)] if rows else []
81
-
82
-
83
- VAL = {} # suivi : n tuiles par (source, type), réparties régulièrement
84
- for key in sorted({(r["source"], r["kind"]) for r in VAL_ALL}):
85
- VAL[key] = spread([r for r in VAL_ALL if (r["source"], r["kind"]) == key], N_VAL * (3 if key[1] == "real" else 1))
86
- print(f"train {len(TRAIN)} ({len(REAL_IDX)} réelles) | validation suivie {sum(map(len, VAL.values()))} | test {len(TEST)} | "
87
- + " | ".join(f"{k} {len(v)}" for k, v in HARDSETS.items()), flush=True)
88
-
89
-
90
- # ------------------------------------------------------------------ affinités + mutex watershed
91
- def fill_holes(inst):
92
- """Chaque zone sans polygone (mer, lac, halo) devient sa propre r��gion."""
93
- holes, n = ndi.label(inst == 0)
94
- if n == 0:
95
- return inst
96
- out = inst.copy()
97
- out[holes > 0] = holes[holes > 0] + inst.max()
98
- return out
99
-
100
-
101
- def affinity_targets(inst):
102
- H, W = inst.shape
103
- aff, mask = np.zeros((K, H, W), np.float32), np.zeros((K, H, W), np.float32)
104
- for k, (dy, dx) in enumerate(OFFSETS):
105
- ys, ye, xs, xe = max(0, -dy), H - max(0, dy), max(0, -dx), W - max(0, dx)
106
- a, b = inst[ys:ye, xs:xe], inst[ys + dy:ye + dy, xs + dx:xe + dx]
107
- aff[k, ys:ye, xs:xe] = a == b
108
- mask[k, ys:ye, xs:xe] = 1 # après fill_holes, tous les pixels ont une région
109
- return aff, mask
110
-
111
-
112
- @njit
113
- def _find(parent, x):
114
- while parent[x] != x:
115
- parent[x] = parent[parent[x]]
116
- x = parent[x]
117
- return x
118
-
119
-
120
- @njit
121
- def _mws_core(n, u, v, rep):
122
- parent = np.arange(n); size = np.ones(n, np.int64); m = len(u)
123
- mutex_a = np.empty(m, np.int64); mutex_b = np.empty(m, np.int64); nm = 0
124
- head = -np.ones(n, np.int64); nxt = np.empty(2 * m, np.int64); val = np.empty(2 * m, np.int64); nl = 0
125
- for e in range(m):
126
- ra, rb = _find(parent, u[e]), _find(parent, v[e])
127
- if ra == rb:
128
- continue
129
- if rep[e]:
130
- val[nl] = nm; nxt[nl] = head[ra]; head[ra] = nl; nl += 1
131
- val[nl] = nm; nxt[nl] = head[rb]; head[rb] = nl; nl += 1
132
- mutex_a[nm] = ra; mutex_b[nm] = rb; nm += 1
133
- else:
134
- small, other = (ra, rb) if size[ra] <= size[rb] else (rb, ra)
135
- blocked = False; p = head[small]
136
- while p != -1:
137
- k = val[p]; x = _find(parent, mutex_a[k]); y = _find(parent, mutex_b[k])
138
- if (x == small and y == other) or (x == other and y == small):
139
- blocked = True; break
140
- p = nxt[p]
141
- if blocked:
142
- continue
143
- parent[small] = other; size[other] += size[small]; p = head[small]
144
- if p != -1:
145
- while nxt[p] != -1:
146
- p = nxt[p]
147
- nxt[p] = head[other]; head[other] = head[small]
148
- out = np.empty(n, np.int64)
149
- for i in range(n):
150
- out[i] = _find(parent, i)
151
- return out
152
-
153
-
154
- def mutex_watershed(aff, stride=CFG["mws_stride"], min_size=CFG["mws_min_size"]):
155
- _, H, W = aff.shape
156
- idx = np.arange(H * W).reshape(H, W)
157
- us, vs, ws, rep = [], [], [], []
158
- for k, (dy, dx) in enumerate(OFFSETS):
159
- a = idx[max(0, -dy):H - max(0, dy), max(0, -dx):W - max(0, dx)]
160
- b = idx[max(0, dy):H - max(0, -dy), max(0, dx):W - max(0, -dx)]
161
- w = aff[k, max(0, -dy):H - max(0, dy), max(0, -dx):W - max(0, dx)].copy()
162
- if k >= N_SHORT:
163
- a, b, w = a[::stride, ::stride], b[::stride, ::stride], 1 - w[::stride, ::stride]
164
- us.append(a.ravel()); vs.append(b.ravel()); ws.append(w.ravel()); rep.append(np.full(a.size, k >= N_SHORT))
165
- u, v, w, rep = map(np.concatenate, (us, vs, ws, rep))
166
- order = np.argsort(-w, kind="stable")
167
- lab = _mws_core(H * W, u[order], v[order], rep[order]).reshape(H, W)
168
- _, lab = np.unique(lab, return_inverse=True); lab = lab.reshape(H, W) + 1
169
- sizes = np.bincount(lab.ravel()); tiny = sizes < min_size; tiny[0] = False
170
- if tiny[lab].any():
171
- _, (iy, ix) = ndi.distance_transform_edt(tiny[lab], return_indices=True)
172
- lab = lab[iy, ix]; _, lab = np.unique(lab, return_inverse=True); lab = lab.reshape(H, W) + 1
173
- return lab.astype(np.int32)
174
-
175
-
176
- # ------------------------------------------------------------------ augmentations
177
- def color_aug(rgb, rng):
178
- """Casse le lien 'couleur = unité' : rotation de teinte, saturation, gains, permutation, gris, contraste, bruit."""
179
- x = rgb.astype(np.float32) / 255
180
- if rng.random() < 0.8:
181
- T = np.array([[0.299, 0.587, 0.114], [0.596, -0.274, -0.322], [0.211, -0.523, 0.312]], np.float32)
182
- a = rng.uniform(-np.pi, np.pi)
183
- Rm = np.array([[1, 0, 0], [0, np.cos(a), -np.sin(a)], [0, np.sin(a), np.cos(a)]], np.float32)
184
- Rm[1:, 1:] *= rng.uniform(0.6, 1.3)
185
- x = x @ (np.linalg.inv(T) @ Rm @ T).T
186
- if rng.random() < 0.3:
187
- x = x[..., rng.permutation(3)]
188
- x = x * rng.uniform(0.85, 1.15, 3) + rng.uniform(-0.06, 0.06, 3)
189
- if rng.random() < 0.1:
190
- x = np.repeat(x.mean(-1, keepdims=True), 3, -1)
191
- m = x.mean()
192
- x = (x - m) * rng.uniform(0.75, 1.2) + m
193
- if rng.random() < 0.3:
194
- x = x + rng.normal(0, rng.uniform(0.005, 0.03), x.shape)
195
- return np.clip(x * 255, 0, 255).astype(np.uint8)
196
-
197
-
198
- class TrainSet(Dataset):
199
- def __len__(self):
200
- return STEPS * BATCH
201
-
202
- def __getitem__(self, i):
203
- rng = np.random.default_rng([i, 20261112])
204
- pool = REAL_IDX if (REAL_IDX and rng.random() < P_REAL) else SYN_IDX
205
- rgb, inst, flt = decode(TRAIN[pool[rng.integers(len(pool))]])
206
- inst = fill_holes(inst) # sur la tuile entière, avant recadrage
207
- y, x = rng.integers(0, rgb.shape[0] - CROP + 1), rng.integers(0, rgb.shape[1] - CROP + 1)
208
- rgb, inst, flt = (a[y:y + CROP, x:x + CROP] for a in (rgb, inst, flt))
209
- k = int(rng.integers(4))
210
- rgb, inst, flt = (np.rot90(a, k) for a in (rgb, inst, flt))
211
- if rng.random() < 0.5:
212
- rgb, inst, flt = rgb[:, ::-1], inst[:, ::-1], flt[:, ::-1]
213
- rgb = color_aug(rgb, rng)
214
- inst = np.ascontiguousarray(inst)
215
- aff, mask = affinity_targets(inst)
216
- if CFG["small_w"]: # les arêtes touchant un petit polygone pèsent plus (jusqu'à x8)
217
- area = np.bincount(inst.ravel())
218
- w = np.clip(np.sqrt(2000.0 / np.maximum(area[inst], 1)), 1.0, 8.0).astype(np.float32)
219
- H, W = inst.shape
220
- for k, (dy, dx) in enumerate(OFFSETS):
221
- ys, ye, xs, xe = max(0, -dy), H - max(0, dy), max(0, -dx), W - max(0, dx)
222
- mask[k, ys:ye, xs:xe] = np.maximum(w[ys:ye, xs:xe], w[ys + dy:ye + dy, xs + dx:xe + dx])
223
- return (torch.from_numpy(np.ascontiguousarray(rgb).transpose(2, 0, 1).copy()), torch.from_numpy(aff),
224
- torch.from_numpy(mask), torch.from_numpy(np.ascontiguousarray(flt, np.float32))[None])
225
-
226
-
227
- # ------------------------------------------------------------------ modèle
228
- def rgb2lab(x):
229
- """x : B x 3 x H x W dans [0,1] -> Lab normalisé (L/50-1, a/64, b/64)."""
230
- lin = torch.where(x <= 0.04045, x / 12.92, ((x + 0.055) / 1.055) ** 2.4)
231
- M = torch.tensor([[0.4124, 0.3576, 0.1805], [0.2126, 0.7152, 0.0722], [0.0193, 0.1192, 0.9505]], device=x.device)
232
- xyz = torch.einsum("ij,bjhw->bihw", M, lin) / torch.tensor([0.95047, 1.0, 1.08883], device=x.device).view(1, 3, 1, 1)
233
- f = torch.where(xyz > 0.008856, xyz.clamp(min=1e-6) ** (1 / 3), 7.787 * xyz + 16 / 116)
234
- L, a, b = 116 * f[:, 1] - 16, 500 * (f[:, 0] - f[:, 1]), 200 * (f[:, 1] - f[:, 2])
235
- return torch.stack([L / 50 - 1, a / 64, b / 64], 1)
236
-
237
-
238
- class GeoCartoNet(nn.Module):
239
- """Entrée : image uint8 B x 3 x H x W (H, W multiples de 32). Sortie : K affinités + 1 carte de failles (logits)."""
240
-
241
- def __init__(self, cfg, pretrained=True):
242
- super().__init__()
243
- self.cfg = cfg
244
- self.register_buffer("mean", torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1))
245
- self.register_buffer("std", torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1))
246
- Arch = {"unet": smp.Unet, "unetpp": smp.UnetPlusPlus}[cfg["arch"]]
247
- self.net = Arch(cfg["encoder"], encoder_weights="imagenet" if pretrained else None, in_channels=6,
248
- classes=len(cfg["offsets"]) + 1)
249
-
250
- def forward(self, x8):
251
- with torch.autocast(x8.device.type, enabled=False): # conversion de couleur en précision complète
252
- x = x8.float() / 255
253
- x = torch.cat([(x - self.mean) / self.std, rgb2lab(x)], 1)
254
- return self.net(x)
255
-
256
-
257
- def dice(p, t, eps=1.0, dims=None):
258
- return 1 - (2 * (p * t).sum(dims) + eps) / (p.sum(dims) + t.sum(dims) + eps)
259
-
260
-
261
- def loss_fn(out, aff, mask, flt):
262
- bce = F.binary_cross_entropy_with_logits(out[:, :K], aff, reduction="none") * mask
263
- la = bce.sum() / mask.sum().clamp(min=1)
264
- if CFG["ohem"]: # + moyenne des 5 % d'arêtes les plus mal prédites
265
- v = bce.flatten()
266
- la = la + v.topk(max(1, v.numel() // 20)).values.mean() / mask.mean().clamp(min=1e-6)
267
- p, t = (1 - torch.sigmoid(out[:, :K])) * mask, (1 - aff) * mask # Dice sur les COUPURES (classe rare)
268
- la = la + dice(p, t, dims=(0, 2, 3)).mean()
269
- lf = F.binary_cross_entropy_with_logits(out[:, K:], flt, pos_weight=torch.tensor(10.0, device=out.device)) \
270
- + dice(torch.sigmoid(out[:, K:]), flt)
271
- return la + 0.5 * lf
272
-
273
-
274
- # ------------------------------------------------------------------ évaluation
275
- @torch.no_grad()
276
- def predict(model, rgb):
277
- x = torch.from_numpy(rgb.transpose(2, 0, 1).copy())[None].to(DEV)
278
- with torch.autocast(DEV, enabled=DEV == "cuda"):
279
- o = model(x)
280
- o = torch.sigmoid(o.float())[0].cpu().numpy()
281
- return o[:K], o[K]
282
-
283
-
284
- def predict_tta(model, rgb):
285
- """Moyenne sur 4 miroirs ; chaque décalage ramené dans le repère d'origine (testé : exact)."""
286
- H, W = rgb.shape[:2]
287
- Y, X = np.mgrid[:H, :W]
288
- idx = {tuple(o): k for k, o in enumerate(OFFSETS)}
289
- acc, accf = 0, 0
290
- for fy in (0, 1):
291
- for fx in (0, 1):
292
- im = rgb[::-1] if fy else rgb
293
- im = im[:, ::-1] if fx else im
294
- A, pf = predict(model, np.ascontiguousarray(im))
295
- py, px = (H - 1 - Y) if fy else Y, (W - 1 - X) if fx else X
296
- out = np.empty_like(A)
297
- for k, (dy, dx) in enumerate(OFFSETS):
298
- t = (-dy if fy else dy, -dx if fx else dx)
299
- if t in idx:
300
- out[k] = A[idx[t]][py, px]
301
- else:
302
- out[k] = A[idx[(-t[0], -t[1])]][np.clip(py + t[0], 0, H - 1), np.clip(px + t[1], 0, W - 1)]
303
- acc = acc + out; accf = accf + pf[py, px]
304
- return acc / 4, accf / 4
305
-
306
-
307
- def tile_metrics(pred, pf, inst, flt):
308
- gt = fill_holes(inst)
309
- are = adapted_rand_error(gt, pred)[0]
310
- split, merge = variation_of_information(gt, pred) # H(pred|vérité) = sur-découpage ; H(vérité|pred) = fusion
311
- small = [0, 0]; big = [0, 0]
312
- ids, area = np.unique(inst[inst > 0], return_counts=True)
313
- if len(ids):
314
- # IoU de chaque instance avec la région prédite qui la recouvre le plus
315
- pair = inst.astype(np.int64) * (pred.max() + 1) + pred
316
- pv, pc = np.unique(pair[inst > 0], return_counts=True)
317
- gi, pi = pv // (pred.max() + 1), pv % (pred.max() + 1)
318
- parea = np.bincount(pred.ravel())
319
- order = np.lexsort((-pc, gi))
320
- first = np.r_[True, gi[order][1:] != gi[order][:-1]]
321
- g, inter, pa = gi[order][first], pc[order][first], parea[pi[order][first]]
322
- a = area[np.searchsorted(ids, g)]
323
- ok = inter / (a + pa - inter) >= 0.5
324
- small = [int(ok[a < 1000].sum()), int((a < 1000).sum())]; big = [int(ok[a >= 1000].sum()), int((a >= 1000).sum())]
325
- mid = [int(ok[(a >= 200) & (a < 1000)].sum()), int(((a >= 200) & (a < 1000)).sum())]
326
- tiny = [int(ok[a < 200].sum()), int((a < 200).sum())]
327
- surf = [float((a * ok).sum()), float(a.sum())]
328
- else:
329
- mid = tiny = [0, 0]; surf = [0.0, 0.0]
330
- pb, gb = pf > 0.5, flt > 0
331
- gd, pd_ = ndi.binary_dilation(gb, iterations=2), ndi.binary_dilation(pb, iterations=2)
332
- return dict(ARAND=are, split=split, merge=merge, s_ok=small[0], s_n=small[1], b_ok=big[0], b_n=big[1],
333
- t_ok=tiny[0], t_n=tiny[1], m_ok=mid[0], m_n=mid[1], surf_ok=surf[0], surf_n=surf[1],
334
- tp=int((pb & gd).sum()), fp=int((pb & ~gd).sum()), fn=int((gb & ~pd_).sum()))
335
-
336
-
337
- def summarize(rows):
338
- d = pd.DataFrame(rows)
339
- f1 = 2 * d.tp.sum() / max(2 * d.tp.sum() + d.fp.sum() + d.fn.sum(), 1) if (d.tp.sum() + d.fn.sum()) > 0 else np.nan
340
- return dict(tuiles=len(d), ARAND=d.ARAND.mean(), sur_decoupage=d.split.mean(), fusion=d["merge"].mean(),
341
- rappel_petites=d.s_ok.sum() / max(d.s_n.sum(), 1), rappel_grandes=d.b_ok.sum() / max(d.b_n.sum(), 1),
342
- rappel_40_200=d.t_ok.sum() / max(d.t_n.sum(), 1), rappel_200_1000=d.m_ok.sum() / max(d.m_n.sum(), 1),
343
- rappel_tous=(d.s_ok.sum() + d.b_ok.sum()) / max(d.s_n.sum() + d.b_n.sum(), 1),
344
- surface_correcte=d.surf_ok.sum() / max(d.surf_n.sum(), 1), F1_failles=f1)
345
-
346
-
347
- def evaluate(model, rows, tta=False):
348
- model.eval()
349
- out = []
350
- for r in rows:
351
- rgb, inst, flt = decode(r)
352
- aff, pf = predict_tta(model, rgb) if tta else predict(model, rgb)
353
- out.append(tile_metrics(mutex_watershed(aff), pf, inst, flt))
354
- return out
355
-
356
-
357
- def val_score(model):
358
- """Score de suivi : moyenne de l'ARAND réel et de l'ARAND synthétique (toutes sources à poids égal)."""
359
- res = {k: summarize(evaluate(model, v)) for k, v in VAL.items()}
360
- real = np.mean([v["ARAND"] for k, v in res.items() if k[1] == "real"])
361
- syn = np.mean([v["ARAND"] for k, v in res.items() if k[1] != "real"])
362
- return 0.5 * (real + syn), real, syn, res
363
-
364
-
365
- # ------------------------------------------------------------------ entraînement
366
- torch.manual_seed(0)
367
- model = GeoCartoNet(CFG).to(DEV)
368
- ema = copy.deepcopy(model).eval() # moyenne glissante des poids (souvent plus stable)
369
- for p in ema.parameters():
370
- p.requires_grad_(False)
371
- EMA_DECAY = 0.999
372
- dl = DataLoader(TrainSet(), BATCH, shuffle=False, num_workers=min(4, os.cpu_count() or 1), pin_memory=DEV == "cuda",
373
- drop_last=True)
374
- opt = torch.optim.AdamW(model.parameters(), LR, weight_decay=1e-4)
375
- sched = torch.optim.lr_scheduler.OneCycleLR(opt, LR, total_steps=STEPS, pct_start=0.05)
376
- scaler = torch.amp.GradScaler("cuda", enabled=DEV == "cuda")
377
- mutex_watershed(np.random.rand(K, 64, 64).astype(np.float32)) # compilation numba
378
- print(f"Paramètres : {sum(p.numel() for p in model.parameters()) / 1e6:.1f} M | {STEPS} pas, lot {BATCH}, recadrage {CROP}", flush=True)
379
-
380
- HIST, best, t0, run_loss, n_loss = [], 1e9, time.time(), 0.0, 0
381
- model.train()
382
- step = 0
383
- for step, batch in enumerate(dl, 1):
384
- x, aff, mask, flt = (t.to(DEV, non_blocking=True) for t in batch)
385
- with torch.autocast(DEV, enabled=DEV == "cuda"):
386
- out = model(x)
387
- loss = loss_fn(out.float(), aff, mask, flt)
388
- if not torch.isfinite(loss):
389
- print(f" pas {step} : perte non finie, lot ignoré", flush=True)
390
- opt.zero_grad(set_to_none=True); sched.step()
391
- continue
392
- opt.zero_grad(set_to_none=True)
393
- scaler.scale(loss).backward()
394
- scaler.unscale_(opt); torch.nn.utils.clip_grad_norm_(model.parameters(), 5.0)
395
- scaler.step(opt); scaler.update(); sched.step()
396
- with torch.no_grad():
397
- d = min(EMA_DECAY, (1 + step) / (10 + step))
398
- for pe, pm in zip(ema.parameters(), model.parameters()):
399
- pe.mul_(d).add_(pm.detach(), alpha=1 - d)
400
- for be, bm in zip(ema.buffers(), model.buffers()):
401
- be.copy_(bm)
402
- run_loss += loss.item(); n_loss += 1
403
- if step % 250 == 0:
404
- print(f"pas {step}/{STEPS} | perte {run_loss / n_loss:.4f} | {(time.time() - t0) / 60:.0f} min", flush=True)
405
- hours = (time.time() - T_START) / 3600
406
- last = step == STEPS or hours > MAX_HOURS
407
- if step % EVAL_EVERY == 0 or last:
408
- row = dict(step=step, loss=run_loss / max(n_loss, 1), heures=round(hours, 2))
409
- for tag, m in (("modele", model), ("ema", ema)):
410
- sc, real, syn, _ = val_score(m)
411
- row.update({f"{tag}_score": sc, f"{tag}_reel": real, f"{tag}_synth": syn})
412
- if sc < best:
413
- best = sc
414
- torch.save({"state_dict": m.state_dict(), "cfg": CFG, "step": step, "val_score": float(sc), "poids": tag}, OUT / "best.pt")
415
- row["meilleur"] = tag
416
- model.train()
417
- HIST.append(row); pd.DataFrame(HIST).to_csv(OUT / "history.csv", index=False)
418
- print(" VALIDATION", json.dumps({k: (round(v, 4) if isinstance(v, float) else v) for k, v in row.items()}), flush=True)
419
- run_loss, n_loss = 0.0, 0
420
- if last:
421
- if step < STEPS:
422
- print(f"Arrêt au pas {step} : limite de {MAX_HOURS} h atteinte.", flush=True)
423
- break
424
- del dl
425
-
426
- # ------------------------------------------------------------------ test final (meilleur point de contrôle)
427
- ck = torch.load(OUT / "best.pt", map_location=DEV)
428
- model = GeoCartoNet(CFG, pretrained=False).to(DEV)
429
- model.load_state_dict(ck["state_dict"]); model.eval()
430
- torch.save({"state_dict": {k: (v.half() if v.is_floating_point() else v) for k, v in ck["state_dict"].items()},
431
- "cfg": CFG, "step": ck["step"]}, OUT / "best_fp16.pt")
432
- print(f"\nMeilleur modèle : pas {ck['step']} ({ck['poids']}), score de validation {ck['val_score']:.4f}", flush=True)
433
-
434
- test_rows = TEST if not N_TEST else spread(TEST, N_TEST)
435
- res = evaluate(model, test_rows, tta=CFG["tta"])
436
- T = pd.DataFrame(res).assign(continent=[r["continent"] for r in test_rows], source=[r["source"] for r in test_rows],
437
- type=[r["kind"] for r in test_rows])
438
- tables = []
439
- for keys in (["continent"], ["source", "type"]):
440
- for name, g in T.groupby(keys):
441
- name = name if isinstance(name, tuple) else (name,)
442
- tables.append({"groupe": " / ".join(name), **summarize(g.to_dict("records"))})
443
- tables.append({"groupe": "TEST COMPLET", **summarize(res)})
444
- for k, rows in HARDSETS.items():
445
- rows = rows if not N_TEST else spread(rows, N_TEST)
446
- tables.append({"groupe": {"hard_mono": "PIÈGE unités de même couleur", "hard_pattern": "PIÈGE unités à motifs"}[k],
447
- **summarize(evaluate(model, rows, tta=CFG["tta"]))})
448
- TM = pd.DataFrame(tables).round(4)
449
- TM.to_csv(OUT / "test_metrics.csv", index=False)
450
- pd.set_option("display.width", 220); pd.set_option("display.max_columns", 20)
451
- print("\n=== RÉSULTATS SUR LE JEU DE TEST (jamais vu à l'entraînement) ==="); print(TM.to_string(index=False), flush=True)
452
- json.dump({"cfg": CFG, "steps": int(ck["step"]), "val_score": float(ck["val_score"]), "test": TM.to_dict("records"),
453
- "train_hours": round((time.time() - T_START) / 3600, 2)}, open(OUT / "metrics.json", "w"), indent=1, ensure_ascii=False)
454
-
455
- # ------------------------------------------------------------------ figures
456
- import matplotlib; matplotlib.use("Agg"); import matplotlib.pyplot as plt
457
- picks = [("réel", spread([r for r in TEST if r["kind"] == "real"], 4))]
458
- picks += [(s, spread([r for r in TEST if r["source"] == s and r["kind"] != "real"], 2)) for s in sorted({r["source"] for r in TEST})]
459
- picks += [(k, spread(v, 3)) for k, v in HARDSETS.items()]
460
- rows_ = [(n, r) for n, rs in picks for r in rs]
461
- fig, ax = plt.subplots(3, len(rows_), figsize=(4 * len(rows_), 12))
462
- for j, (n, r) in enumerate(rows_):
463
- rgb, inst, flt = decode(r); aff, pf = predict(model, rgb); pred = mutex_watershed(aff)
464
- ax[0, j].imshow(rgb); ax[0, j].set_title(f"{n}\n{r['source']}", fontsize=9)
465
- ax[1, j].imshow(fill_holes(inst) % 20, cmap="tab20", interpolation="nearest"); ax[1, j].set_title("vérité", fontsize=9)
466
- ax[2, j].imshow(pred % 20, cmap="tab20", interpolation="nearest")
467
- ax[2, j].imshow(np.ma.masked_less(pf, 0.5), cmap="gray_r", interpolation="nearest"); ax[2, j].set_title("modèle", fontsize=9)
468
- for a in ax.ravel():
469
- a.axis("off")
470
- plt.tight_layout(); plt.savefig(OUT / "test_examples.png", dpi=55); plt.close()
471
- print("Fichiers :", sorted(p.name for p in OUT.iterdir()), f"| durée totale {(time.time() - T_START) / 3600:.2f} h", flush=True)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
vectorize.py DELETED
@@ -1,56 +0,0 @@
1
- """GeoCarto – vectorise a georeferenced geological map raster into a GeoPackage (QGIS / ArcGIS).
2
-
3
- python vectorize.py carte.tif legende.csv sortie.gpkg [--model best_fp16.pt] [--tta]
4
-
5
- legende.csv : one row per legend entry, columns
6
- code (required, as printed on the map), nom, age, lithologie (optional),
7
- and EITHER x0,y0,x1,y1 = pixel box of the legend swatch in carte.tif (best),
8
- OR couleur_hex = colour of the swatch (e.g. #f2c38a).
9
- Unit names come only from text read on the map and from this legend.
10
- """
11
- import argparse, os, sys
12
- import numpy as np, pandas as pd, rasterio
13
- from skimage.color import rgb2lab
14
- HERE = os.path.dirname(os.path.abspath(__file__)); sys.path.insert(0, HERE)
15
- import pipeline as PL
16
-
17
-
18
- def load_legend(csv, rgb):
19
- L = pd.read_csv(csv, dtype={"code": str}).fillna("")
20
- lab = None
21
- feats, hexes = [], []
22
- for r in L.itertuples():
23
- if all(hasattr(r, k) and str(getattr(r, k)) != "" for k in ("x0", "y0", "x1", "y1")):
24
- x0, y0, x1, y1 = (int(float(getattr(r, k))) for k in ("x0", "y0", "x1", "y1"))
25
- px = rgb[y0:y1, x0:x1].reshape(-1, 3)
26
- feats.append(PL.swatch_features(rgb2lab(px[None])[0])); hexes.append("#%02x%02x%02x" % tuple(int(v) for v in np.median(px, 0)))
27
- elif getattr(r, "couleur_hex", ""):
28
- h = r.couleur_hex.lstrip("#"); c = np.array([[int(h[i:i + 2], 16) for i in (0, 2, 4)]], np.uint8)
29
- feats.append(PL.swatch_features(np.repeat(rgb2lab(c[None])[0], 50, 0))); hexes.append("#" + h)
30
- else:
31
- raise SystemExit(f"légende : la ligne {r.code} n'a ni case (x0,y0,x1,y1) ni couleur_hex")
32
- out = pd.DataFrame(index=L.code.values)
33
- for c in ("nom", "age", "lithologie"):
34
- out[c] = L[c].values if c in L else ""
35
- out["nom"] = np.where(out.nom == "", out.index, out.nom)
36
- out["couleur_hex"] = hexes; out["feat"] = feats
37
- return out
38
-
39
-
40
- if __name__ == "__main__":
41
- ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
42
- ap.add_argument("raster"); ap.add_argument("legend"); ap.add_argument("out")
43
- ap.add_argument("--model", default=os.path.join(HERE, "best_fp16.pt"))
44
- ap.add_argument("--tta", action="store_true", help="moyenne sur 4 symétries (plus lent, pas meilleur en moyenne)")
45
- a = ap.parse_args()
46
- import easyocr
47
- with rasterio.open(a.raster) as s:
48
- rgb = np.moveaxis(s.read([1, 2, 3]), 0, -1).astype(np.uint8); tr, crs = s.transform, s.crs
49
- legend = load_legend(a.legend, rgb)
50
- print(f"carte {rgb.shape[1]}×{rgb.shape[0]} px | légende : {len(legend)} unités")
51
- model = PL.load_model(a.model)
52
- reader = easyocr.Reader(["en"], gpu=False, verbose=False)
53
- P, F, lab, pf = PL.run(rgb, tr, crs, model, legend, reader, os.path.join(HERE, "naming_model.pkl"), a.out,
54
- tta=a.tta, ranker_pkl=os.path.join(HERE, "legend_ranker.pkl"))
55
- n = P.code.ne("").sum()
56
- print(f"\n{len(P)} unités ({n} nommées, {len(P) - n} 'Unité non identifiée' avec suggestions), {len(F)} failles -> {a.out}")
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
water_model.pkl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a1a1cb40dd1f3ea2de836e854ac6fcfd7365623954f581126d12b388ed8861fd
3
+ size 569282