Rank ChangeMamba in LEVIR comparison

README.md

@@ -179,18 +179,20 @@ Every dataset below is a real YAML file under `configs/datasets/`. The harness e
 
 ## Results and released checkpoints
 
-Only completed `metrics_test.json` evaluations are eligible for rankings. Rankings are computed per dataset by test-set F1 (descending), and the canonical `best_model.pth` checkpoint for every ranked model is released. Separately requested checkpoint-only artifacts are listed below without fabricated metrics or a rank.
+Only completed `metrics_test.json` evaluations are eligible for rankings. Rankings are computed per dataset by test-set F1 (descending), and the canonical `best_model.pth` checkpoint for every ranked model is released.
 
 All accuracy values are fractions. GPU memory is the PyTorch peak reserved memory for inference, with peak allocated memory used only when reserved memory is absent. FPS is model-only throughput. A dash means the evaluator did not record that metric; values are never estimated.
 
 The current evaluator records BF1 but not boundary mean IoU (BmIoU), so BmIoU remains explicitly unavailable rather than being inferred from BF1. Boundary IoU and boundary F1 are distinct measures.
 
+The LEVIR-CD+ comparison combines the project-compatible `levir_cd_test_as_val` and `levir_val_as_test` result records. The ChangeMamba row uses the completed `levir_val_as_test` evaluation over 5,568 samples.
+
 ### Dataset summary
 
 | Dataset | Tested models | F1 leader | Best test F1 |
 | --- | --- | --- | --- |
 | DSIFN-CD | 14 | BIT-CD | 0.6732 |
-| LEVIR-CD+ (test-as-val protocol) | 13 | CGNet | 0.8138 |
+| LEVIR-CD+ comparison | 14 | ChangeMamba | 0.8753 |
 | SYSU-CD | 8 | ChangeMamba | 0.8216 |
 | WHU-CD | 8 | ChangeMamba | 0.9521 |
 
@@ -213,23 +215,24 @@ The current evaluator records BF1 but not boundary mean IoU (BmIoU), so BmIoU re
 | 13 | HANet | 0.5347 | 0.5817 | 0.8193 | 0.6109 | 0.4754 | 0.2418 | — | 17.67 | 11.36 | 42.36 | 3.03 | [Download](https://huggingface.co/dineth18/CD-Models/resolve/main/results/hanet/dsifn_cd/checkpoints/best_model.pth?download=true) |
 | 14 | IFNet | 0.4579 | 0.5102 | 0.7524 | 0.6153 | 0.3647 | 0.1325 | — | 82.26 | 6.08 | 104.88 | 50.71 | [Download](https://huggingface.co/dineth18/CD-Models/resolve/main/results/ifnet/dsifn_cd/checkpoints/best_model.pth?download=true) |
 
-### LEVIR-CD+ (test-as-val protocol)
+### LEVIR-CD+ comparison
 
 | Rank | Model | F1 | mIoU | Overall accuracy | Recall | Precision | BF1 | BmIoU | GFLOPs | GPU (GB) | FPS | Parameters (M) | Checkpoint |
 | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
-| 1 | CGNet | 0.8138 | 0.8356 | 0.9856 | 0.7746 | 0.8572 | 0.7651 | — | 82.23 | 6.01 | 44.01 | 38.99 | [Download](https://huggingface.co/dineth18/CD-Models/resolve/main/results/cgnet/levir_cd_test_as_val/checkpoints/best_model.pth?download=true) |
-| 2 | Siam-NestedUNet | 0.8046 | 0.8282 | 0.9839 | 0.8129 | 0.7965 | 0.7386 | — | — | 2.67 | 108.71 | — | [Download](https://huggingface.co/dineth18/CD-Models/resolve/main/results/siam_nestedunet/levir_cd_test_as_val/checkpoints/best_model.pth?download=true) |
-| 3 | SChanger | 0.8007 | 0.8253 | 0.9836 | 0.8112 | 0.7906 | 0.7364 | — | — | 2.45 | 48.96 | — | [Download](https://huggingface.co/dineth18/CD-Models/resolve/main/results/schanger/levir_cd_test_as_val/checkpoints/best_model.pth?download=true) |
-| 4 | BIT-CD | 0.7996 | 0.8247 | 0.9838 | 0.7946 | 0.8047 | 0.7143 | — | 10.63 | 2.27 | 236.60 | 12.40 | [Download](https://huggingface.co/dineth18/CD-Models/resolve/main/results/bit_cd/levir_cd_test_as_val/checkpoints/best_model.pth?download=true) |
-| 5 | HANet | 0.7841 | 0.8131 | 0.9819 | 0.8064 | 0.7630 | 0.7043 | — | — | 5.04 | 84.44 | — | [Download](https://huggingface.co/dineth18/CD-Models/resolve/main/results/hanet/levir_cd_test_as_val/checkpoints/best_model.pth?download=true) |
-| 6 | BiFA | 0.7772 | 0.8084 | 0.9818 | 0.7785 | 0.7760 | 0.6581 | — | 53.00 | 15.14 | 30.38 | 9.87 | [Download](https://huggingface.co/dineth18/CD-Models/resolve/main/results/bifa/levir_cd_test_as_val/checkpoints/best_model.pth?download=true) |
-| 7 | FC-Siam-conc | 0.7650 | 0.7999 | 0.9809 | 0.7623 | 0.7678 | 0.7175 | — | 5.33 | 1.02 | 308.81 | 1.55 | [Download](https://huggingface.co/dineth18/CD-Models/resolve/main/results/fc_siam_conc/levir_cd_test_as_val/checkpoints/best_model.pth?download=true) |
-| 8 | STANet | 0.7544 | 0.7921 | 0.9791 | 0.7865 | 0.7249 | 0.6388 | — | — | 4.45 | 174.00 | — | [Download](https://huggingface.co/dineth18/CD-Models/resolve/main/results/stanet/levir_cd_test_as_val/checkpoints/best_model.pth?download=true) |
-| 9 | ChangeFormer | 0.7488 | 0.7887 | 0.9795 | 0.7505 | 0.7472 | 0.6525 | — | 21.18 | 4.03 | 157.64 | 29.75 | [Download](https://huggingface.co/dineth18/CD-Models/resolve/main/results/changeformer/levir_cd_test_as_val/checkpoints/best_model.pth?download=true) |
-| 10 | DSAMNet | 0.7444 | 0.7850 | 0.9779 | 0.7897 | 0.7040 | 0.5370 | — | 75.39 | 2.90 | 37.84 | 16.95 | [Download](https://huggingface.co/dineth18/CD-Models/resolve/main/results/dsamnet/levir_cd_test_as_val/checkpoints/best_model.pth?download=true) |
-| 11 | IFNet | 0.7410 | 0.7833 | 0.9786 | 0.7510 | 0.7313 | 0.3489 | — | 82.26 | 6.08 | 105.44 | 50.71 | [Download](https://huggingface.co/dineth18/CD-Models/resolve/main/results/ifnet/levir_cd_test_as_val/checkpoints/best_model.pth?download=true) |
-| 12 | FC-EF | 0.7319 | 0.7766 | 0.9768 | 0.7785 | 0.6905 | 0.6497 | — | 3.58 | 0.74 | 457.44 | 1.35 | [Download](https://huggingface.co/dineth18/CD-Models/resolve/main/results/fc_ef/levir_cd_test_as_val/checkpoints/best_model.pth?download=true) |
-| 13 | FC-Siam-diff | 0.7196 | 0.7667 | 0.9725 | 0.8650 | 0.6160 | 0.5447 | — | 4.73 | 1.06 | 262.80 | 1.35 | [Download](https://huggingface.co/dineth18/CD-Models/resolve/main/results/fc_siam_diff/levir_cd_test_as_val/checkpoints/best_model.pth?download=true) |
+| 1 | ChangeMamba | 0.8753 | 0.8840 | 0.9901 | 0.8572 | 0.8942 | 0.8307 | — | 28.70 | 1.93 | 76.82 | 54.00 | [Download](https://huggingface.co/dineth18/CD-Models/resolve/main/results/changemamba/levir_val_as_test/checkpoints/best_model.pth?download=true) |
+| 2 | CGNet | 0.8138 | 0.8356 | 0.9856 | 0.7746 | 0.8572 | 0.7651 | — | 82.23 | 6.01 | 44.01 | 38.99 | [Download](https://huggingface.co/dineth18/CD-Models/resolve/main/results/cgnet/levir_cd_test_as_val/checkpoints/best_model.pth?download=true) |
+| 3 | Siam-NestedUNet | 0.8046 | 0.8282 | 0.9839 | 0.8129 | 0.7965 | 0.7386 | — | — | 2.67 | 108.71 | — | [Download](https://huggingface.co/dineth18/CD-Models/resolve/main/results/siam_nestedunet/levir_cd_test_as_val/checkpoints/best_model.pth?download=true) |
+| 4 | SChanger | 0.8007 | 0.8253 | 0.9836 | 0.8112 | 0.7906 | 0.7364 | — | — | 2.45 | 48.96 | — | [Download](https://huggingface.co/dineth18/CD-Models/resolve/main/results/schanger/levir_cd_test_as_val/checkpoints/best_model.pth?download=true) |
+| 5 | BIT-CD | 0.7996 | 0.8247 | 0.9838 | 0.7946 | 0.8047 | 0.7143 | — | 10.63 | 2.27 | 236.60 | 12.40 | [Download](https://huggingface.co/dineth18/CD-Models/resolve/main/results/bit_cd/levir_cd_test_as_val/checkpoints/best_model.pth?download=true) |
+| 6 | HANet | 0.7841 | 0.8131 | 0.9819 | 0.8064 | 0.7630 | 0.7043 | — | — | 5.04 | 84.44 | — | [Download](https://huggingface.co/dineth18/CD-Models/resolve/main/results/hanet/levir_cd_test_as_val/checkpoints/best_model.pth?download=true) |
+| 7 | BiFA | 0.7772 | 0.8084 | 0.9818 | 0.7785 | 0.7760 | 0.6581 | — | 53.00 | 15.14 | 30.38 | 9.87 | [Download](https://huggingface.co/dineth18/CD-Models/resolve/main/results/bifa/levir_cd_test_as_val/checkpoints/best_model.pth?download=true) |
+| 8 | FC-Siam-conc | 0.7650 | 0.7999 | 0.9809 | 0.7623 | 0.7678 | 0.7175 | — | 5.33 | 1.02 | 308.81 | 1.55 | [Download](https://huggingface.co/dineth18/CD-Models/resolve/main/results/fc_siam_conc/levir_cd_test_as_val/checkpoints/best_model.pth?download=true) |
+| 9 | STANet | 0.7544 | 0.7921 | 0.9791 | 0.7865 | 0.7249 | 0.6388 | — | — | 4.45 | 174.00 | — | [Download](https://huggingface.co/dineth18/CD-Models/resolve/main/results/stanet/levir_cd_test_as_val/checkpoints/best_model.pth?download=true) |
+| 10 | ChangeFormer | 0.7488 | 0.7887 | 0.9795 | 0.7505 | 0.7472 | 0.6525 | — | 21.18 | 4.03 | 157.64 | 29.75 | [Download](https://huggingface.co/dineth18/CD-Models/resolve/main/results/changeformer/levir_cd_test_as_val/checkpoints/best_model.pth?download=true) |
+| 11 | DSAMNet | 0.7444 | 0.7850 | 0.9779 | 0.7897 | 0.7040 | 0.5370 | — | 75.39 | 2.90 | 37.84 | 16.95 | [Download](https://huggingface.co/dineth18/CD-Models/resolve/main/results/dsamnet/levir_cd_test_as_val/checkpoints/best_model.pth?download=true) |
+| 12 | IFNet | 0.7410 | 0.7833 | 0.9786 | 0.7510 | 0.7313 | 0.3489 | — | 82.26 | 6.08 | 105.44 | 50.71 | [Download](https://huggingface.co/dineth18/CD-Models/resolve/main/results/ifnet/levir_cd_test_as_val/checkpoints/best_model.pth?download=true) |
+| 13 | FC-EF | 0.7319 | 0.7766 | 0.9768 | 0.7785 | 0.6905 | 0.6497 | — | 3.58 | 0.74 | 457.44 | 1.35 | [Download](https://huggingface.co/dineth18/CD-Models/resolve/main/results/fc_ef/levir_cd_test_as_val/checkpoints/best_model.pth?download=true) |
+| 14 | FC-Siam-diff | 0.7196 | 0.7667 | 0.9725 | 0.8650 | 0.6160 | 0.5447 | — | 4.73 | 1.06 | 262.80 | 1.35 | [Download](https://huggingface.co/dineth18/CD-Models/resolve/main/results/fc_siam_diff/levir_cd_test_as_val/checkpoints/best_model.pth?download=true) |
 
 ### SYSU-CD
 
@@ -257,14 +260,6 @@ The current evaluator records BF1 but not boundary mean IoU (BmIoU), so BmIoU re
 | 7 | FC-Siam-conc | 0.7789 | 0.8062 | 0.9756 | 0.8907 | 0.6920 | 0.4733 | — | 5.33 | 1.03 | 397.86 | 1.55 | [Download](https://huggingface.co/dineth18/CD-Models/resolve/main/results/fc_siam_conc/whu_cd/checkpoints/best_model.pth?download=true) |
 | 8 | FC-Siam-diff | 0.7135 | 0.7580 | 0.9631 | 0.9540 | 0.5699 | 0.2148 | — | 4.73 | 1.09 | 567.87 | 1.35 | [Download](https://huggingface.co/dineth18/CD-Models/resolve/main/results/fc_siam_diff/whu_cd/checkpoints/best_model.pth?download=true) |
 
-### Additional checkpoint without completed test metrics
-
-This trained checkpoint is published on request, but it is not included in the F1 rankings because its `metrics_test.json` evaluation has not completed. The protocol is LEVIR-CD+ validation-as-test, not the test-as-validation benchmark above.
-
-| Dataset | Model | Evaluation status | Download | Size |
-| --- | --- | --- | --- | --- |
-| LEVIR-CD+ (val-as-test protocol) | ChangeMamba | Metrics pending | [Download](https://huggingface.co/dineth18/CD-Models/resolve/main/results/changemamba/levir_val_as_test/checkpoints/best_model.pth?download=true) | 619.4 MiB |
-
 ### Datasets without a released checkpoint
 
 WildFire-S2, KATE-CD-256, the standard LEVIR-CD+ configuration, and Custom-CD currently have no eligible completed test result. Their validation-only or incomplete checkpoints are intentionally not uploaded.

results/changemamba/levir_val_as_test/metrics_test.json

@@ -0,0 +1,76 @@
+{
+  "adapter": {
+    "checkpoint_format": "ChangeMamba training checkpoint with model key, state_dict key, or raw state dict",
+    "final_output_for_metrics": "raw two-channel logits",
+    "input_format": "two tensors (A, B), each [B, 3, H, W]",
+    "model_class_path": "model_repos/ChangeMamba/changedetection/models/ChangeMambaBCD.py:ChangeMambaBCD",
+    "notes": "ChangeMambaBCD with VMamba backbone; upstream training checkpoint stores weights under the model key.",
+    "output_format": "two-channel logits [B, 2, H, W]"
+  },
+  "bf1": 0.8306516149038876,
+  "boundary_precision": 0.8341108230812149,
+  "boundary_recall": 0.8272209900665211,
+  "boundary_tolerance": 2,
+  "checkpoint": "/new-home/buddhiw/CD-Models/results/changemamba/levir_val_as_test/checkpoints/LEVIR/MambaBCD_Small_1783799387.0019937/best_model.pth",
+  "dataset": "levir_val_as_test",
+  "f1": 0.875312539142707,
+  "f1_0": 0.9948183398928466,
+  "flops": 28704817152.0,
+  "flops_g": 28.704817152,
+  "flops_input_shape": [
+    [
+      1,
+      3,
+      256,
+      256
+    ],
+    [
+      1,
+      3,
+      256,
+      256
+    ]
+  ],
+  "flops_library": "thop",
+  "fn": 2123436.0,
+  "fp": 1507299.0,
+  "fps": 76.81665317190178,
+  "fps_end_to_end": 6.080985465348349,
+  "fps_model_only": 76.81665317190178,
+  "gpu_index": 0,
+  "gpu_mem_allocated_peak_gb": 1.4883003234863281,
+  "gpu_mem_reserved_peak_gb": 1.93359375,
+  "gpu_mem_used_max_gb": 2.63427734375,
+  "gpu_mem_used_mean_gb": 2.6313484073693827,
+  "gpu_name": "NVIDIA GeForce RTX 3090 Ti",
+  "gpu_profiling_error": null,
+  "gpu_util_max": 54.0,
+  "gpu_util_mean": 7.914405010438413,
+  "gpu_util_min": 0.0,
+  "iou": 0.7782718119422073,
+  "iou_0": 0.9896901120691365,
+  "kappa": 0.8701331917825822,
+  "miou": 0.8839809620056719,
+  "model": "changemamba",
+  "num_timed_images": 5552,
+  "oa": 0.9900501760943182,
+  "params": 53999618.0,
+  "params_m": 53.999618,
+  "precision": 0.894234094844145,
+  "prediction_dir": "/new-home/buddhiw/CD-Models/results/changemamba/levir_val_as_test/predictions/test",
+  "profiling_errors": [],
+  "recall": 0.8571751425197601,
+  "split": "test",
+  "status": "complete",
+  "test_num_samples": 5568,
+  "threshold": 0.5,
+  "timestamp": "2026-07-23T06:31:43.486999+00:00",
+  "timing_device": "cuda:0",
+  "tn": 348529737.0,
+  "tp": 12743976.0,
+  "trainable_params": 53999618.0,
+  "trainable_params_m": 53.999618,
+  "visual_dir": "/new-home/buddhiw/CD-Models/results/changemamba/levir_val_as_test/visuals/selected_20",
+  "visual_sample_manifest": "/new-home/buddhiw/CD-Models/results/qualitative_samples/levir_val_as_test/sample_manifest.json",
+  "warmup_batches": 2
+}

tools/generate_hf_model_card.py

@@ -14,11 +14,14 @@ REPO_ID = "dineth18/CD-Models"
 
 DATASET_LABELS = {
     "dsifn_cd": "DSIFN-CD",
-    "levir_cd_test_as_val": "LEVIR-CD+ (test-as-val protocol)",
+    "levir_cd_test_as_val": "LEVIR-CD+ comparison",
     "sysu_cd": "SYSU-CD",
     "whu_cd": "WHU-CD",
 }
 DATASET_ORDER = tuple(DATASET_LABELS)
+DATASET_ALIASES = {
+    "levir_val_as_test": "levir_cd_test_as_val",
+}
 MODEL_LABELS = {
     "bifa": "BiFA",
     "bit_cd": "BIT-CD",
@@ -42,7 +45,8 @@ def load_rows() -> dict[str, list[dict[str, Any]]]:
     for path in sorted(RESULTS.glob("*/*/metrics_test.json")):
         with path.open(encoding="utf-8") as handle:
             row = json.load(handle)
-        dataset = str(row.get("dataset") or path.parent.name)
+        source_dataset = str(row.get("dataset") or path.parent.name)
+        dataset = DATASET_ALIASES.get(source_dataset, source_dataset)
         if dataset not in DATASET_LABELS:
             continue
         if row.get("split") != "test" or row.get("status") != "complete":
@@ -51,6 +55,7 @@ def load_rows() -> dict[str, list[dict[str, Any]]]:
             continue
         row["_path"] = path
         row["_model"] = str(row.get("model") or path.parents[1].name)
+        row["_source_dataset"] = source_dataset
         grouped[dataset].append(row)
     for rows in grouped.values():
         rows.sort(key=lambda row: (-float(row["f1"]), row["_model"]))
@@ -113,12 +118,14 @@ def build_results() -> tuple[str, list[Path]]:
     lines = [
         "## Results and released checkpoints",
         "",
-        "Only completed `metrics_test.json` evaluations are eligible for rankings. Rankings are computed per dataset by test-set F1 (descending), and the canonical `best_model.pth` checkpoint for every ranked model is released. Separately requested checkpoint-only artifacts are listed below without fabricated metrics or a rank.",
+        "Only completed `metrics_test.json` evaluations are eligible for rankings. Rankings are computed per dataset by test-set F1 (descending), and the canonical `best_model.pth` checkpoint for every ranked model is released.",
         "",
         "All accuracy values are fractions. GPU memory is the PyTorch peak reserved memory for inference, with peak allocated memory used only when reserved memory is absent. FPS is model-only throughput. A dash means the evaluator did not record that metric; values are never estimated.",
         "",
         "The current evaluator records BF1 but not boundary mean IoU (BmIoU), so BmIoU remains explicitly unavailable rather than being inferred from BF1. Boundary IoU and boundary F1 are distinct measures.",
         "",
+        "The LEVIR-CD+ comparison combines the project-compatible `levir_cd_test_as_val` and `levir_val_as_test` result records. The ChangeMamba row uses the completed `levir_val_as_test` evaluation over 5,568 samples.",
+        "",
         "### Dataset summary",
         "",
     ]
@@ -189,29 +196,6 @@ def build_results() -> tuple[str, list[Path]]:
         else:
             lines.append("No completed test-set result is available, so no checkpoint is published.")
 
-    pending_checkpoint = RESULTS / "changemamba" / "levir_val_as_test" / "checkpoints" / "best_model.pth"
-    if pending_checkpoint.is_file():
-        checkpoints.append(pending_checkpoint)
-        lines.extend(
-            [
-                "",
-                "### Additional checkpoint without completed test metrics",
-                "",
-                "This trained checkpoint is published on request, but it is not included in the F1 rankings because its `metrics_test.json` evaluation has not completed. The protocol is LEVIR-CD+ validation-as-test, not the test-as-validation benchmark above.",
-                "",
-                *markdown_table(
-                    ["Dataset", "Model", "Evaluation status", "Download", "Size"],
-                    [[
-                        "LEVIR-CD+ (val-as-test protocol)",
-                        "ChangeMamba",
-                        "Metrics pending",
-                        f"[Download]({download_url(pending_checkpoint)})",
-                        f"{pending_checkpoint.stat().st_size / (1024 ** 2):.1f} MiB",
-                    ]],
-                ),
-            ]
-        )
-
     lines.extend(
         [
             "",