card: what the two artefacts are, which gates they pass, and on what
Browse files
README.md
ADDED
|
@@ -0,0 +1,251 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: agpl-3.0
|
| 3 |
+
library_name: onnx
|
| 4 |
+
pipeline_tag: object-detection
|
| 5 |
+
tags:
|
| 6 |
+
- waste
|
| 7 |
+
- recycling
|
| 8 |
+
- smart-bin-recognition
|
| 9 |
+
- yolo11
|
| 10 |
+
- onnx
|
| 11 |
+
- int8
|
| 12 |
+
base_model:
|
| 13 |
+
- Ultralytics/YOLO11
|
| 14 |
+
datasets:
|
| 15 |
+
- arudaev/smart-bin-detect
|
| 16 |
+
---
|
| 17 |
+
|
| 18 |
+
# Smart Bin Recognition β validator and identifier
|
| 19 |
+
|
| 20 |
+
Two ONNX graphs behind a web app that answers *"what is this bin, and what goes
|
| 21 |
+
in it?"* from a phone camera. They run **server-side**; nothing here is
|
| 22 |
+
downloaded to a device.
|
| 23 |
+
|
| 24 |
+
**One of them may ship. The other may not, and the service refuses to start on
|
| 25 |
+
it.** Both are published because an artefact that failed its gate is evidence,
|
| 26 |
+
not waste, and hiding it would make the next person repeat the measurement.
|
| 27 |
+
|
| 28 |
+
| | **validator** | **identifier** |
|
| 29 |
+
|---|---|---|
|
| 30 |
+
| question | *is there a bin?* | *which bin?* |
|
| 31 |
+
| file | `v1/validator-v1.onnx` | `v1/identifier-v1.onnx` |
|
| 32 |
+
| architecture | YOLO11n detection | YOLO11s-cls |
|
| 33 |
+
| input | 448 Γ 448, NCHW, static batch 1 | 320 Γ 320, NCHW, **dynamic batch** |
|
| 34 |
+
| weights | int8 (U8S8, per-channel) | int8 (U8S8, per-channel) |
|
| 35 |
+
| size | 3.15 MB | 5.81 MB |
|
| 36 |
+
| **`may_ship`** | **`false`** | **`true`** |
|
| 37 |
+
|
| 38 |
+
**Every number below carries the split it was measured on and the hardware it
|
| 39 |
+
was measured on.** A figure without both is not quotable, here or in the project
|
| 40 |
+
that produced it.
|
| 41 |
+
|
| 42 |
+
---
|
| 43 |
+
|
| 44 |
+
## The class list, and its order
|
| 45 |
+
|
| 46 |
+
**Read the sidecar, not this table.** `classes` in each `*.json` is the
|
| 47 |
+
authoritative order and the service reads it from there. The order **is** the
|
| 48 |
+
ONNX output index; reordering it silently invalidates every deployed model, and
|
| 49 |
+
the project pins it with a test.
|
| 50 |
+
|
| 51 |
+
**validator** β 1 class:
|
| 52 |
+
|
| 53 |
+
| index | id |
|
| 54 |
+
|---:|---|
|
| 55 |
+
| 0 | `bin` |
|
| 56 |
+
|
| 57 |
+
**identifier** β 3 classes, **alphabetical**, which is the order the training
|
| 58 |
+
framework emitted and therefore the order the head produces. It is *not* the
|
| 59 |
+
order the taxonomy lists them in:
|
| 60 |
+
|
| 61 |
+
| index | id | what it is |
|
| 62 |
+
|---:|---|---|
|
| 63 |
+
| 0 | `igloo` | a glass/bottle bank |
|
| 64 |
+
| 1 | `wheelie_large` | a 660β1100 L four-wheeled container |
|
| 65 |
+
| 2 | `wheelie_small` | a 120β240 L two-wheeled household bin |
|
| 66 |
+
|
| 67 |
+
These are **form factors β shapes, not meanings**. The model never predicts a
|
| 68 |
+
waste stream. Meaning comes from a per-jurisdiction region pack (a JSON file)
|
| 69 |
+
that maps *(form factor, colour, region)* to a stream, so adding a country is a
|
| 70 |
+
data change rather than a retrain.
|
| 71 |
+
|
| 72 |
+
---
|
| 73 |
+
|
| 74 |
+
## The identifier β passes every gate
|
| 75 |
+
|
| 76 |
+
| gate | budget | measured | split / hardware |
|
| 77 |
+
|---|---:|---:|---|
|
| 78 |
+
| int8 accuracy cost | β€ 0.02 top-1 | **0.0000** | `test`, Kaggle T4 training run |
|
| 79 |
+
| median latency | β€ 25 ms / crop | **9.921 ms** | GCE `n2-standard-4`, 2 of 4 vCPU pinned, Intel Cascade Lake, `europe-west3-a`, onnxruntime 1.29.0, `representative: true` |
|
| 80 |
+
| p95 latency | β | 11.409 ms | same |
|
| 81 |
+
| hardware named | required | yes | same |
|
| 82 |
+
|
| 83 |
+
`gate_result: {"failures": [], "unmeasured": [], "may_ship": true}`.
|
| 84 |
+
|
| 85 |
+
### Its evidence is thin, and the card says so
|
| 86 |
+
|
| 87 |
+
**`test` top-1 is 1.0000 on 47 crops.**
|
| 88 |
+
|
| 89 |
+
| class | crops in `test` | capture clusters in `test` |
|
| 90 |
+
|---|---:|---:|
|
| 91 |
+
| `wheelie_small` | 25 | 9 |
|
| 92 |
+
| `wheelie_large` | 19 | 8 |
|
| 93 |
+
| **`igloo`** | **3** | **2** |
|
| 94 |
+
|
| 95 |
+
- 47 correct out of 47 is **not** certainty. By the rule of three the 95 % lower
|
| 96 |
+
bound is **0.936**.
|
| 97 |
+
- **`igloo`'s contribution is three crops from two scenes.** Its per-class number
|
| 98 |
+
is noisy and must never be quoted clean.
|
| 99 |
+
- **The better estimate is 0.9834** β out-of-fold accuracy over all 403 crops
|
| 100 |
+
under `GroupKFold` on capture cluster, against a 0.6823 majority-class
|
| 101 |
+
baseline. A random split of these crops measures memorisation: 403 crops come
|
| 102 |
+
from 100 capture clusters, and the largest holds 18 photographs of one bin.
|
| 103 |
+
|
| 104 |
+
**The task is close to saturated**, so this measurement can distinguish *"int8
|
| 105 |
+
and fp32 differ by at most one crop"* from *"they differ by two or more"* and not
|
| 106 |
+
much finer. The gate passed honestly; it did not pass by a margin the data can
|
| 107 |
+
resolve.
|
| 108 |
+
|
| 109 |
+
---
|
| 110 |
+
|
| 111 |
+
## The validator β trained, real, and it cannot ship
|
| 112 |
+
|
| 113 |
+
It is not a failed training run. It is a working detector that **int8
|
| 114 |
+
quantisation destroys**.
|
| 115 |
+
|
| 116 |
+
| gate | budget | measured | split / hardware |
|
| 117 |
+
|---|---:|---:|---|
|
| 118 |
+
| **int8 accuracy cost** | β€ 0.02 mAP@0.5 | **0.727** β | `test` |
|
| 119 |
+
| median latency | β€ 50 ms | 18.252 ms β
| GCE `n2-standard-4`, Cascade Lake, 2 vCPU pinned, `representative: true` |
|
| 120 |
+
| p95 latency | β | 21.345 ms | same |
|
| 121 |
+
|
| 122 |
+
| | value | split |
|
| 123 |
+
|---|---:|---|
|
| 124 |
+
| mAP@0.5, PyTorch fp32 reference | **0.7524** | `test` |
|
| 125 |
+
| mAP@0.5, the int8 graph published here | **0.025** | `test` |
|
| 126 |
+
| specificity on background frames | 0.9793 | 2 662 hard negatives |
|
| 127 |
+
|
| 128 |
+
`gate_result: {"failures": ["int8 quantisation cost 0.727 map50 (max 0.02)"], "may_ship": false}`.
|
| 129 |
+
|
| 130 |
+
**The service loads the validator unconditionally and refuses to start on an
|
| 131 |
+
artefact whose sidecar says `may_ship: false`. So nothing is deployed, and that
|
| 132 |
+
is correct rather than pending.**
|
| 133 |
+
|
| 134 |
+
### What is known about why
|
| 135 |
+
|
| 136 |
+
- **Quantising the detection head is what collapses it.** Leaving `/model.23/`
|
| 137 |
+
in fp32 takes the graph from 0.015 to **0.7481** on `val` β a fifty-fold
|
| 138 |
+
recovery β for about +5.7 ms and +1.2 MB on an x86 proxy.
|
| 139 |
+
- **It does not follow that nothing else matters.** That graph is still quantised
|
| 140 |
+
everywhere else and still loses **0.0252** against a 0.02 budget. It misses by
|
| 141 |
+
**0.0052**, and the residual is unattributed.
|
| 142 |
+
- The three remedies onnxruntime names for this failure mode β S8S8,
|
| 143 |
+
`reduce_range`, U8U8 β all stay at collapse, as does per-tensor. This is *not*
|
| 144 |
+
the x86 saturation case it resembles.
|
| 145 |
+
- **Post-training int8 over the whole graph is not viable for this
|
| 146 |
+
architecture.** Any future YOLO11 detection export here starts from
|
| 147 |
+
`exclude_head=True`.
|
| 148 |
+
|
| 149 |
+
### fp32 is the open route, and it was measured
|
| 150 |
+
|
| 151 |
+
Measured 2026-08-22, both formats on **one** Cascade Lake instance with the arms
|
| 152 |
+
alternated:
|
| 153 |
+
|
| 154 |
+
| | int8 | fp32 |
|
| 155 |
+
|---|---:|---:|
|
| 156 |
+
| validator p50 | 17.921 ms | **24.605 ms** |
|
| 157 |
+
| against a 50 ms budget | passes | **passes** |
|
| 158 |
+
| concurrent scanners @ 1 bin | 5 | **4** |
|
| 159 |
+
| accuracy cost | β0.727 mAP | 0.0 |
|
| 160 |
+
|
| 161 |
+
**An fp32 validator meets the latency budget with 25 ms to spare and costs one
|
| 162 |
+
concurrent scanner.** The ship gate currently refuses unquantised artefacts
|
| 163 |
+
outright, and its stated reason β *"it will not meet the latency budget"* β is
|
| 164 |
+
false on this architecture at this input size. Splitting that gate into
|
| 165 |
+
per-format profiles is staged for the maintainer's decision and is **not**
|
| 166 |
+
merged. **No fp32 artefact is published in this repository.**
|
| 167 |
+
|
| 168 |
+
---
|
| 169 |
+
|
| 170 |
+
## Serving contract
|
| 171 |
+
|
| 172 |
+
Both graphs, from the sidecars:
|
| 173 |
+
|
| 174 |
+
- **Normalisation:** scale `1/255`, mean `[0,0,0]`, std `[1,1,1]`. No ImageNet
|
| 175 |
+
statistics.
|
| 176 |
+
- **Input name:** `images`. Layout `NCHW`.
|
| 177 |
+
- **NMS is not in the graph.** Apply it yourself: `iou 0.45`, `score 0.35`.
|
| 178 |
+
- **The identifier accepts a batch > 1 and the validator does not.** A frame can
|
| 179 |
+
hold several bins β a bank of six is a normal input, not an edge case β so the
|
| 180 |
+
crops go through one call. The validator sees exactly one frame, every time.
|
| 181 |
+
|
| 182 |
+
Read `imgsz`, `classes`, `normalisation` and `nms` **from the sidecar** rather
|
| 183 |
+
than from this card. The service does, which is why a model swap needs no code
|
| 184 |
+
change.
|
| 185 |
+
|
| 186 |
+
---
|
| 187 |
+
|
| 188 |
+
## Coverage gap β six of ten form factors have no data at all
|
| 189 |
+
|
| 190 |
+
The taxonomy defines ten form factors. The identifier was trained on **three**.
|
| 191 |
+
|
| 192 |
+
| form factor | training crops | status |
|
| 193 |
+
|---|---:|---|
|
| 194 |
+
| `wheelie_small` | 247 | trained |
|
| 195 |
+
| `wheelie_large` | 115 | trained |
|
| 196 |
+
| `igloo` | 40 | trained |
|
| 197 |
+
| `street_basket` | **1** | **dropped** β one crop in one capture cluster cannot be split across train/val/test, so it can be neither trained nor evaluated |
|
| 198 |
+
| `underground`, `textile_bank`, `sack`, `crate`, `wall_unit`, `container_bank` | **0** | **no data** |
|
| 199 |
+
|
| 200 |
+
**Those six keep their ids.** An id with no training data is a coverage gap, not
|
| 201 |
+
a deletion. Everything the identifier has never seen resolves to `unknown`, which
|
| 202 |
+
is a designed state with a real interface, and the honest answer.
|
| 203 |
+
|
| 204 |
+
**Open Images cannot close this gap.** A frozen survey of 384 boxes found it to
|
| 205 |
+
be a street-litter corpus β 35 % `street_basket`, and **zero** `underground`,
|
| 206 |
+
`textile_bank` or `wall_unit`.
|
| 207 |
+
|
| 208 |
+
## There is no geographic holdout
|
| 209 |
+
|
| 210 |
+
Every training frame comes from **one city in one week** (Deggendorf, Bavaria),
|
| 211 |
+
plus Open Images frames whose `region_id` is `unknown` because the source does
|
| 212 |
+
not record where a photograph was taken. **No split in this data answers "does it
|
| 213 |
+
work in another city."** The targets `min_recall_heldout_city` and
|
| 214 |
+
`min_formfactor_acc_heldout_city` are reported as **unmeasurable**, not as
|
| 215 |
+
passed.
|
| 216 |
+
|
| 217 |
+
Do not quote the predecessor project's 95.2 % mAP as a baseline for these models.
|
| 218 |
+
It was measured on a random split of one week's photographs in one city.
|
| 219 |
+
|
| 220 |
+
---
|
| 221 |
+
|
| 222 |
+
## What this must not be used for
|
| 223 |
+
|
| 224 |
+
- **Do not use the validator as published.** It scores 0.025 mAP@0.5. It is here
|
| 225 |
+
as the evidence behind a `may_ship: false`, not as a detector.
|
| 226 |
+
- **Do not derive disposal advice from these models alone.** They output a shape.
|
| 227 |
+
Being confidently wrong about what goes in which bin is this product's worst
|
| 228 |
+
failure mode, and the mapping from shape to rule lives in a per-region pack
|
| 229 |
+
that is versioned, sourced and separately reviewed.
|
| 230 |
+
- **Do not deploy this outside Deggendorf and expect the numbers above.** They
|
| 231 |
+
are in-distribution numbers on one city, and the card says so twice because it
|
| 232 |
+
is the most likely way to be wrong with them.
|
| 233 |
+
- **Do not treat a colour attribute as measured.** As of 2026-08-22 colour
|
| 234 |
+
agreement against hand-labelled ground truth is **0.5625** on bin bodies, and
|
| 235 |
+
lid colour is **not measured at all** β an upper-band sampler scored 0.1966 and
|
| 236 |
+
was deliberately left unwired. Any rule matching on `lid_color` is unreachable.
|
| 237 |
+
- **Do not fine-tune on user-contributed images without human label review.**
|
| 238 |
+
High-agreement machine labels may auto-accept over a public corpus; never over
|
| 239 |
+
a frame a user contributed, and never for a form factor that has no data yet.
|
| 240 |
+
|
| 241 |
+
## Provenance
|
| 242 |
+
|
| 243 |
+
Trained on [`arudaev/smart-bin-detect`](https://huggingface.co/datasets/arudaev/smart-bin-detect)
|
| 244 |
+
at revision `8666aa23` β 18 954 frames: 370 legacy, 1 110 Open Images bins,
|
| 245 |
+
17 474 background. The identifier additionally uses a private crop pool whose
|
| 246 |
+
403 crops were **all** adjudicated by a human, blind, in a single pass; against
|
| 247 |
+
that pass the pool's shipped streamβshape proposals were wrong on **116 of
|
| 248 |
+
403 (28.8 %)**.
|
| 249 |
+
|
| 250 |
+
Splits are **group-aware on capture cluster**, so two photographs of the same
|
| 251 |
+
bin can never straddle a split.
|