| --- |
| license: agpl-3.0 |
| library_name: onnx |
| pipeline_tag: object-detection |
| tags: |
| - waste |
| - recycling |
| - smart-bin-recognition |
| - yolo11 |
| - onnx |
| - int8 |
| base_model: |
| - Ultralytics/YOLO11 |
| datasets: |
| - arudaev/smart-bin-detect |
| --- |
| |
| # Smart Bin Recognition — validator and identifier |
|
|
| Two ONNX graphs behind a web app that answers *"what is this bin, and what goes |
| in it?"* from a phone camera. They run **server-side**; nothing here is |
| downloaded to a device. |
|
|
| **One of them may ship. The other may not, and the service refuses to start on |
| it.** Both are published because an artefact that failed its gate is evidence, |
| not waste, and hiding it would make the next person repeat the measurement. |
|
|
| | | **validator** | **identifier** | |
| |---|---|---| |
| | question | *is there a bin?* | *which bin?* | |
| | file | `v1/validator-v1.onnx` | `v1/identifier-v1.onnx` | |
| | architecture | YOLO11n detection | YOLO11s-cls | |
| | input | 448 × 448, NCHW, static batch 1 | 320 × 320, NCHW, **dynamic batch** | |
| | weights | int8 (U8S8, per-channel) | int8 (U8S8, per-channel) | |
| | size | 3.15 MB | 5.81 MB | |
| | **`may_ship`** | **`false`** | **`true`** | |
| |
| **Every number below carries the split it was measured on and the hardware it |
| was measured on.** A figure without both is not quotable, here or in the project |
| that produced it. |
| |
| --- |
| |
| ## The class list, and its order |
| |
| **Read the sidecar, not this table.** `classes` in each `*.json` is the |
| authoritative order and the service reads it from there. The order **is** the |
| ONNX output index; reordering it silently invalidates every deployed model, and |
| the project pins it with a test. |
| |
| **validator** — 1 class: |
| |
| | index | id | |
| |---:|---| |
| | 0 | `bin` | |
| |
| **identifier** — 3 classes, **alphabetical**, which is the order the training |
| framework emitted and therefore the order the head produces. It is *not* the |
| order the taxonomy lists them in: |
| |
| | index | id | what it is | |
| |---:|---|---| |
| | 0 | `igloo` | a glass/bottle bank | |
| | 1 | `wheelie_large` | a 660–1100 L four-wheeled container | |
| | 2 | `wheelie_small` | a 120–240 L two-wheeled household bin | |
| |
| These are **form factors — shapes, not meanings**. The model never predicts a |
| waste stream. Meaning comes from a per-jurisdiction region pack (a JSON file) |
| that maps *(form factor, colour, region)* to a stream, so adding a country is a |
| data change rather than a retrain. |
| |
| --- |
| |
| ## The identifier — passes every gate |
| |
| | gate | budget | measured | split / hardware | |
| |---|---:|---:|---| |
| | int8 accuracy cost | ≤ 0.02 top-1 | **0.0000** | `test`, Kaggle T4 training run | |
| | median latency | ≤ 25 ms / crop | **9.921 ms** | GCE `n2-standard-4`, 2 of 4 vCPU pinned, Intel Cascade Lake, `europe-west3-a`, onnxruntime 1.29.0, `representative: true` | |
| | p95 latency | – | 11.409 ms | same | |
| | hardware named | required | yes | same | |
| |
| `gate_result: {"failures": [], "unmeasured": [], "may_ship": true}`. |
| |
| ### Its evidence is thin, and the card says so |
| |
| **`test` top-1 is 1.0000 on 47 crops.** |
| |
| | class | crops in `test` | capture clusters in `test` | |
| |---|---:|---:| |
| | `wheelie_small` | 25 | 9 | |
| | `wheelie_large` | 19 | 8 | |
| | **`igloo`** | **3** | **2** | |
| |
| - 47 correct out of 47 is **not** certainty. By the rule of three the 95 % lower |
| bound is **0.936**. |
| - **`igloo`'s contribution is three crops from two scenes.** Its per-class number |
| is noisy and must never be quoted clean. |
| - **The better estimate is 0.9834** — out-of-fold accuracy over all 403 crops |
| under `GroupKFold` on capture cluster, against a 0.6823 majority-class |
| baseline. A random split of these crops measures memorisation: 403 crops come |
| from 100 capture clusters, and the largest holds 18 photographs of one bin. |
| |
| **The task is close to saturated**, so this measurement can distinguish *"int8 |
| and fp32 differ by at most one crop"* from *"they differ by two or more"* and not |
| much finer. The gate passed honestly; it did not pass by a margin the data can |
| resolve. |
| |
| --- |
| |
| ## The validator — trained, real, and it cannot ship |
| |
| It is not a failed training run. It is a working detector that **int8 |
| quantisation destroys**. |
| |
| | gate | budget | measured | split / hardware | |
| |---|---:|---:|---| |
| | **int8 accuracy cost** | ≤ 0.02 mAP@0.5 | **0.727** ❌ | `test` | |
| | median latency | ≤ 50 ms | 18.252 ms ✅ | GCE `n2-standard-4`, Cascade Lake, 2 vCPU pinned, `representative: true` | |
| | p95 latency | – | 21.345 ms | same | |
| |
| | | value | split | |
| |---|---:|---| |
| | mAP@0.5, PyTorch fp32 reference | **0.7524** | `test` | |
| | mAP@0.5, the int8 graph published here | **0.025** | `test` | |
| | specificity on background frames | 0.9793 | 2 662 hard negatives | |
| |
| `gate_result: {"failures": ["int8 quantisation cost 0.727 map50 (max 0.02)"], "may_ship": false}`. |
| |
| **The service loads the validator unconditionally and refuses to start on an |
| artefact whose sidecar says `may_ship: false`. So nothing is deployed, and that |
| is correct rather than pending.** |
|
|
| ### What is known about why |
|
|
| - **Quantising the detection head is what collapses it.** Leaving `/model.23/` |
| in fp32 takes the graph from 0.015 to **0.7481** on `val` — a fifty-fold |
| recovery — for about +5.7 ms and +1.2 MB on an x86 proxy. |
| - **It does not follow that nothing else matters.** That graph is still quantised |
| everywhere else and still loses **0.0252** against a 0.02 budget. It misses by |
| **0.0052**, and the residual is unattributed. |
| - The three remedies onnxruntime names for this failure mode — S8S8, |
| `reduce_range`, U8U8 — all stay at collapse, as does per-tensor. This is *not* |
| the x86 saturation case it resembles. |
| - **Post-training int8 over the whole graph is not viable for this |
| architecture.** Any future YOLO11 detection export here starts from |
| `exclude_head=True`. |
|
|
| ### fp32 is the open route, and it was measured |
|
|
| Measured 2026-08-22, both formats on **one** Cascade Lake instance with the arms |
| alternated: |
|
|
| | | int8 | fp32 | |
| |---|---:|---:| |
| | validator p50 | 17.921 ms | **24.605 ms** | |
| | against a 50 ms budget | passes | **passes** | |
| | concurrent scanners @ 1 bin | 5 | **4** | |
| | accuracy cost | −0.727 mAP | 0.0 | |
|
|
| **An fp32 validator meets the latency budget with 25 ms to spare and costs one |
| concurrent scanner.** The ship gate currently refuses unquantised artefacts |
| outright, and its stated reason — *"it will not meet the latency budget"* — is |
| false on this architecture at this input size. Splitting that gate into |
| per-format profiles is staged for the maintainer's decision and is **not** |
| merged. **No fp32 artefact is published in this repository.** |
|
|
| --- |
|
|
| ## Serving contract |
|
|
| Both graphs, from the sidecars: |
|
|
| - **Normalisation:** scale `1/255`, mean `[0,0,0]`, std `[1,1,1]`. No ImageNet |
| statistics. |
| - **Input name:** `images`. Layout `NCHW`. |
| - **NMS is not in the graph.** Apply it yourself: `iou 0.45`, `score 0.35`. |
| - **The identifier accepts a batch > 1 and the validator does not.** A frame can |
| hold several bins — a bank of six is a normal input, not an edge case — so the |
| crops go through one call. The validator sees exactly one frame, every time. |
|
|
| Read `imgsz`, `classes`, `normalisation` and `nms` **from the sidecar** rather |
| than from this card. The service does, which is why a model swap needs no code |
| change. |
|
|
| --- |
|
|
| ## Coverage gap — six of ten form factors have no data at all |
|
|
| The taxonomy defines ten form factors. The identifier was trained on **three**. |
|
|
| | form factor | training crops | status | |
| |---|---:|---| |
| | `wheelie_small` | 247 | trained | |
| | `wheelie_large` | 115 | trained | |
| | `igloo` | 40 | trained | |
| | `street_basket` | **1** | **dropped** — one crop in one capture cluster cannot be split across train/val/test, so it can be neither trained nor evaluated | |
| | `underground`, `textile_bank`, `sack`, `crate`, `wall_unit`, `container_bank` | **0** | **no data** | |
|
|
| **Those six keep their ids.** An id with no training data is a coverage gap, not |
| a deletion. Everything the identifier has never seen resolves to `unknown`, which |
| is a designed state with a real interface, and the honest answer. |
|
|
| **Open Images cannot close this gap.** A frozen survey of 384 boxes found it to |
| be a street-litter corpus — 35 % `street_basket`, and **zero** `underground`, |
| `textile_bank` or `wall_unit`. |
|
|
| ## There is no geographic holdout |
|
|
| Every training frame comes from **one city in one week** (Deggendorf, Bavaria), |
| plus Open Images frames whose `region_id` is `unknown` because the source does |
| not record where a photograph was taken. **No split in this data answers "does it |
| work in another city."** The targets `min_recall_heldout_city` and |
| `min_formfactor_acc_heldout_city` are reported as **unmeasurable**, not as |
| passed. |
|
|
| Do not quote the predecessor project's 95.2 % mAP as a baseline for these models. |
| It was measured on a random split of one week's photographs in one city. |
|
|
| --- |
|
|
| ## What this must not be used for |
|
|
| - **Do not use the validator as published.** It scores 0.025 mAP@0.5. It is here |
| as the evidence behind a `may_ship: false`, not as a detector. |
| - **Do not derive disposal advice from these models alone.** They output a shape. |
| Being confidently wrong about what goes in which bin is this product's worst |
| failure mode, and the mapping from shape to rule lives in a per-region pack |
| that is versioned, sourced and separately reviewed. |
| - **Do not deploy this outside Deggendorf and expect the numbers above.** They |
| are in-distribution numbers on one city, and the card says so twice because it |
| is the most likely way to be wrong with them. |
| - **Do not treat a colour attribute as measured.** As of 2026-08-22 colour |
| agreement against hand-labelled ground truth is **0.5625** on bin bodies, and |
| lid colour is **not measured at all** — an upper-band sampler scored 0.1966 and |
| was deliberately left unwired. Any rule matching on `lid_color` is unreachable. |
| - **Do not fine-tune on user-contributed images without human label review.** |
| High-agreement machine labels may auto-accept over a public corpus; never over |
| a frame a user contributed, and never for a form factor that has no data yet. |
|
|
| ## Provenance |
|
|
| Trained on [`arudaev/smart-bin-detect`](https://huggingface.co/datasets/arudaev/smart-bin-detect) |
| at revision `8666aa23` — 18 954 frames: 370 legacy, 1 110 Open Images bins, |
| 17 474 background. The identifier additionally uses a private crop pool whose |
| 403 crops were **all** adjudicated by a human, blind, in a single pass; against |
| that pass the pool's shipped stream→shape proposals were wrong on **116 of |
| 403 (28.8 %)**. |
|
|
| Splits are **group-aware on capture cluster**, so two photographs of the same |
| bin can never straddle a split. |
|
|
