arudaev commited on
Commit
3bdfea4
Β·
verified Β·
1 Parent(s): 78c76fc

card: what the two artefacts are, which gates they pass, and on what

Browse files
Files changed (1) hide show
  1. README.md +251 -0
README.md ADDED
@@ -0,0 +1,251 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: agpl-3.0
3
+ library_name: onnx
4
+ pipeline_tag: object-detection
5
+ tags:
6
+ - waste
7
+ - recycling
8
+ - smart-bin-recognition
9
+ - yolo11
10
+ - onnx
11
+ - int8
12
+ base_model:
13
+ - Ultralytics/YOLO11
14
+ datasets:
15
+ - arudaev/smart-bin-detect
16
+ ---
17
+
18
+ # Smart Bin Recognition β€” validator and identifier
19
+
20
+ Two ONNX graphs behind a web app that answers *"what is this bin, and what goes
21
+ in it?"* from a phone camera. They run **server-side**; nothing here is
22
+ downloaded to a device.
23
+
24
+ **One of them may ship. The other may not, and the service refuses to start on
25
+ it.** Both are published because an artefact that failed its gate is evidence,
26
+ not waste, and hiding it would make the next person repeat the measurement.
27
+
28
+ | | **validator** | **identifier** |
29
+ |---|---|---|
30
+ | question | *is there a bin?* | *which bin?* |
31
+ | file | `v1/validator-v1.onnx` | `v1/identifier-v1.onnx` |
32
+ | architecture | YOLO11n detection | YOLO11s-cls |
33
+ | input | 448 Γ— 448, NCHW, static batch 1 | 320 Γ— 320, NCHW, **dynamic batch** |
34
+ | weights | int8 (U8S8, per-channel) | int8 (U8S8, per-channel) |
35
+ | size | 3.15 MB | 5.81 MB |
36
+ | **`may_ship`** | **`false`** | **`true`** |
37
+
38
+ **Every number below carries the split it was measured on and the hardware it
39
+ was measured on.** A figure without both is not quotable, here or in the project
40
+ that produced it.
41
+
42
+ ---
43
+
44
+ ## The class list, and its order
45
+
46
+ **Read the sidecar, not this table.** `classes` in each `*.json` is the
47
+ authoritative order and the service reads it from there. The order **is** the
48
+ ONNX output index; reordering it silently invalidates every deployed model, and
49
+ the project pins it with a test.
50
+
51
+ **validator** β€” 1 class:
52
+
53
+ | index | id |
54
+ |---:|---|
55
+ | 0 | `bin` |
56
+
57
+ **identifier** β€” 3 classes, **alphabetical**, which is the order the training
58
+ framework emitted and therefore the order the head produces. It is *not* the
59
+ order the taxonomy lists them in:
60
+
61
+ | index | id | what it is |
62
+ |---:|---|---|
63
+ | 0 | `igloo` | a glass/bottle bank |
64
+ | 1 | `wheelie_large` | a 660–1100 L four-wheeled container |
65
+ | 2 | `wheelie_small` | a 120–240 L two-wheeled household bin |
66
+
67
+ These are **form factors β€” shapes, not meanings**. The model never predicts a
68
+ waste stream. Meaning comes from a per-jurisdiction region pack (a JSON file)
69
+ that maps *(form factor, colour, region)* to a stream, so adding a country is a
70
+ data change rather than a retrain.
71
+
72
+ ---
73
+
74
+ ## The identifier β€” passes every gate
75
+
76
+ | gate | budget | measured | split / hardware |
77
+ |---|---:|---:|---|
78
+ | int8 accuracy cost | ≀ 0.02 top-1 | **0.0000** | `test`, Kaggle T4 training run |
79
+ | median latency | ≀ 25 ms / crop | **9.921 ms** | GCE `n2-standard-4`, 2 of 4 vCPU pinned, Intel Cascade Lake, `europe-west3-a`, onnxruntime 1.29.0, `representative: true` |
80
+ | p95 latency | – | 11.409 ms | same |
81
+ | hardware named | required | yes | same |
82
+
83
+ `gate_result: {"failures": [], "unmeasured": [], "may_ship": true}`.
84
+
85
+ ### Its evidence is thin, and the card says so
86
+
87
+ **`test` top-1 is 1.0000 on 47 crops.**
88
+
89
+ | class | crops in `test` | capture clusters in `test` |
90
+ |---|---:|---:|
91
+ | `wheelie_small` | 25 | 9 |
92
+ | `wheelie_large` | 19 | 8 |
93
+ | **`igloo`** | **3** | **2** |
94
+
95
+ - 47 correct out of 47 is **not** certainty. By the rule of three the 95 % lower
96
+ bound is **0.936**.
97
+ - **`igloo`'s contribution is three crops from two scenes.** Its per-class number
98
+ is noisy and must never be quoted clean.
99
+ - **The better estimate is 0.9834** β€” out-of-fold accuracy over all 403 crops
100
+ under `GroupKFold` on capture cluster, against a 0.6823 majority-class
101
+ baseline. A random split of these crops measures memorisation: 403 crops come
102
+ from 100 capture clusters, and the largest holds 18 photographs of one bin.
103
+
104
+ **The task is close to saturated**, so this measurement can distinguish *"int8
105
+ and fp32 differ by at most one crop"* from *"they differ by two or more"* and not
106
+ much finer. The gate passed honestly; it did not pass by a margin the data can
107
+ resolve.
108
+
109
+ ---
110
+
111
+ ## The validator β€” trained, real, and it cannot ship
112
+
113
+ It is not a failed training run. It is a working detector that **int8
114
+ quantisation destroys**.
115
+
116
+ | gate | budget | measured | split / hardware |
117
+ |---|---:|---:|---|
118
+ | **int8 accuracy cost** | ≀ 0.02 mAP@0.5 | **0.727** ❌ | `test` |
119
+ | median latency | ≀ 50 ms | 18.252 ms βœ… | GCE `n2-standard-4`, Cascade Lake, 2 vCPU pinned, `representative: true` |
120
+ | p95 latency | – | 21.345 ms | same |
121
+
122
+ | | value | split |
123
+ |---|---:|---|
124
+ | mAP@0.5, PyTorch fp32 reference | **0.7524** | `test` |
125
+ | mAP@0.5, the int8 graph published here | **0.025** | `test` |
126
+ | specificity on background frames | 0.9793 | 2 662 hard negatives |
127
+
128
+ `gate_result: {"failures": ["int8 quantisation cost 0.727 map50 (max 0.02)"], "may_ship": false}`.
129
+
130
+ **The service loads the validator unconditionally and refuses to start on an
131
+ artefact whose sidecar says `may_ship: false`. So nothing is deployed, and that
132
+ is correct rather than pending.**
133
+
134
+ ### What is known about why
135
+
136
+ - **Quantising the detection head is what collapses it.** Leaving `/model.23/`
137
+ in fp32 takes the graph from 0.015 to **0.7481** on `val` β€” a fifty-fold
138
+ recovery β€” for about +5.7 ms and +1.2 MB on an x86 proxy.
139
+ - **It does not follow that nothing else matters.** That graph is still quantised
140
+ everywhere else and still loses **0.0252** against a 0.02 budget. It misses by
141
+ **0.0052**, and the residual is unattributed.
142
+ - The three remedies onnxruntime names for this failure mode β€” S8S8,
143
+ `reduce_range`, U8U8 β€” all stay at collapse, as does per-tensor. This is *not*
144
+ the x86 saturation case it resembles.
145
+ - **Post-training int8 over the whole graph is not viable for this
146
+ architecture.** Any future YOLO11 detection export here starts from
147
+ `exclude_head=True`.
148
+
149
+ ### fp32 is the open route, and it was measured
150
+
151
+ Measured 2026-08-22, both formats on **one** Cascade Lake instance with the arms
152
+ alternated:
153
+
154
+ | | int8 | fp32 |
155
+ |---|---:|---:|
156
+ | validator p50 | 17.921 ms | **24.605 ms** |
157
+ | against a 50 ms budget | passes | **passes** |
158
+ | concurrent scanners @ 1 bin | 5 | **4** |
159
+ | accuracy cost | βˆ’0.727 mAP | 0.0 |
160
+
161
+ **An fp32 validator meets the latency budget with 25 ms to spare and costs one
162
+ concurrent scanner.** The ship gate currently refuses unquantised artefacts
163
+ outright, and its stated reason β€” *"it will not meet the latency budget"* β€” is
164
+ false on this architecture at this input size. Splitting that gate into
165
+ per-format profiles is staged for the maintainer's decision and is **not**
166
+ merged. **No fp32 artefact is published in this repository.**
167
+
168
+ ---
169
+
170
+ ## Serving contract
171
+
172
+ Both graphs, from the sidecars:
173
+
174
+ - **Normalisation:** scale `1/255`, mean `[0,0,0]`, std `[1,1,1]`. No ImageNet
175
+ statistics.
176
+ - **Input name:** `images`. Layout `NCHW`.
177
+ - **NMS is not in the graph.** Apply it yourself: `iou 0.45`, `score 0.35`.
178
+ - **The identifier accepts a batch > 1 and the validator does not.** A frame can
179
+ hold several bins β€” a bank of six is a normal input, not an edge case β€” so the
180
+ crops go through one call. The validator sees exactly one frame, every time.
181
+
182
+ Read `imgsz`, `classes`, `normalisation` and `nms` **from the sidecar** rather
183
+ than from this card. The service does, which is why a model swap needs no code
184
+ change.
185
+
186
+ ---
187
+
188
+ ## Coverage gap β€” six of ten form factors have no data at all
189
+
190
+ The taxonomy defines ten form factors. The identifier was trained on **three**.
191
+
192
+ | form factor | training crops | status |
193
+ |---|---:|---|
194
+ | `wheelie_small` | 247 | trained |
195
+ | `wheelie_large` | 115 | trained |
196
+ | `igloo` | 40 | trained |
197
+ | `street_basket` | **1** | **dropped** β€” one crop in one capture cluster cannot be split across train/val/test, so it can be neither trained nor evaluated |
198
+ | `underground`, `textile_bank`, `sack`, `crate`, `wall_unit`, `container_bank` | **0** | **no data** |
199
+
200
+ **Those six keep their ids.** An id with no training data is a coverage gap, not
201
+ a deletion. Everything the identifier has never seen resolves to `unknown`, which
202
+ is a designed state with a real interface, and the honest answer.
203
+
204
+ **Open Images cannot close this gap.** A frozen survey of 384 boxes found it to
205
+ be a street-litter corpus β€” 35 % `street_basket`, and **zero** `underground`,
206
+ `textile_bank` or `wall_unit`.
207
+
208
+ ## There is no geographic holdout
209
+
210
+ Every training frame comes from **one city in one week** (Deggendorf, Bavaria),
211
+ plus Open Images frames whose `region_id` is `unknown` because the source does
212
+ not record where a photograph was taken. **No split in this data answers "does it
213
+ work in another city."** The targets `min_recall_heldout_city` and
214
+ `min_formfactor_acc_heldout_city` are reported as **unmeasurable**, not as
215
+ passed.
216
+
217
+ Do not quote the predecessor project's 95.2 % mAP as a baseline for these models.
218
+ It was measured on a random split of one week's photographs in one city.
219
+
220
+ ---
221
+
222
+ ## What this must not be used for
223
+
224
+ - **Do not use the validator as published.** It scores 0.025 mAP@0.5. It is here
225
+ as the evidence behind a `may_ship: false`, not as a detector.
226
+ - **Do not derive disposal advice from these models alone.** They output a shape.
227
+ Being confidently wrong about what goes in which bin is this product's worst
228
+ failure mode, and the mapping from shape to rule lives in a per-region pack
229
+ that is versioned, sourced and separately reviewed.
230
+ - **Do not deploy this outside Deggendorf and expect the numbers above.** They
231
+ are in-distribution numbers on one city, and the card says so twice because it
232
+ is the most likely way to be wrong with them.
233
+ - **Do not treat a colour attribute as measured.** As of 2026-08-22 colour
234
+ agreement against hand-labelled ground truth is **0.5625** on bin bodies, and
235
+ lid colour is **not measured at all** β€” an upper-band sampler scored 0.1966 and
236
+ was deliberately left unwired. Any rule matching on `lid_color` is unreachable.
237
+ - **Do not fine-tune on user-contributed images without human label review.**
238
+ High-agreement machine labels may auto-accept over a public corpus; never over
239
+ a frame a user contributed, and never for a form factor that has no data yet.
240
+
241
+ ## Provenance
242
+
243
+ Trained on [`arudaev/smart-bin-detect`](https://huggingface.co/datasets/arudaev/smart-bin-detect)
244
+ at revision `8666aa23` β€” 18 954 frames: 370 legacy, 1 110 Open Images bins,
245
+ 17 474 background. The identifier additionally uses a private crop pool whose
246
+ 403 crops were **all** adjudicated by a human, blind, in a single pass; against
247
+ that pass the pool's shipped stream→shape proposals were wrong on **116 of
248
+ 403 (28.8 %)**.
249
+
250
+ Splits are **group-aware on capture cluster**, so two photographs of the same
251
+ bin can never straddle a split.