v2 2026-10-15: vocab v2, F1 0.847 vs regex 0.580 (silver eval)

b132c4f verified 12 days ago

4.92 kB

---
license: apache-2.0
language:
- en
pipeline_tag: token-classification
base_model: BAAI/bge-small-en-v1.5
datasets:
- danielrosehill/inventory-spec-lines
tags:
- inventory
- product-specifications
- information-extraction
- onnx
---
# inventory-spec-extractor

A small model (33M parameters, int8 ONNX, about 34 MB) that finds structured
attributes in one line of a free-text product specification:

| input line | output spans |
|---|---|
| `Output: 12V 3A` | `12V` output_voltage, `3A` output_current |
| `Length: 2m, USB-C to USB-A` | `2m` cable_length, `USB-C` connector, `USB-A` connector |
| `Inlet: IEC C14, Outlets: 8x C13` | `IEC C14` connector, `C13` connector |
| `Plug: Israeli (Type H), Power: 2200W` | `Type H` plug_type, `2200W` power |
| `Power Source: 2x AAA batteries` | `AAA` battery_type, `batteries` power_source |
| `S/N: C02XG1ABJG5H` | `C02XG1ABJG5H` serial_number |
The model finds and labels spans; it does **not** convert units or pick enum
spellings. Do that in deterministic code: "100-240V AC" → 100–240 V, AC. The model
is built to run on CPU in the browser or on a small server through onnxruntime-web,
at about 20 ms per line.
## Labels (25 attributes, BIO)
`manufacturer`, `model_number`, `mpn`, `serial_number`, `version`, `power_source`, `input_voltage`, `input_current`, `output_voltage`, `output_current`, `rated_voltage`, `rated_current`, `power`, `battery_capacity`, `battery_type`, `connector`, `plug_type`, `cable_length`, `diameter`, `volume`, `storage_capacity`, `pack_count`, `material`, `colour`, `ip_rating`
Input text is `{product_type} | {line}` (product type may be empty). Labels sit on
the first word-piece of each word; continuation pieces extend the open span.
`labels.json` maps logit index → `O` / `B-attr` / `I-attr`.

## Evaluation

Validation split of
[`inventory-spec-lines`](https://huggingface.co/datasets/danielrosehill/inventory-spec-lines)
(1477 lines, split by item, never trained on). Scoring is span-level
exact match on (start, end, attribute). **The eval labels are silver** (Claude Haiku
5.5, not hand-checked), so these numbers measure agreement with that teacher. The
baseline is a regex extractor (key names plus number+unit patterns).

| | precision | recall | F1 |
|---|---|---|---|
| regex baseline | 0.804 | 0.454 | 0.580 |
| **this model** | 0.834 | 0.860 | **0.847** |

| attribute | n | baseline F1 | model F1 | model P | model R |
|---|---|---|---|---|---|
| `material` | 184 | 0.54 | **0.93** | 0.93 | 0.92 |
| `connector` | 140 | 0.36 | **0.84** | 0.78 | 0.90 |
| `colour` | 72 | 0.82 | **0.88** | 0.82 | 0.94 |
| `battery_type` | 26 | 0.27 | **0.77** | 0.86 | 0.69 |
| `output_voltage` | 26 | 0.67 | **0.83** | 0.95 | 0.73 |
| `power_source` | 26 | 0.00 | **0.78** | 0.80 | 0.77 |
| `plug_type` | 24 | 0.70 | **0.77** | 0.78 | 0.75 |
| `output_current` | 20 | 0.97 | **0.63** | 0.67 | 0.60 |
| `input_voltage` | 15 | 0.73 | **0.77** | 0.75 | 0.80 |
| `rated_voltage` | 15 | 0.80 | **0.97** | 0.94 | 1.00 |
| `storage_capacity` | 12 | 0.79 | **0.72** | 0.69 | 0.75 |
| `manufacturer` | 11 | 0.00 | **0.70** | 0.67 | 0.73 |
| `cable_length` | 10 | 0.63 | **0.91** | 0.83 | 1.00 |
| `volume` | 10 | 0.76 | **1.00** | 1.00 | 1.00 |
| `pack_count` | 9 | 0.18 | **0.78** | 0.64 | 1.00 |
| `power` | 8 | 0.93 | **0.80** | 0.86 | 0.75 |
| `diameter` | 6 | 0.80 | **0.77** | 0.71 | 0.83 |
| `version` | 6 | 0.00 | **0.29** | 1.00 | 0.17 |
| `battery_capacity` | 5 | 1.00 | **1.00** | 1.00 | 1.00 |
| `rated_current` | 4 | 0.80 | **0.67** | 0.60 | 0.75 |
| `input_current` | 3 | 1.00 | **0.00** | 0.00 | 0.00 |
| `model_number` | 2 | 0.00 | **1.00** | 1.00 | 1.00 |
| `ip_rating` | 1 | 1.00 | **1.00** | 1.00 | 1.00 |

## Known weaknesses

- **Currents.** `output_current`, `rated_current` and `input_current` score below
  the regex baseline. On mixed lines ("Power: 12V, 3A, 36W") the model can label an
  amp value as `power`. A unit check in the caller catches this: an amp value is
  never a power.
- **Decimals in some contexts.** "0.5A" has been seen cut to "0".
- **`version`** is weak (few examples), and "Gen 2" can come out as `model_number`.
- **`mpn`** has only 6 training examples.
- **`serial_number`** is learned only from generated serials, because real serial
  lines are removed from the public data. It handles `Serial Number:` / `S/N:` / `SN:`
  lines well; it has not been measured on real serials.
## Training
Fine-tuned from `BAAI/bge-small-en-v1.5` for 6 epochs on a Modal T4. Lines with
attributes that have fewer than 300 spans were repeated 4×, and 300 synthetic
serial lines were added. Trained 2026-10-15. Code, labelling and redaction
pipeline: `danielrosehill/Inventory-Spec-Extractor`.
## Files
`model_quantized.onnx` (int8, what to load), `tokenizer.json`, `labels.json`.
Inputs: `input_ids`, `attention_mask`, `token_type_ids` (int64 [batch, seq]).
Output: `logits` float32 [batch, seq, 51].

	---
	license: apache-2.0
	language:
	- en
	pipeline_tag: token-classification
	base_model: BAAI/bge-small-en-v1.5
	datasets:
	- danielrosehill/inventory-spec-lines
	tags:
	- inventory
	- product-specifications
	- information-extraction
	- onnx
	---

	# inventory-spec-extractor

	A small model (33M parameters, int8 ONNX, about 34 MB) that finds structured
	attributes in one line of a free-text product specification:

	\| input line \| output spans \|
	\|---\|---\|
	\| `Output: 12V 3A` \| `12V` output_voltage, `3A` output_current \|
	\| `Length: 2m, USB-C to USB-A` \| `2m` cable_length, `USB-C` connector, `USB-A` connector \|
	\| `Inlet: IEC C14, Outlets: 8x C13` \| `IEC C14` connector, `C13` connector \|
	\| `Plug: Israeli (Type H), Power: 2200W` \| `Type H` plug_type, `2200W` power \|
	\| `Power Source: 2x AAA batteries` \| `AAA` battery_type, `batteries` power_source \|
	\| `S/N: C02XG1ABJG5H` \| `C02XG1ABJG5H` serial_number \|

	The model finds and labels spans; it does not convert units or pick enum
	spellings. Do that in deterministic code: "100-240V AC" → 100–240 V, AC. The model
	is built to run on CPU in the browser or on a small server through onnxruntime-web,
	at about 20 ms per line.

	## Labels (25 attributes, BIO)

	`manufacturer`, `model_number`, `mpn`, `serial_number`, `version`, `power_source`, `input_voltage`, `input_current`, `output_voltage`, `output_current`, `rated_voltage`, `rated_current`, `power`, `battery_capacity`, `battery_type`, `connector`, `plug_type`, `cable_length`, `diameter`, `volume`, `storage_capacity`, `pack_count`, `material`, `colour`, `ip_rating`

	Input text is `{product_type} \| {line}` (product type may be empty). Labels sit on
	the first word-piece of each word; continuation pieces extend the open span.
	`labels.json` maps logit index → `O` / `B-attr` / `I-attr`.

	## Evaluation

	Validation split of
	[`inventory-spec-lines`](https://huggingface.co/datasets/danielrosehill/inventory-spec-lines)
	(1477 lines, split by item, never trained on). Scoring is span-level
	exact match on (start, end, attribute). The eval labels are silver (Claude Haiku
	5.5, not hand-checked), so these numbers measure agreement with that teacher. The
	baseline is a regex extractor (key names plus number+unit patterns).

	\| \| precision \| recall \| F1 \|
	\|---\|---\|---\|---\|
	\| regex baseline \| 0.804 \| 0.454 \| 0.580 \|
	\| this model \| 0.834 \| 0.860 \| 0.847 \|

	\| attribute \| n \| baseline F1 \| model F1 \| model P \| model R \|
	\|---\|---\|---\|---\|---\|---\|
	\| `material` \| 184 \| 0.54 \| 0.93 \| 0.93 \| 0.92 \|
	\| `connector` \| 140 \| 0.36 \| 0.84 \| 0.78 \| 0.90 \|
	\| `colour` \| 72 \| 0.82 \| 0.88 \| 0.82 \| 0.94 \|
	\| `battery_type` \| 26 \| 0.27 \| 0.77 \| 0.86 \| 0.69 \|
	\| `output_voltage` \| 26 \| 0.67 \| 0.83 \| 0.95 \| 0.73 \|
	\| `power_source` \| 26 \| 0.00 \| 0.78 \| 0.80 \| 0.77 \|
	\| `plug_type` \| 24 \| 0.70 \| 0.77 \| 0.78 \| 0.75 \|
	\| `output_current` \| 20 \| 0.97 \| 0.63 \| 0.67 \| 0.60 \|
	\| `input_voltage` \| 15 \| 0.73 \| 0.77 \| 0.75 \| 0.80 \|
	\| `rated_voltage` \| 15 \| 0.80 \| 0.97 \| 0.94 \| 1.00 \|
	\| `storage_capacity` \| 12 \| 0.79 \| 0.72 \| 0.69 \| 0.75 \|
	\| `manufacturer` \| 11 \| 0.00 \| 0.70 \| 0.67 \| 0.73 \|
	\| `cable_length` \| 10 \| 0.63 \| 0.91 \| 0.83 \| 1.00 \|
	\| `volume` \| 10 \| 0.76 \| 1.00 \| 1.00 \| 1.00 \|
	\| `pack_count` \| 9 \| 0.18 \| 0.78 \| 0.64 \| 1.00 \|
	\| `power` \| 8 \| 0.93 \| 0.80 \| 0.86 \| 0.75 \|
	\| `diameter` \| 6 \| 0.80 \| 0.77 \| 0.71 \| 0.83 \|
	\| `version` \| 6 \| 0.00 \| 0.29 \| 1.00 \| 0.17 \|
	\| `battery_capacity` \| 5 \| 1.00 \| 1.00 \| 1.00 \| 1.00 \|
	\| `rated_current` \| 4 \| 0.80 \| 0.67 \| 0.60 \| 0.75 \|
	\| `input_current` \| 3 \| 1.00 \| 0.00 \| 0.00 \| 0.00 \|
	\| `model_number` \| 2 \| 0.00 \| 1.00 \| 1.00 \| 1.00 \|
	\| `ip_rating` \| 1 \| 1.00 \| 1.00 \| 1.00 \| 1.00 \|

	## Known weaknesses

	- Currents. `output_current`, `rated_current` and `input_current` score below
	the regex baseline. On mixed lines ("Power: 12V, 3A, 36W") the model can label an
	amp value as `power`. A unit check in the caller catches this: an amp value is
	never a power.
	- Decimals in some contexts. "0.5A" has been seen cut to "0".
	- `version` is weak (few examples), and "Gen 2" can come out as `model_number`.
	- `mpn` has only 6 training examples.
	- `serial_number` is learned only from generated serials, because real serial
	lines are removed from the public data. It handles `Serial Number:` / `S/N:` / `SN:`
	lines well; it has not been measured on real serials.

	## Training

	Fine-tuned from `BAAI/bge-small-en-v1.5` for 6 epochs on a Modal T4. Lines with
	attributes that have fewer than 300 spans were repeated 4×, and 300 synthetic
	serial lines were added. Trained 2026-10-15. Code, labelling and redaction
	pipeline: `danielrosehill/Inventory-Spec-Extractor`.

	## Files

	`model_quantized.onnx` (int8, what to load), `tokenizer.json`, `labels.json`.
	Inputs: `input_ids`, `attention_mask`, `token_type_ids` (int64 [batch, seq]).
	Output: `logits` float32 [batch, seq, 51].