Bundle scibert-uniform-negatives-s21-7dc7a1893e from a9087e98e6

905ca6e verified 17 days ago

8.97 kB

---
license: apache-2.0
language: en
library_name: pytorch
base_model: allenai/scibert_scivocab_cased
tags: [information-extraction, relation-extraction, named-entity-recognition, scientific-text, lab-protocols]
---
# Model card: ProtocolLedger span-relation extractor

| | |
|---|---|
| Served model | `scibert-uniform-negatives-s21-7dc7a1893e` |
| Architecture | SpERT-style span and directed-relation classifier over a shared encoder (independent implementation of Eberts and Ulges, arXiv:1909.07755) |
| Encoder | `allenai/scibert_scivocab_cased` at revision `ddf0be025f8e432a1870e34811997ba6725bf04a` (Apache-2.0), fine-tuned |
| Training data | WNUT-2020 Wet Lab Protocols, cleaned **train** split (314 protocols); see the [dataset card](https://github.com/professor3333/ProtocolLedger/blob/main/docs/dataset_card.md) |
| Selection | dev end-to-end relation micro-F1; configuration chosen on a 3-seed mean, then the **median** seed served |
| Status | Evaluated once on the locked test split (2026-10-11) after targets were frozen |

## Intended use

The model extracts actions, materials, quantities, equipment and locations, conditions,
and the directed relations between them, from English wet-lab protocol text. Its output
is a set of **suggested** preparation-sheet entries, each linked to its source
characters, which a reviewer confirms or corrects in the ProtocolLedger interface before
export.

## Not for

- Judging whether a protocol is scientifically complete, correct or safe.
- Executing or automating protocols from unreviewed output.
- Using unreviewed results as final records.
- Filling in implicit arguments, or treating "not explicitly stated" as "not required".
  The system never does this; downstream users should not either.
- Entity or relation types outside the 18-entity and 14-relation schema.
- Non-English text, or PDF / OCR input.
- Other domains or protocol sources without local validation.

## Provenance

- **Encoder.** SciBERT cased, pinned by Hugging Face commit. Case preservation is forced
  and checked at load time, because the cased checkpoint ships no tokenizer config and
  would otherwise lowercase `mM` to `mm`.
- **Encoder view.** Characters the vocabulary cannot represent (for example the micro
  sign in `µL`, which SciBERT maps to `[UNK]`) are substituted only in the text handed to
  the encoder. Offsets always refer to the original text.
- **Heads.**
  - A span is the max-pool of its words plus a width embedding.
  - Entity logits come from the span concatenated with the CLS vector.
  - Relation logits use both spans, the max-pooled words between them, both entity types
    and a signed distance bucket.
  - Relations are multilabel (sigmoid).
- **Training.**
  - AdamW: encoder learning rate 2e-5, heads 1e-3, 10% warmup with linear decay.
  - Batch size 16, at most 10 epochs, patience 3, mixed precision on one Tesla T4 (Kaggle).
  - About 100 sampled negative spans and pairs per step, sampled uniformly.
  - Seed 21; best epoch 9.
- **Reproducibility.** Code commit, split-manifest hash, encoder revision, checkpoint
  hash and the selection-report hash are all recorded in the bundle's `bundle.json`. The
  bundle was rebuilt on CPU and reproduces the GPU dev scores exactly (entity F1 0.8003,
  relation F1 0.6389).
## Training data and splits

The released WNUT-2020 splits were the starting point, with these changes:
- **Deduplication.** Exact and near-duplicate copies and revisions were removed by
  document, always keeping the copy in the most protected split.
- **Correction ledger.** 291 whitespace-trimming corrections were applied identically to
  every split.
- **Locked test.** `test_data_2020` is never used for any decision.

Cleaned sizes: train 314, dev 112, test 111 protocols.

## Metrics (dev, 112 protocols)

| Metric | Selected configuration (3 seeds) | Served seed | CRF + logistic regression |
|---|---:|---:|---:|
| Strict entity micro-F1 | 0.800 | 0.800 | 0.756 |
| **End-to-end relation micro-F1** (primary) | **0.640 ± 0.004** | 0.639 [0.614, 0.665] | 0.582 |
| Relation F1 given gold entities | 0.826 | 0.830 | 0.812 |
| Quantity attachment accuracy | 0.758 | 0.757 | 0.708 |

**Proposed targets** (not yet frozen):
- Entity F1 ≥ 0.75: met.
- At least 5 points above the traditional baseline: met (+5.8).
- **End-to-end relation F1 ≥ 0.65: not met** (0.640).

**Confidence** (served seed, joint score P(relation) × P(head) × P(tail)):
- **Precision.** 0.920 at 30% coverage. 95% precision is reachable only at 18.7%
  coverage, so the target of 95% precision at ≥ 30% coverage is not met.
- **Calibration.** Isotonic calibration, cross-fitted by document on dev, reduces the
  expected calibration error from 0.183 to 0.009.

**Error profile** (served seed):
- **Correct:** 62.7% of gold relations.
- **Endpoint entity wrong:** 27.7%.
- **Missed:** 4.8%.
- **Genuine attachment errors:** 2.9%.
- **Wrong label:** 1.8%.

**Slices:**

| Slice | Served seed | CRF + logistic regression |
|---|---:|---:|
| Reagent recall, names unseen in train | 0.658 | 0.570 |
| Reagent recall, names seen in train | 0.903 | 0.873 |
| Relation recall, documents without a template sibling in train | 0.613 | 0.524 |
| Relation recall, documents with a template sibling in train | 0.740 | 0.719 |

**Reference point.** Duplicate copies of the same protocol in the corpus agree with each
other at entity F1 0.863 and relation F1 0.711.

## Locked test results (evaluated once)

| Metric (cleaned `test_data_2020`, 111 protocols) | SciBERT (served) | CRF + logistic regression |
|---|---:|---:|
| Strict entity micro-F1 | 0.766 [0.748, 0.784] | 0.703 |
| **End-to-end relation micro-F1** | **0.580** [0.552, 0.606] | 0.504 |
| Relation F1 given gold entities | 0.787 | 0.768 |
| Quantity attachment accuracy | 0.699 | 0.629 |
| Reagent recall, names unseen in train | 0.762 | 0.631 |
| Precision at 30% coverage (joint score) | 0.905 | — |
| Coverage at 95% precision | 13.4% | — |
| ECE raw / with the dev-fitted calibrator | 0.172 / 0.021 | — |

Frozen targets on test:
- **Met:** entity F1 ≥ 0.75, at least 5 points above the traditional baseline (+7.6),
  and robustness (−0.01).
- **Not met:** end-to-end relation F1 ≥ 0.65, and 95% precision at ≥ 30% coverage.

Test is harder than dev for every system. Full report:
[reports/test/locked-test.md](https://github.com/professor3333/ProtocolLedger/blob/main/reports/test/locked-test.md).

## Inference performance

Measured on an Apple M1 (8 GB) with 4 inference threads: 200 sequential requests of dev
protocols up to 500 words, each runtime in a fresh process
([report](https://github.com/professor3333/ProtocolLedger/blob/main/reports/performance/scibert-uniform-negatives-s21-7dc7a1893e-runtimes.md)).

| Runtime | Relation F1 | p50 | p95 | Peak RSS |
|---|---:|---:|---:|---:|
| **PyTorch FP32 (served)** | 0.6389 | 0.48 s | **2.07 s** | 938 MiB |
| ONNX Runtime FP32 | 0.6389 | 2.29 s | 6.94 s | 1,058 MiB |
| ONNX Runtime INT8 (dynamic) | 0.6361 | 0.93 s | 2.90 s | 918 MiB |

On this ARM machine PyTorch FP32 is the fastest runtime and loses no accuracy, so it is
served. INT8 stays within the 1-point accuracy gate but is slower here. Re-measure on the
deployment host (likely x86) with `scripts/optimize_bundle.py` before changing runtimes.

## Known failure modes

- **Entity errors cascade into relations.** Most relation errors come from a wrong or
  missed endpoint entity.
- **Open-vocabulary types are weakest.** Modifier, Method, Device and Generic-Measure
  score lowest.
- **Unseen reagent names** are found less often than seen ones.
- **Rare relations** are unreliable: Commands (37 dev instances), Of-Type (26) and
  Product.
- **Implicit references** such as "it", and cross-step coreference, are not resolved.
- **Negation, alternatives and conditional instructions** are not modelled.
- **Text glued together by table flattening** (for example `100 µlof`) cannot be
  segmented.
- **Template overlap.** Protocols that share a vendor template with training data score
  higher than the rest.

## Release process

- **Selection.** Thresholds and the configuration are chosen on dev only, never on test.
- **Bundle.** Weights are stored as safetensors (no pickle). The encoder config and
  tokenizer are included, and every file is hash-verified on load, so serving needs no
  network access.
- **Release.** A release pins the image digest together with the bundle hash.
  `scripts/release.py` smoke-tests each activation and restores the previous image and
  bundle together if it fails.

## Loading

This repository is a ProtocolLedger bundle (`bundle.json` lists every file with its SHA-256). Load it with the project's verified loader:

```python
from pathlib import Path
from protocolledger.serve.bundle import load_bundle
bundle = load_bundle(Path('scibert-uniform-negatives-s21-7dc7a1893e'))
entities, relations = bundle.extractor.extract('Add 5 mL of water to the tube.')
```

	---
	license: apache-2.0
	language: en
	library_name: pytorch
	base_model: allenai/scibert_scivocab_cased
	tags: [information-extraction, relation-extraction, named-entity-recognition, scientific-text, lab-protocols]
	---

	# Model card: ProtocolLedger span-relation extractor

	\| \| \|
	\|---\|---\|
	\| Served model \| `scibert-uniform-negatives-s21-7dc7a1893e` \|
	\| Architecture \| SpERT-style span and directed-relation classifier over a shared encoder (independent implementation of Eberts and Ulges, arXiv:1909.07755) \|
	\| Encoder \| `allenai/scibert_scivocab_cased` at revision `ddf0be025f8e432a1870e34811997ba6725bf04a` (Apache-2.0), fine-tuned \|
	\| Training data \| WNUT-2020 Wet Lab Protocols, cleaned train split (314 protocols); see the [dataset card](https://github.com/professor3333/ProtocolLedger/blob/main/docs/dataset_card.md) \|
	\| Selection \| dev end-to-end relation micro-F1; configuration chosen on a 3-seed mean, then the median seed served \|
	\| Status \| Evaluated once on the locked test split (2026-10-11) after targets were frozen \|

	## Intended use

	The model extracts actions, materials, quantities, equipment and locations, conditions,
	and the directed relations between them, from English wet-lab protocol text. Its output
	is a set of suggested preparation-sheet entries, each linked to its source
	characters, which a reviewer confirms or corrects in the ProtocolLedger interface before
	export.

	## Not for

	- Judging whether a protocol is scientifically complete, correct or safe.
	- Executing or automating protocols from unreviewed output.
	- Using unreviewed results as final records.
	- Filling in implicit arguments, or treating "not explicitly stated" as "not required".
	The system never does this; downstream users should not either.
	- Entity or relation types outside the 18-entity and 14-relation schema.
	- Non-English text, or PDF / OCR input.
	- Other domains or protocol sources without local validation.

	## Provenance

	- Encoder. SciBERT cased, pinned by Hugging Face commit. Case preservation is forced
	and checked at load time, because the cased checkpoint ships no tokenizer config and
	would otherwise lowercase `mM` to `mm`.
	- Encoder view. Characters the vocabulary cannot represent (for example the micro
	sign in `µL`, which SciBERT maps to `[UNK]`) are substituted only in the text handed to
	the encoder. Offsets always refer to the original text.
	- Heads.
	- A span is the max-pool of its words plus a width embedding.
	- Entity logits come from the span concatenated with the CLS vector.
	- Relation logits use both spans, the max-pooled words between them, both entity types
	and a signed distance bucket.
	- Relations are multilabel (sigmoid).
	- Training.
	- AdamW: encoder learning rate 2e-5, heads 1e-3, 10% warmup with linear decay.
	- Batch size 16, at most 10 epochs, patience 3, mixed precision on one Tesla T4 (Kaggle).
	- About 100 sampled negative spans and pairs per step, sampled uniformly.
	- Seed 21; best epoch 9.
	- Reproducibility. Code commit, split-manifest hash, encoder revision, checkpoint
	hash and the selection-report hash are all recorded in the bundle's `bundle.json`. The
	bundle was rebuilt on CPU and reproduces the GPU dev scores exactly (entity F1 0.8003,
	relation F1 0.6389).

	## Training data and splits

	The released WNUT-2020 splits were the starting point, with these changes:
	- Deduplication. Exact and near-duplicate copies and revisions were removed by
	document, always keeping the copy in the most protected split.
	- Correction ledger. 291 whitespace-trimming corrections were applied identically to
	every split.
	- Locked test. `test_data_2020` is never used for any decision.

	Cleaned sizes: train 314, dev 112, test 111 protocols.

	## Metrics (dev, 112 protocols)

	\| Metric \| Selected configuration (3 seeds) \| Served seed \| CRF + logistic regression \|
	\|---\|---:\|---:\|---:\|
	\| Strict entity micro-F1 \| 0.800 \| 0.800 \| 0.756 \|
	\| End-to-end relation micro-F1 (primary) \| 0.640 ± 0.004 \| 0.639 [0.614, 0.665] \| 0.582 \|
	\| Relation F1 given gold entities \| 0.826 \| 0.830 \| 0.812 \|
	\| Quantity attachment accuracy \| 0.758 \| 0.757 \| 0.708 \|

	Proposed targets (not yet frozen):
	- Entity F1 ≥ 0.75: met.
	- At least 5 points above the traditional baseline: met (+5.8).
	- End-to-end relation F1 ≥ 0.65: not met (0.640).

	Confidence (served seed, joint score P(relation) × P(head) × P(tail)):
	- Precision. 0.920 at 30% coverage. 95% precision is reachable only at 18.7%
	coverage, so the target of 95% precision at ≥ 30% coverage is not met.
	- Calibration. Isotonic calibration, cross-fitted by document on dev, reduces the
	expected calibration error from 0.183 to 0.009.

	Error profile (served seed):
	- Correct: 62.7% of gold relations.
	- Endpoint entity wrong: 27.7%.
	- Missed: 4.8%.
	- Genuine attachment errors: 2.9%.
	- Wrong label: 1.8%.

	Slices:

	\| Slice \| Served seed \| CRF + logistic regression \|
	\|---\|---:\|---:\|
	\| Reagent recall, names unseen in train \| 0.658 \| 0.570 \|
	\| Reagent recall, names seen in train \| 0.903 \| 0.873 \|
	\| Relation recall, documents without a template sibling in train \| 0.613 \| 0.524 \|
	\| Relation recall, documents with a template sibling in train \| 0.740 \| 0.719 \|

	Reference point. Duplicate copies of the same protocol in the corpus agree with each
	other at entity F1 0.863 and relation F1 0.711.

	## Locked test results (evaluated once)

	\| Metric (cleaned `test_data_2020`, 111 protocols) \| SciBERT (served) \| CRF + logistic regression \|
	\|---\|---:\|---:\|
	\| Strict entity micro-F1 \| 0.766 [0.748, 0.784] \| 0.703 \|
	\| End-to-end relation micro-F1 \| 0.580 [0.552, 0.606] \| 0.504 \|
	\| Relation F1 given gold entities \| 0.787 \| 0.768 \|
	\| Quantity attachment accuracy \| 0.699 \| 0.629 \|
	\| Reagent recall, names unseen in train \| 0.762 \| 0.631 \|
	\| Precision at 30% coverage (joint score) \| 0.905 \| — \|
	\| Coverage at 95% precision \| 13.4% \| — \|
	\| ECE raw / with the dev-fitted calibrator \| 0.172 / 0.021 \| — \|

	Frozen targets on test:
	- Met: entity F1 ≥ 0.75, at least 5 points above the traditional baseline (+7.6),
	and robustness (−0.01).
	- Not met: end-to-end relation F1 ≥ 0.65, and 95% precision at ≥ 30% coverage.

	Test is harder than dev for every system. Full report:
	[reports/test/locked-test.md](https://github.com/professor3333/ProtocolLedger/blob/main/reports/test/locked-test.md).

	## Inference performance

	Measured on an Apple M1 (8 GB) with 4 inference threads: 200 sequential requests of dev
	protocols up to 500 words, each runtime in a fresh process
	([report](https://github.com/professor3333/ProtocolLedger/blob/main/reports/performance/scibert-uniform-negatives-s21-7dc7a1893e-runtimes.md)).

	\| Runtime \| Relation F1 \| p50 \| p95 \| Peak RSS \|
	\|---\|---:\|---:\|---:\|---:\|
	\| PyTorch FP32 (served) \| 0.6389 \| 0.48 s \| 2.07 s \| 938 MiB \|
	\| ONNX Runtime FP32 \| 0.6389 \| 2.29 s \| 6.94 s \| 1,058 MiB \|
	\| ONNX Runtime INT8 (dynamic) \| 0.6361 \| 0.93 s \| 2.90 s \| 918 MiB \|

	On this ARM machine PyTorch FP32 is the fastest runtime and loses no accuracy, so it is
	served. INT8 stays within the 1-point accuracy gate but is slower here. Re-measure on the
	deployment host (likely x86) with `scripts/optimize_bundle.py` before changing runtimes.

	## Known failure modes

	- Entity errors cascade into relations. Most relation errors come from a wrong or
	missed endpoint entity.
	- Open-vocabulary types are weakest. Modifier, Method, Device and Generic-Measure
	score lowest.
	- Unseen reagent names are found less often than seen ones.
	- Rare relations are unreliable: Commands (37 dev instances), Of-Type (26) and
	Product.
	- Implicit references such as "it", and cross-step coreference, are not resolved.
	- Negation, alternatives and conditional instructions are not modelled.
	- Text glued together by table flattening (for example `100 µlof`) cannot be
	segmented.
	- Template overlap. Protocols that share a vendor template with training data score
	higher than the rest.

	## Release process

	- Selection. Thresholds and the configuration are chosen on dev only, never on test.
	- Bundle. Weights are stored as safetensors (no pickle). The encoder config and
	tokenizer are included, and every file is hash-verified on load, so serving needs no
	network access.
	- Release. A release pins the image digest together with the bundle hash.
	`scripts/release.py` smoke-tests each activation and restores the previous image and
	bundle together if it fails.

	## Loading

	This repository is a ProtocolLedger bundle (`bundle.json` lists every file with its SHA-256). Load it with the project's verified loader:

	```python
	from pathlib import Path
	from protocolledger.serve.bundle import load_bundle

	bundle = load_bundle(Path('scibert-uniform-negatives-s21-7dc7a1893e'))
	entities, relations = bundle.extractor.extract('Add 5 mL of water to the tube.')
	```