Bundle scibert-uniform-negatives-s21-7dc7a1893e from a9087e98e6

README.md

@@ -0,0 +1,186 @@
+---
+license: apache-2.0
+language: en
+library_name: pytorch
+base_model: allenai/scibert_scivocab_cased
+tags: [information-extraction, relation-extraction, named-entity-recognition, scientific-text, lab-protocols]
+---
+
+# Model card: ProtocolLedger span-relation extractor
+
+| | |
+|---|---|
+| Served model | `scibert-uniform-negatives-s21-7dc7a1893e` |
+| Architecture | SpERT-style span and directed-relation classifier over a shared encoder (independent implementation of Eberts and Ulges, arXiv:1909.07755) |
+| Encoder | `allenai/scibert_scivocab_cased` at revision `ddf0be025f8e432a1870e34811997ba6725bf04a` (Apache-2.0), fine-tuned |
+| Training data | WNUT-2020 Wet Lab Protocols, cleaned **train** split (314 protocols); see the [dataset card](https://github.com/professor3333/ProtocolLedger/blob/main/docs/dataset_card.md) |
+| Selection | dev end-to-end relation micro-F1; configuration chosen on a 3-seed mean, then the **median** seed served |
+| Status | Evaluated once on the locked test split (2026-10-11) after targets were frozen |
+
+## Intended use
+
+The model extracts actions, materials, quantities, equipment and locations, conditions,
+and the directed relations between them, from English wet-lab protocol text. Its output
+is a set of **suggested** preparation-sheet entries, each linked to its source
+characters, which a reviewer confirms or corrects in the ProtocolLedger interface before
+export.
+
+## Not for
+
+- Judging whether a protocol is scientifically complete, correct or safe.
+- Executing or automating protocols from unreviewed output.
+- Using unreviewed results as final records.
+- Filling in implicit arguments, or treating "not explicitly stated" as "not required".
+  The system never does this; downstream users should not either.
+- Entity or relation types outside the 18-entity and 14-relation schema.
+- Non-English text, or PDF / OCR input.
+- Other domains or protocol sources without local validation.
+
+## Provenance
+
+- **Encoder.** SciBERT cased, pinned by Hugging Face commit. Case preservation is forced
+  and checked at load time, because the cased checkpoint ships no tokenizer config and
+  would otherwise lowercase `mM` to `mm`.
+- **Encoder view.** Characters the vocabulary cannot represent (for example the micro
+  sign in `µL`, which SciBERT maps to `[UNK]`) are substituted only in the text handed to
+  the encoder. Offsets always refer to the original text.
+- **Heads.**
+  - A span is the max-pool of its words plus a width embedding.
+  - Entity logits come from the span concatenated with the CLS vector.
+  - Relation logits use both spans, the max-pooled words between them, both entity types
+    and a signed distance bucket.
+  - Relations are multilabel (sigmoid).
+- **Training.**
+  - AdamW: encoder learning rate 2e-5, heads 1e-3, 10% warmup with linear decay.
+  - Batch size 16, at most 10 epochs, patience 3, mixed precision on one Tesla T4 (Kaggle).
+  - About 100 sampled negative spans and pairs per step, sampled uniformly.
+  - Seed 21; best epoch 9.
+- **Reproducibility.** Code commit, split-manifest hash, encoder revision, checkpoint
+  hash and the selection-report hash are all recorded in the bundle's `bundle.json`. The
+  bundle was rebuilt on CPU and reproduces the GPU dev scores exactly (entity F1 0.8003,
+  relation F1 0.6389).
+
+## Training data and splits
+
+The released WNUT-2020 splits were the starting point, with these changes:
+- **Deduplication.** Exact and near-duplicate copies and revisions were removed by
+  document, always keeping the copy in the most protected split.
+- **Correction ledger.** 291 whitespace-trimming corrections were applied identically to
+  every split.
+- **Locked test.** `test_data_2020` is never used for any decision.
+
+Cleaned sizes: train 314, dev 112, test 111 protocols.
+
+## Metrics (dev, 112 protocols)
+
+| Metric | Selected configuration (3 seeds) | Served seed | CRF + logistic regression |
+|---|---:|---:|---:|
+| Strict entity micro-F1 | 0.800 | 0.800 | 0.756 |
+| **End-to-end relation micro-F1** (primary) | **0.640 ± 0.004** | 0.639 [0.614, 0.665] | 0.582 |
+| Relation F1 given gold entities | 0.826 | 0.830 | 0.812 |
+| Quantity attachment accuracy | 0.758 | 0.757 | 0.708 |
+
+**Proposed targets** (not yet frozen):
+- Entity F1 ≥ 0.75: met.
+- At least 5 points above the traditional baseline: met (+5.8).
+- **End-to-end relation F1 ≥ 0.65: not met** (0.640).
+
+**Confidence** (served seed, joint score P(relation) × P(head) × P(tail)):
+- **Precision.** 0.920 at 30% coverage. 95% precision is reachable only at 18.7%
+  coverage, so the target of 95% precision at ≥ 30% coverage is not met.
+- **Calibration.** Isotonic calibration, cross-fitted by document on dev, reduces the
+  expected calibration error from 0.183 to 0.009.
+
+**Error profile** (served seed):
+- **Correct:** 62.7% of gold relations.
+- **Endpoint entity wrong:** 27.7%.
+- **Missed:** 4.8%.
+- **Genuine attachment errors:** 2.9%.
+- **Wrong label:** 1.8%.
+
+**Slices:**
+
+| Slice | Served seed | CRF + logistic regression |
+|---|---:|---:|
+| Reagent recall, names unseen in train | 0.658 | 0.570 |
+| Reagent recall, names seen in train | 0.903 | 0.873 |
+| Relation recall, documents without a template sibling in train | 0.613 | 0.524 |
+| Relation recall, documents with a template sibling in train | 0.740 | 0.719 |
+
+**Reference point.** Duplicate copies of the same protocol in the corpus agree with each
+other at entity F1 0.863 and relation F1 0.711.
+
+## Locked test results (evaluated once)
+
+| Metric (cleaned `test_data_2020`, 111 protocols) | SciBERT (served) | CRF + logistic regression |
+|---|---:|---:|
+| Strict entity micro-F1 | 0.766 [0.748, 0.784] | 0.703 |
+| **End-to-end relation micro-F1** | **0.580** [0.552, 0.606] | 0.504 |
+| Relation F1 given gold entities | 0.787 | 0.768 |
+| Quantity attachment accuracy | 0.699 | 0.629 |
+| Reagent recall, names unseen in train | 0.762 | 0.631 |
+| Precision at 30% coverage (joint score) | 0.905 | — |
+| Coverage at 95% precision | 13.4% | — |
+| ECE raw / with the dev-fitted calibrator | 0.172 / 0.021 | — |
+
+Frozen targets on test:
+- **Met:** entity F1 ≥ 0.75, at least 5 points above the traditional baseline (+7.6),
+  and robustness (−0.01).
+- **Not met:** end-to-end relation F1 ≥ 0.65, and 95% precision at ≥ 30% coverage.
+
+Test is harder than dev for every system. Full report:
+[reports/test/locked-test.md](https://github.com/professor3333/ProtocolLedger/blob/main/reports/test/locked-test.md).
+
+## Inference performance
+
+Measured on an Apple M1 (8 GB) with 4 inference threads: 200 sequential requests of dev
+protocols up to 500 words, each runtime in a fresh process
+([report](https://github.com/professor3333/ProtocolLedger/blob/main/reports/performance/scibert-uniform-negatives-s21-7dc7a1893e-runtimes.md)).
+
+| Runtime | Relation F1 | p50 | p95 | Peak RSS |
+|---|---:|---:|---:|---:|
+| **PyTorch FP32 (served)** | 0.6389 | 0.48 s | **2.07 s** | 938 MiB |
+| ONNX Runtime FP32 | 0.6389 | 2.29 s | 6.94 s | 1,058 MiB |
+| ONNX Runtime INT8 (dynamic) | 0.6361 | 0.93 s | 2.90 s | 918 MiB |
+
+On this ARM machine PyTorch FP32 is the fastest runtime and loses no accuracy, so it is
+served. INT8 stays within the 1-point accuracy gate but is slower here. Re-measure on the
+deployment host (likely x86) with `scripts/optimize_bundle.py` before changing runtimes.
+
+## Known failure modes
+
+- **Entity errors cascade into relations.** Most relation errors come from a wrong or
+  missed endpoint entity.
+- **Open-vocabulary types are weakest.** Modifier, Method, Device and Generic-Measure
+  score lowest.
+- **Unseen reagent names** are found less often than seen ones.
+- **Rare relations** are unreliable: Commands (37 dev instances), Of-Type (26) and
+  Product.
+- **Implicit references** such as "it", and cross-step coreference, are not resolved.
+- **Negation, alternatives and conditional instructions** are not modelled.
+- **Text glued together by table flattening** (for example `100 µlof`) cannot be
+  segmented.
+- **Template overlap.** Protocols that share a vendor template with training data score
+  higher than the rest.
+
+## Release process
+
+- **Selection.** Thresholds and the configuration are chosen on dev only, never on test.
+- **Bundle.** Weights are stored as safetensors (no pickle). The encoder config and
+  tokenizer are included, and every file is hash-verified on load, so serving needs no
+  network access.
+- **Release.** A release pins the image digest together with the bundle hash.
+  `scripts/release.py` smoke-tests each activation and restores the previous image and
+  bundle together if it fails.
+
+## Loading
+
+This repository is a ProtocolLedger bundle (`bundle.json` lists every file with its SHA-256). Load it with the project's verified loader:
+
+```python
+from pathlib import Path
+from protocolledger.serve.bundle import load_bundle
+
+bundle = load_bundle(Path('scibert-uniform-negatives-s21-7dc7a1893e'))
+entities, relations = bundle.extractor.extract('Add 5 mL of water to the tube.')
+```

bundle.json

@@ -0,0 +1,577 @@
+{
+ "bundle_format": 1,
+ "model_version": "scibert-uniform-negatives-s21-7dc7a1893e",
+ "system": "span-relation",
+ "preprocessing_version": "neural-1",
+ "neural_config": {
+  "neural_version": "1",
+  "encoder": "scibert",
+  "freeze_encoder": false,
+  "max_span_words": 10,
+  "width_embedding_dim": 25,
+  "type_embedding_dim": 25,
+  "distance_embedding_dim": 25,
+  "use_type_features": true,
+  "use_distance_features": true,
+  "dropout": 0.1,
+  "negative_spans": 100,
+  "negative_pairs": 100,
+  "hard_negatives": false,
+  "learning_rate": 2e-05,
+  "head_learning_rate": 0.001,
+  "weight_decay": 0.01,
+  "warmup_fraction": 0.1,
+  "batch_size": 16,
+  "grad_accumulation": 1,
+  "epochs": 10,
+  "patience": 3,
+  "relation_threshold": 0.5,
+  "restrict_type_pairs": true,
+  "max_subwords": 510,
+  "seed": 21,
+  "mixed_precision": true
+ },
+ "entity_labels": [
+  "Action",
+  "Amount",
+  "Concentration",
+  "Device",
+  "Generic-Measure",
+  "Location",
+  "Measure-Type",
+  "Mention",
+  "Method",
+  "Modifier",
+  "Numerical",
+  "Reagent",
+  "Seal",
+  "Size",
+  "Speed",
+  "Temperature",
+  "Time",
+  "pH"
+ ],
+ "relation_labels": [
+  "Acts-on",
+  "Commands",
+  "Coreference-Link",
+  "Count",
+  "Measure",
+  "Measure-Type-Link",
+  "Meronym",
+  "Mod-Link",
+  "Of-Type",
+  "Or",
+  "Product",
+  "Setting",
+  "Site",
+  "Using"
+ ],
+ "type_pairs": [
+  [
+   "Action",
+   "Action"
+  ],
+  [
+   "Action",
+   "Amount"
+  ],
+  [
+   "Action",
+   "Concentration"
+  ],
+  [
+   "Action",
+   "Device"
+  ],
+  [
+   "Action",
+   "Generic-Measure"
+  ],
+  [
+   "Action",
+   "Location"
+  ],
+  [
+   "Action",
+   "Measure-Type"
+  ],
+  [
+   "Action",
+   "Mention"
+  ],
+  [
+   "Action",
+   "Method"
+  ],
+  [
+   "Action",
+   "Modifier"
+  ],
+  [
+   "Action",
+   "Numerical"
+  ],
+  [
+   "Action",
+   "Reagent"
+  ],
+  [
+   "Action",
+   "Seal"
+  ],
+  [
+   "Action",
+   "Size"
+  ],
+  [
+   "Action",
+   "Speed"
+  ],
+  [
+   "Action",
+   "Temperature"
+  ],
+  [
+   "Action",
+   "Time"
+  ],
+  [
+   "Action",
+   "pH"
+  ],
+  [
+   "Amount",
+   "Amount"
+  ],
+  [
+   "Amount",
+   "Mention"
+  ],
+  [
+   "Amount",
+   "Modifier"
+  ],
+  [
+   "Amount",
+   "Numerical"
+  ],
+  [
+   "Amount",
+   "Reagent"
+  ],
+  [
+   "Concentration",
+   "Concentration"
+  ],
+  [
+   "Concentration",
+   "Modifier"
+  ],
+  [
+   "Concentration",
+   "Reagent"
+  ],
+  [
+   "Device",
+   "Amount"
+  ],
+  [
+   "Device",
+   "Concentration"
+  ],
+  [
+   "Device",
+   "Device"
+  ],
+  [
+   "Device",
+   "Generic-Measure"
+  ],
+  [
+   "Device",
+   "Location"
+  ],
+  [
+   "Device",
+   "Method"
+  ],
+  [
+   "Device",
+   "Modifier"
+  ],
+  [
+   "Device",
+   "Numerical"
+  ],
+  [
+   "Device",
+   "Reagent"
+  ],
+  [
+   "Device",
+   "Size"
+  ],
+  [
+   "Device",
+   "Speed"
+  ],
+  [
+   "Device",
+   "Temperature"
+  ],
+  [
+   "Device",
+   "Time"
+  ],
+  [
+   "Generic-Measure",
+   "Generic-Measure"
+  ],
+  [
+   "Generic-Measure",
+   "Measure-Type"
+  ],
+  [
+   "Generic-Measure",
+   "Modifier"
+  ],
+  [
+   "Location",
+   "Amount"
+  ],
+  [
+   "Location",
+   "Concentration"
+  ],
+  [
+   "Location",
+   "Device"
+  ],
+  [
+   "Location",
+   "Generic-Measure"
+  ],
+  [
+   "Location",
+   "Location"
+  ],
+  [
+   "Location",
+   "Mention"
+  ],
+  [
+   "Location",
+   "Modifier"
+  ],
+  [
+   "Location",
+   "Numerical"
+  ],
+  [
+   "Location",
+   "Reagent"
+  ],
+  [
+   "Location",
+   "Seal"
+  ],
+  [
+   "Location",
+   "Size"
+  ],
+  [
+   "Location",
+   "Temperature"
+  ],
+  [
+   "Location",
+   "Time"
+  ],
+  [
+   "Measure-Type",
+   "Amount"
+  ],
+  [
+   "Measure-Type",
+   "Measure-Type"
+  ],
+  [
+   "Measure-Type",
+   "Modifier"
+  ],
+  [
+   "Measure-Type",
+   "Reagent"
+  ],
+  [
+   "Mention",
+   "Action"
+  ],
+  [
+   "Mention",
+   "Amount"
+  ],
+  [
+   "Mention",
+   "Concentration"
+  ],
+  [
+   "Mention",
+   "Device"
+  ],
+  [
+   "Mention",
+   "Generic-Measure"
+  ],
+  [
+   "Mention",
+   "Location"
+  ],
+  [
+   "Mention",
+   "Mention"
+  ],
+  [
+   "Mention",
+   "Modifier"
+  ],
+  [
+   "Mention",
+   "Numerical"
+  ],
+  [
+   "Mention",
+   "Reagent"
+  ],
+  [
+   "Mention",
+   "pH"
+  ],
+  [
+   "Method",
+   "Device"
+  ],
+  [
+   "Method",
+   "Method"
+  ],
+  [
+   "Method",
+   "Modifier"
+  ],
+  [
+   "Modifier",
+   "Amount"
+  ],
+  [
+   "Modifier",
+   "Generic-Measure"
+  ],
+  [
+   "Modifier",
+   "Location"
+  ],
+  [
+   "Modifier",
+   "Modifier"
+  ],
+  [
+   "Modifier",
+   "Numerical"
+  ],
+  [
+   "Modifier",
+   "Reagent"
+  ],
+  [
+   "Modifier",
+   "Size"
+  ],
+  [
+   "Modifier",
+   "Speed"
+  ],
+  [
+   "Modifier",
+   "Temperature"
+  ],
+  [
+   "Modifier",
+   "Time"
+  ],
+  [
+   "Numerical",
+   "Measure-Type"
+  ],
+  [
+   "Numerical",
+   "Modifier"
+  ],
+  [
+   "Numerical",
+   "Numerical"
+  ],
+  [
+   "Reagent",
+   "Amount"
+  ],
+  [
+   "Reagent",
+   "Concentration"
+  ],
+  [
+   "Reagent",
+   "Device"
+  ],
+  [
+   "Reagent",
+   "Generic-Measure"
+  ],
+  [
+   "Reagent",
+   "Location"
+  ],
+  [
+   "Reagent",
+   "Measure-Type"
+  ],
+  [
+   "Reagent",
+   "Mention"
+  ],
+  [
+   "Reagent",
+   "Modifier"
+  ],
+  [
+   "Reagent",
+   "Numerical"
+  ],
+  [
+   "Reagent",
+   "Reagent"
+  ],
+  [
+   "Reagent",
+   "Seal"
+  ],
+  [
+   "Reagent",
+   "Size"
+  ],
+  [
+   "Reagent",
+   "pH"
+  ],
+  [
+   "Seal",
+   "Device"
+  ],
+  [
+   "Seal",
+   "Location"
+  ],
+  [
+   "Seal",
+   "Mention"
+  ],
+  [
+   "Seal",
+   "Modifier"
+  ],
+  [
+   "Seal",
+   "Numerical"
+  ],
+  [
+   "Seal",
+   "Seal"
+  ],
+  [
+   "Seal",
+   "Size"
+  ],
+  [
+   "Size",
+   "Modifier"
+  ],
+  [
+   "Size",
+   "Size"
+  ],
+  [
+   "Speed",
+   "Modifier"
+  ],
+  [
+   "Speed",
+   "Speed"
+  ],
+  [
+   "Temperature",
+   "Modifier"
+  ],
+  [
+   "Temperature",
+   "Temperature"
+  ],
+  [
+   "Time",
+   "Modifier"
+  ],
+  [
+   "Time",
+   "Time"
+  ],
+  [
+   "pH",
+   "Modifier"
+  ]
+ ],
+ "hidden_size": 768,
+ "files": {
+  "encoder/config.json": {
+   "path": "encoder/config.json",
+   "sha256": "4d08a96f73e258cd3fb48598f5ff887f402c9153a1101034b78b639bd49b2e6a"
+  },
+  "model.safetensors": {
+   "path": "model.safetensors",
+   "sha256": "117953d1d93aa6b737aba2b118b28efcc81112410d71c8dd887125da7048821a"
+  },
+  "tokenizer/tokenizer.json": {
+   "path": "tokenizer/tokenizer.json",
+   "sha256": "7d7f7f94ebc356bfdbfd1fc2dde4d5a9a15d24dc9dd38c642b6392e8c6cb7462"
+  },
+  "tokenizer/tokenizer_config.json": {
+   "path": "tokenizer/tokenizer_config.json",
+   "sha256": "4462f2399527c65c2eaaa75bf548f35fd0bf321f28c2d48e52b409767b6f4c86"
+  }
+ },
+ "runtime": {
+  "protocolledger": "0.0.1",
+  "python": "3.12.6",
+  "torch": "2.14.1"
+ },
+ "training": {
+  "run": "scibert-uniform-negatives-s21",
+  "report": "reports/neural/scibert-uniform-negatives-s21-dev.json",
+  "report_sha256": "d5a7a9e3363ab14ea1f31f542f4f940817d51e75de60e323c27b56a45c177d18",
+  "checkpoint_sha256": "7dc7a1893e1567293c27922ef25966e9ef9c6627a1f71ef386e8c0736b5513b7",
+  "best_epoch": 9,
+  "encoder": "allenai/scibert_scivocab_cased",
+  "encoder_revision": "ddf0be025f8e432a1870e34811997ba6725bf04a",
+  "splits_manifest_sha256": "e0136ae42662b7857cad7b25d4dc56a01eca4808b98e0544974624b0d1d1710c",
+  "code_commit": "95b2d5c2124e3f9146653dc78005d423b3df8d2a"
+ },
+ "evaluation": {
+  "split": "dev",
+  "device": "cpu",
+  "entity_f1": 0.8003,
+  "relation_f1_end_to_end": 0.6389,
+  "reported_gpu": {
+   "entity_f1": 0.8003,
+   "relation_f1_end_to_end": 0.6389
+  }
+ }
+}

encoder/config.json

@@ -0,0 +1,24 @@
+{
+  "add_cross_attention": false,
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": null,
+  "classifier_dropout": null,
+  "eos_token_id": null,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "is_decoder": false,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.19.0",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 31116
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:117953d1d93aa6b737aba2b118b28efcc81112410d71c8dd887125da7048821a
+size 440041032

tokenizer/tokenizer_config.json

@@ -0,0 +1,15 @@
+{
+  "backend": "tokenizers",
+  "cls_token": "[CLS]",
+  "do_lower_case": false,
+  "is_local": false,
+  "local_files_only": false,
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}