# Fresh 012 comparison

Run: 2 October 2026. Article analysis updated 3 October. Source release: v0.0.1.

## Exact software-name scores

Same frozen 480-token windows, 64-token overlap. Exact offsets; coverage masks exclude unjudged regions. Softcite primary projection includes explicit software names and language fields, excluding implicit names. These are matched-window evaluations, not whole-document pipeline timings.

| Dataset | Model | Precision % | Recall % | F1 % |
|---|---|---:|---:|---:|
| fresh30 | OSSoMeX 012 | 89.5 | 66.7 | 76.4 |
| fresh30 | Softcite Wapiti | 92.3 | 23.5 | 37.5 |
| fresh30 | Softcite SciBERT | 86.5 | 62.7 | 72.7 |
| exposed50 | OSSoMeX 012 | 86.2 | 42.1 | 56.6 |
| exposed50 | Softcite Wapiti | 90.7 | 29.3 | 44.3 |
| exposed50 | Softcite SciBERT | 70.7 | 30.8 | 42.9 |
| reserve27 | OSSoMeX 012 | 88.9 | 43.8 | 58.7 |
| reserve27 | Softcite Wapiti | 100.0 | 42.5 | 59.6 |
| reserve27 | Softcite SciBERT | 88.6 | 42.5 | 57.4 |
| regression7 | OSSoMeX 012 | 85.7 | 58.8 | 69.8 |
| regression7 | Softcite Wapiti | 75.0 | 17.6 | 28.6 |
| regression7 | Softcite SciBERT | 83.3 | 39.2 | 53.3 |
| softcite-gold-paragraphs | OSSoMeX 012 | 78.4 | 72.2 | 75.1 |
| softcite-gold-paragraphs | Softcite Wapiti | 93.7 | 92.7 | 93.2 |
| softcite-gold-paragraphs | Softcite SciBERT | 91.5 | 92.7 | 92.1 |

## Confidence and scope

The local OpenAlex/regression references are agent-provisional and development-exposed. On fresh30, 012 minus SciBERT is +3.7 F1 points, 95% paired-bootstrap interval -11.7 to +20.5. Gold is -16.9 points, paper-clustered interval -22.6 to -12.0. The 461 positive paragraphs come from 233 articles, not the entire published holdout. Paragraph/document views overlap; do not pool them. Baseline training membership was not independently audited.

Only 012 was freshly evaluated among OSSoMeX checkpoints. Its name does not establish that it is better than 006.

## Other tasks

| Measure | 012 | Wapiti | SciBERT |
|---|---:|---:|---:|
| Corrected SoMeSci10 name recall, 145 references | 71.7% | 36.6% | 51.0% |
| Corrected SoFAIR6 name recall, 120 references | 75.8% | 6.7% | 38.3% |
| Regression7 name-version link F1, 28 reference links | 79.2% | 18.2% | 27.8% |
| 20-paper catalogue coverage, 666 links | 55.1% | 5.3% | 40.1% |

SoMeSci/SoFAIR are previously inspected small subsets with unaudited negative coverage: recall only, not precision/F1. SoFAIR uses the disclosed posthoc whitespace-only normalization sensitivity. Catalogue matches are not mention recall or validated project resolution; the selected papers are heavily biomedical/bioinformatics.

Fresh30 sentiment observed macro-F1 is 20.5%, excluding absent classes. All three selected positive alias pairs were missed. These attributes are not established advantages.

## Runtime

90 warm observations per arm: 012 full pipeline 16.44 snippets/s on Apple MPS (median 37.2 ms, P95 166.3 ms); Wapiti 3.63/s (5.0 ms, 1352.0 ms); SciBERT 1.28/s (625.9 ms, 1882.9 ms). Baselines ran in amd64 CPU containers on an ARM host. A native-CPU 012 control achieved 11.37/s. Hardware/runtime differences prevent architecture-level speed claims. Wapiti had the fastest median and more empty-name responses. A dense list took 012 about 90 seconds because linking/alias candidates were classified individually.

## Bugs and improvement priorities

- Legacy SoMeSci/SoFAIR import offsets were invalid; repaired imports remain partially annotated.
- Softcite response encoding and leading-whitespace offsets needed explicit integration controls. No fuzzy matching was used to hide offset failures.
- 011/012 detector training includes contradictory positive-text/empty-label copies. Ablate this under matched training effort.
- The 012 detector can reject familiar software in certain contexts; [diagnostic](https://github.com/krushiraj/OSSoMeX/blob/v0.0.12/docs/research/detector-regression.md).
- Batch and constrain pair candidates while measuring true-link candidate recall.

## Evidence and reproduction

[Aggregate score records](benchmark-012-scores.json) are tracked without source text. The [experiment archive](https://github.com/krushiraj/OSSoMeX/blob/v0.0.12/experiments/README.md) contains the training and comparison Python helpers. Full raw responses, source documents, manifests, bootstrap receipts and timing controls remain in the local `reports/scibert-v2/threeway-012-2026-10-02/` artifact directory and must be obtained separately for complete reproduction. The source release alone is not the full evidence dataset.

[Roadmap](https://github.com/krushiraj/OSSoMeX/blob/v0.0.12/ROADMAP.md) describes the independent evaluation and training work needed before stronger claims.