secbert-finer139

sec-bert-base fine-tuned for financial named-entity recognition on FiNER-139 (279 IOB2 XBRL-tag labels). It marks numeric entity spans (revenue, debt, shares, percentages, etc.) in SEC-filing text and is used as the entity-tagging stage of a semantic-fingerprinting DLP pipeline.

• Base model: nlpaueb/sec-bert-base
• Task: token classification (NER), 279 labels, IOB2 tagging
• Dataset: FiNER-139 — 900,384 train / 112,494 dev / 108,378 test

Results

Entity-level micro P/R/F1 (seqeval-style IOB2 exact-span match).

Split	F1	Precision	Recall
Dev	0.8470	0.8338	0.8605
Test	0.8181	0.8018	0.8352

Per-epoch dev F1 during the final run: 0.8292 → 0.8428 → 0.8470.

Weakest test classes (candidates for future data augmentation): ShareBasedCompensation (0.35), OperatingLeasesRentExpenseNet (0.39), PaymentsToAcquireBusinessesNetOfCashAcquired (0.51), StockRepurchasedDuringPeriodShares, LeaseAndRentalExpense, TreasuryStockValueAcquiredCostMethod, AssetImpairmentCharges (0.60). Full per-class table is in ner_test_results.json.

Training procedure

Trained in two stages: an initial full-corpus run reached dev F1 0.7757 (2 epochs), then a warm-started resume (pinned to that checkpoint) continued for 3 more epochs on the full corpus. Before the resume, a short learning-rate sweep (4 trials, 1,500 steps each on a 50k subsample, all warm-started from the pinned checkpoint) selected the LR pair.

Hyperparameter	Value
Optimizer	AdamW (eps 1e-8, weight decay 0.01)
LR (backbone / head)	1e-5 / 2e-5 (sweep winner)
Layer-wise LR decay	0.95
Schedule	cosine, warmup ratio 0.05
Epochs (resume)	3
Batch size	16
Max sequence length	256
Classifier dropout	0.2
Label smoothing	0.05
Grad clip	1.0
Precision	fp16

Label–subword alignment: the first sub-token of a word receives the tag; continuation sub-tokens are masked with -100.

Usage

from transformers import AutoTokenizer, AutoModelForTokenClassification, pipeline

tok = AutoTokenizer.from_pretrained("shidey/secbert-finer139")
mdl = AutoModelForTokenClassification.from_pretrained("shidey/secbert-finer139")
ner = pipeline("token-classification", model=mdl, tokenizer=tok, aggregation_strategy="simple")

ner("Revenue increased to $2.1 billion, up 14% from the prior year.")

Intended use & limitations

• Intended: entity-span tagging of English SEC-filing / financial-report text, as a preprocessing stage for downstream relation extraction and document fingerprinting.
• Limitations: trained only on FiNER-139 (US-GAAP XBRL numeric tags, English SEC filings); rare and semantically similar tags (e.g. lease/stock-based-compensation variants) have lower F1; not intended for general-domain NER.

License

Inherits cc-by-sa-4.0 from the sec-bert-base backbone and the FiNER-139 dataset.