Text Classification
Transformers
Safetensors
Korean
xlm-roberta
reranker
cross-encoder
dart
opendart
financial-disclosure
korean
record-linkage
convertible-bond
text-embeddings-inference
Instructions to use ThakiCloud/EvidenceLink-DART-Reranker-KO with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ThakiCloud/EvidenceLink-DART-Reranker-KO with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="ThakiCloud/EvidenceLink-DART-Reranker-KO")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSequenceClassification tokenizer = AutoTokenizer.from_pretrained("ThakiCloud/EvidenceLink-DART-Reranker-KO") model = AutoModelForSequenceClassification.from_pretrained("ThakiCloud/EvidenceLink-DART-Reranker-KO", device_map="auto") - Notebooks
- Google Colab
- Kaggle
File size: 11,565 Bytes
039683c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 | ---
license: apache-2.0
base_model: BAAI/bge-reranker-v2-m3
base_model_relation: finetune
library_name: transformers
pipeline_tag: text-classification
language:
- ko
tags:
- reranker
- cross-encoder
- dart
- opendart
- financial-disclosure
- korean
- record-linkage
- convertible-bond
---
# EvidenceLink-DART-Reranker-KO
π°π· νκ΅μ΄ μΉ΄λ: [MODEL_CARD.ko.md](./MODEL_CARD.ko.md)
> **PASS_V24 (G1βG5 passed) β the verdict of one pre-registered experiment (`prereg_E_v2_4`), not a general claim.**
> Sealed test = 648 correction filings from 79 issuers never seen in training. Gold links come from DART's own
> correction chain, not from human review. Natural "no match" cases were too few (25 < 30), so gate G6 is
> **UNDERPOWERED** and generalization of the NONE decision to natural cases is **not established**.
A Korean cross-encoder that links a DART **correction filing** for a convertible-bond issuance decision
(`[κΈ°μ¬μ μ ] μ£Όμμ¬νλ³΄κ³ μ(μ νμ¬μ±κΆλ°νκ²°μ )`) back to the **original filing (the right bond series/round)** it amends,
or answers **NONE** when the original is not in the candidate set. It is a full fine-tune of
[`BAAI/bge-reranker-v2-m3`](https://huggingface.co/BAAI/bge-reranker-v2-m3) trained only on public OpenDART filings.
The hard part is not finding the issuer. It is choosing among several filings of the **same issuer** (different series,
same day, near-identical tables) and refusing to link when none of them is right.
## At a glance
| | |
|---|---|
| Task | correction filing β one of K candidate originals, or NONE |
| Scope | DART convertible-bond issuance decisions (μ νμ¬μ±κΆλ°νκ²°μ ) and their corrections, filed 2022β2026 |
| Sealed test (CLEAN, n=499) | Top-1 **0.978** vs BM25 0.7375 Β· rule T0 0.7174 |
| NONE precision / recall (CLEAN) | **1.0 / 0.9308** (BM25 0.5548 / 0.6615) |
| Same-issuer wrong-link rate (CLEAN) | **0.0246** (BM25 0.1387 Β· T0 0.1432) |
| Paired issuer-cluster bootstrap (10,000) | Top-1 vs BM25 **+0.2387**, 95% CI [0.1645, 0.3091] |
| Decision rule | argmax score over candidates; NONE if the top logit < **β8.4375** (frozen on dev before the sealed run) |
| Weights | `model.safetensors` sha256 `a402de3dc80fa8d45b42551ef381f063ec997fdba7d5d6e31462108fd48f6c19` |
| Training data | OpenDART public filings only. No AI-Hub data. No filing text is shipped here. |
## What it does
| Output | Meaning |
|---|---|
| `pred = <candidate id>` | the candidate original filing this correction amends (highest score, score β₯ threshold) |
| `pred = None` | NONE: no candidate scored above the frozen threshold |
| `scores` | raw logit per candidate (pointwise; not calibrated probabilities) |
## Quick start
```bash
pip install torch transformers huggingface_hub
hf download ThakiCloud/EvidenceLink-DART-Reranker-KO --local-dir evidencelink
cd evidencelink
python evidencelink.py --selftest # decoder + card format checks (no model)
python evidencelink.py --model . --example examples/synthetic_query.json
```
```python
from evidencelink import load, query_card, candidate_card, link, dart_zip_to_text
tok, model = load(".") # fp32, CPU or GPU
q = query_card(corp_name, correction_text) # text from dart_zip_to_text(<OpenDART document.xml zip>)
cands = [{"id": rcept_no, "card": candidate_card(corp_name, rcept_dt, text)}
for rcept_no, rcept_dt, text in originals] # order: receipt number descending
print(link(tok, model, q, cands)) # {"pred": ... or None, "top_score": ..., "scores": {...}}
```
**Use the input format in `evidencelink.py`.** The model was trained on a specific card: a `[μμ½]` summary line
(series, principal, conversion price, coupon, yield, refixing floor) followed by the masked filing text
(first-submission date and 14-digit receipt numbers are replaced by `[MASK]`/`[RCEPT]`). Raw filing text gives
different scores, and the frozen threshold no longer applies.
**Decode OpenDART documents with the included decoder (`dart_zip_to_text`).** Filings before mid-2022 are cp949.
Reading them as UTF-8 with `errors="replace"` silently destroys the fields the model relies on (see Limitations).
`examples/synthetic_query.json` is a fictional issuer generated from a template. It is not a DART filing.
## What is in this package
| File | Role |
|---|---|
| `model.safetensors` | fine-tuned weights (fp32), sha256 `a402de3dc80fa8d4β¦` |
| `config.json`, `tokenizer.json`, `tokenizer_config.json` | model config and tokenizer (unchanged from the base) |
| `evidencelink.py` | decoder v2, field extraction, card builder, scoring, NONE rule |
| `examples/synthetic_query.json` | synthetic query + 3 candidates |
| `LICENSE`, `NOTICE` | Apache-2.0, attribution |
## How the candidate set is built (evaluation protocol)
For each correction filing: up to 10 original filings of the same issuer received within the previous 3 years
(most recent first), plus up to 3 filings of other issuers received within Β±10 days (hard negatives).
Candidates are ordered by receipt number, descending. For 0.9499 of CLEAN queries the true original falls inside this
window; the remaining 25 queries are the natural NONE cases (original filed more than 3 years earlier). In another
105 CLEAN queries the true original was removed on purpose to create NONE cases (NONE_SYNTH below).
## Sealed test results (sealed4, prereg_E_v2_4)
648 correction-filing queries from 79 issuers. Issuers in sealed4 were excluded from training and dev.
**CLEAN** (n=499; 369 linked, 130 NONE) drops queries whose card shows the receipt date of any candidate, so a model
cannot shortcut through a visible date. CLEAN is the primary set for all gates.
### CLEAN (primary, n=499)
| Arm | Top-1 | Linked acc | NONE precision | NONE recall | Same-issuer wrong-link rate |
|---|---|---|---|---|---|
| **This model (H3, seed 0)** | **0.978** | **0.9946** | **1.0** | **0.9308** | **0.0246** |
| BM25 (card vs card, NONE threshold on dev) | 0.7375 | 0.7642 | 0.5548 | 0.6615 | 0.1387 |
| T0 rule (field-overlap count, NONE threshold on dev) | 0.7174 | 0.6856 | 0.5899 | 0.8077 | 0.1432 |
| bge-m3 bi-encoder, zero-shot | 0.5431 | 0.5339 | 0.3289 | 0.5692 | 0.1723 |
All arms were measured by us, in the same sealed job, on the same candidate sets.
### Confidence intervals (issuer-cluster paired bootstrap, 10,000 resamples, seed 1109)
| Difference in Top-1 | Mean | 95% CI |
|---|---|---|
| This model β BM25, CLEAN | +0.2387 | [0.1645, 0.3091] |
| This model β T0, CLEAN | +0.2573 | [0.1635, 0.3497] |
| This model β BM25, ALL | +0.2263 | [0.1616, 0.2907] |
| This model β T0, ALL | +0.2633 | [0.1747, 0.3499] |
ALL (n=648) Top-1: this model 0.9722 Β· BM25 0.7454 Β· T0 0.7068.
### Pre-registered gates
| Gate | Condition (CLEAN) | Result |
|---|---|---|
| G1 | Top-1 vs BM25, bootstrap CI lower bound > 0 | pass |
| G2 | same-issuer wrong-link rate β€ 0.05 and below BM25 and T0 | pass (0.0246) |
| G3 | NONE precision and recall β₯ 0.80 | pass (1.0 / 0.9308) |
| G4 | linked accuracy β₯ 0.90 | pass (0.9946) |
| G5 | ALL Top-1 β₯ BM25 | pass (0.9722 vs 0.7454) |
| G6 | natural-NONE recall β₯ 0.80, evaluated only if n β₯ 30 | **UNDERPOWERED** (n=25) β excluded from the verdict |
### By filing year (CLEAN Top-1)
| Year | n | This model | BM25 | T0 |
|---|---|---|---|---|
| 2022 | 100 | 0.97 | 0.57 | 0.68 |
| 2023 | 67 | 0.9552 | 0.7463 | 0.7612 |
| 2024 | 140 | 0.9857 | 0.8143 | 0.7286 |
| 2025 | 109 | 0.9817 | 0.7798 | 0.7064 |
| 2026 | 83 | 0.988 | 0.747 | 0.7229 |
### NONE by stratum (CLEAN)
| Stratum | n | This model | BM25 | T0 |
|---|---|---|---|---|
| NONE_SYNTH (true original removed from the set) | 105 | 0.9333 | 0.6 | 0.7714 |
| NONE_NATURAL (true original outside the 3-year window) | 25 | 0.92 | 0.92 | 0.96 |
On the 25 natural NONE cases the model is **not** better than the rule baseline (T0 0.96). With n=25 none of these
numbers separates the arms.
## Limitations β do not overstate
- **Verdict scope.** PASS_V24 is the verdict of `prereg_E_v2_4` on sealed4 only. It does not certify other filing types,
other years, or customer documents (contracts, spreadsheets).
- **G6 UNDERPOWERED.** Only 25 natural NONE cases exist in sealed4 (target β₯ 30). Most NONE cases (105 of 130 in CLEAN)
were built by removing the true original from the candidate set. Whether the NONE decision generalizes to natural
"no original in range" cases is **not established**.
- **Gold is not human-reviewed.** The gold original comes from DART's correction chain (the filing family DART itself
links when a correction is submitted). It is rule/chain based. Training labels are weaker still: a rule that matches
the first-submission date written in the correction (that date is masked in the model input). On CLEAN queries where
the weak rule decides, it agrees with gold 0.8868 of the time.
- **Dev has almost no 2022β2023 data.** Dev = 93 queries from 11 issuers (2022: 0, 2023: 1, 2024: 26, 2025: 31,
2026: 35). Epoch and the NONE threshold were selected on that dev. The 2022β2023 sealed numbers above are therefore
out-of-distribution for model selection.
- **Truncation.** All checked pairs (4,000 train, 432 dev) exceed 1024 tokens and are truncated (`longest_first`). The `[μμ½]`
summary line is placed first so it survives; filing text deep in the form is not seen.
- **Encoding dependency (decoder v2).** OpenDART documents before mid-2022 are cp949. An earlier pipeline decoded them as
UTF-8 with replacement characters, which damaged 1,427 cached documents and collapsed field extraction for 2022β2023
(the cause of the previous NO_GO on sealed3). With decoder v2, query series/principal extraction on sealed4 is
0.948β0.992 per year. If you feed text decoded another way, expect lower accuracy.
- **sealed3 re-score is a report-only diagnostic.** Re-scoring the earlier H2 model (not this checkpoint) on the
already-exposed sealed3 set after the decoder fix raised its CLEAN Top-1 from 0.6571 to 0.9455 on the same 385
queries. That set was exposed before, so it is evidence about the input defect, not a pass criterion.
- **Candidate window.** Only originals filed within 3 years before the correction (10 most recent of the same issuer)
are candidates. An older original can never be linked; the correct output is then NONE.
- **Seed.** Two seeds were trained; seed 1 tied on dev (0.9085) and seed 0 was chosen by the pre-registered tie rule
before sealed evaluation. Seed 1 was not evaluated on sealed4.
## Training settings (from pre-registration)
| | |
|---|---|
| Base | `BAAI/bge-reranker-v2-m3`, full fine-tune, pointwise BCE |
| Hyperparameters | max_len 1024, batch 16, 2 epochs (best epoch 1), lr 2e-5, positive weight 3 |
| Negatives | all other candidates in the set (same-issuer other series + other-issuer near-date) |
| Train | 1,430 queries, 205 issuers, 2022β2026 |
| Dev | 93 queries, 11 issuers; selection = macro average of per-year Top-1 (0.9085) |
| Sealed4 | 648 queries, 79 issuers, evaluated once |
## Data source
OpenDART (Financial Supervisory Service, Korea) public disclosure filings: convertible-bond issuance decisions and
their corrections, accessed through the OpenDART API. This repository contains **no** filing text, extracted fields,
queries, candidate sets, or labels. **No AI-Hub data was used.**
## License
Apache-2.0, matching the base model `BAAI/bge-reranker-v2-m3` (Apache-2.0). See `LICENSE` and `NOTICE`.
|