Text Classification
Transformers
Safetensors
Korean
xlm-roberta
reranker
cross-encoder
dart
opendart
financial-disclosure
korean
record-linkage
convertible-bond
text-embeddings-inference
Instructions to use ThakiCloud/EvidenceLink-DART-Reranker-KO with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ThakiCloud/EvidenceLink-DART-Reranker-KO with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="ThakiCloud/EvidenceLink-DART-Reranker-KO")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSequenceClassification tokenizer = AutoTokenizer.from_pretrained("ThakiCloud/EvidenceLink-DART-Reranker-KO") model = AutoModelForSequenceClassification.from_pretrained("ThakiCloud/EvidenceLink-DART-Reranker-KO", device_map="auto") - Notebooks
- Google Colab
- Kaggle
|
Download README.md from ThakiCloud/EvidenceLink-DART-Reranker-KO: direct link, hf CLI and curl.
- Browser
- Download file 11.6 kB
-
https://huggingface.co/ThakiCloud/EvidenceLink-DART-Reranker-KO/resolve/main/README.md
- Command line
-
hf download hf://ThakiCloud/EvidenceLink-DART-Reranker-KO/README.md
-
curl -L -o README.md https://huggingface.co/ThakiCloud/EvidenceLink-DART-Reranker-KO/resolve/main/README.md
11.6 kB
| license: apache-2.0 | |
| base_model: BAAI/bge-reranker-v2-m3 | |
| base_model_relation: finetune | |
| library_name: transformers | |
| pipeline_tag: text-classification | |
| language: | |
| - ko | |
| tags: | |
| - reranker | |
| - cross-encoder | |
| - dart | |
| - opendart | |
| - financial-disclosure | |
| - korean | |
| - record-linkage | |
| - convertible-bond | |
| # EvidenceLink-DART-Reranker-KO | |
| π°π· νκ΅μ΄ μΉ΄λ: [MODEL_CARD.ko.md](./MODEL_CARD.ko.md) | |
| > **PASS_V24 (G1βG5 passed) β the verdict of one pre-registered experiment (`prereg_E_v2_4`), not a general claim.** | |
| > Sealed test = 648 correction filings from 79 issuers never seen in training. Gold links come from DART's own | |
| > correction chain, not from human review. Natural "no match" cases were too few (25 < 30), so gate G6 is | |
| > **UNDERPOWERED** and generalization of the NONE decision to natural cases is **not established**. | |
| A Korean cross-encoder that links a DART **correction filing** for a convertible-bond issuance decision | |
| (`[κΈ°μ¬μ μ ] μ£Όμμ¬νλ³΄κ³ μ(μ νμ¬μ±κΆλ°νκ²°μ )`) back to the **original filing (the right bond series/round)** it amends, | |
| or answers **NONE** when the original is not in the candidate set. It is a full fine-tune of | |
| [`BAAI/bge-reranker-v2-m3`](https://huggingface.co/BAAI/bge-reranker-v2-m3) trained only on public OpenDART filings. | |
| The hard part is not finding the issuer. It is choosing among several filings of the **same issuer** (different series, | |
| same day, near-identical tables) and refusing to link when none of them is right. | |
| ## At a glance | |
| | | | | |
| |---|---| | |
| | Task | correction filing β one of K candidate originals, or NONE | | |
| | Scope | DART convertible-bond issuance decisions (μ νμ¬μ±κΆλ°νκ²°μ ) and their corrections, filed 2022β2026 | | |
| | Sealed test (CLEAN, n=499) | Top-1 **0.978** vs BM25 0.7375 Β· rule T0 0.7174 | | |
| | NONE precision / recall (CLEAN) | **1.0 / 0.9308** (BM25 0.5548 / 0.6615) | | |
| | Same-issuer wrong-link rate (CLEAN) | **0.0246** (BM25 0.1387 Β· T0 0.1432) | | |
| | Paired issuer-cluster bootstrap (10,000) | Top-1 vs BM25 **+0.2387**, 95% CI [0.1645, 0.3091] | | |
| | Decision rule | argmax score over candidates; NONE if the top logit < **β8.4375** (frozen on dev before the sealed run) | | |
| | Weights | `model.safetensors` sha256 `a402de3dc80fa8d45b42551ef381f063ec997fdba7d5d6e31462108fd48f6c19` | | |
| | Training data | OpenDART public filings only. No AI-Hub data. No filing text is shipped here. | | |
| ## What it does | |
| | Output | Meaning | | |
| |---|---| | |
| | `pred = <candidate id>` | the candidate original filing this correction amends (highest score, score β₯ threshold) | | |
| | `pred = None` | NONE: no candidate scored above the frozen threshold | | |
| | `scores` | raw logit per candidate (pointwise; not calibrated probabilities) | | |
| ## Quick start | |
| ```bash | |
| pip install torch transformers huggingface_hub | |
| hf download ThakiCloud/EvidenceLink-DART-Reranker-KO --local-dir evidencelink | |
| cd evidencelink | |
| python evidencelink.py --selftest # decoder + card format checks (no model) | |
| python evidencelink.py --model . --example examples/synthetic_query.json | |
| ``` | |
| ```python | |
| from evidencelink import load, query_card, candidate_card, link, dart_zip_to_text | |
| tok, model = load(".") # fp32, CPU or GPU | |
| q = query_card(corp_name, correction_text) # text from dart_zip_to_text(<OpenDART document.xml zip>) | |
| cands = [{"id": rcept_no, "card": candidate_card(corp_name, rcept_dt, text)} | |
| for rcept_no, rcept_dt, text in originals] # order: receipt number descending | |
| print(link(tok, model, q, cands)) # {"pred": ... or None, "top_score": ..., "scores": {...}} | |
| ``` | |
| **Use the input format in `evidencelink.py`.** The model was trained on a specific card: a `[μμ½]` summary line | |
| (series, principal, conversion price, coupon, yield, refixing floor) followed by the masked filing text | |
| (first-submission date and 14-digit receipt numbers are replaced by `[MASK]`/`[RCEPT]`). Raw filing text gives | |
| different scores, and the frozen threshold no longer applies. | |
| **Decode OpenDART documents with the included decoder (`dart_zip_to_text`).** Filings before mid-2022 are cp949. | |
| Reading them as UTF-8 with `errors="replace"` silently destroys the fields the model relies on (see Limitations). | |
| `examples/synthetic_query.json` is a fictional issuer generated from a template. It is not a DART filing. | |
| ## What is in this package | |
| | File | Role | | |
| |---|---| | |
| | `model.safetensors` | fine-tuned weights (fp32), sha256 `a402de3dc80fa8d4β¦` | | |
| | `config.json`, `tokenizer.json`, `tokenizer_config.json` | model config and tokenizer (unchanged from the base) | | |
| | `evidencelink.py` | decoder v2, field extraction, card builder, scoring, NONE rule | | |
| | `examples/synthetic_query.json` | synthetic query + 3 candidates | | |
| | `LICENSE`, `NOTICE` | Apache-2.0, attribution | | |
| ## How the candidate set is built (evaluation protocol) | |
| For each correction filing: up to 10 original filings of the same issuer received within the previous 3 years | |
| (most recent first), plus up to 3 filings of other issuers received within Β±10 days (hard negatives). | |
| Candidates are ordered by receipt number, descending. For 0.9499 of CLEAN queries the true original falls inside this | |
| window; the remaining 25 queries are the natural NONE cases (original filed more than 3 years earlier). In another | |
| 105 CLEAN queries the true original was removed on purpose to create NONE cases (NONE_SYNTH below). | |
| ## Sealed test results (sealed4, prereg_E_v2_4) | |
| 648 correction-filing queries from 79 issuers. Issuers in sealed4 were excluded from training and dev. | |
| **CLEAN** (n=499; 369 linked, 130 NONE) drops queries whose card shows the receipt date of any candidate, so a model | |
| cannot shortcut through a visible date. CLEAN is the primary set for all gates. | |
| ### CLEAN (primary, n=499) | |
| | Arm | Top-1 | Linked acc | NONE precision | NONE recall | Same-issuer wrong-link rate | | |
| |---|---|---|---|---|---| | |
| | **This model (H3, seed 0)** | **0.978** | **0.9946** | **1.0** | **0.9308** | **0.0246** | | |
| | BM25 (card vs card, NONE threshold on dev) | 0.7375 | 0.7642 | 0.5548 | 0.6615 | 0.1387 | | |
| | T0 rule (field-overlap count, NONE threshold on dev) | 0.7174 | 0.6856 | 0.5899 | 0.8077 | 0.1432 | | |
| | bge-m3 bi-encoder, zero-shot | 0.5431 | 0.5339 | 0.3289 | 0.5692 | 0.1723 | | |
| All arms were measured by us, in the same sealed job, on the same candidate sets. | |
| ### Confidence intervals (issuer-cluster paired bootstrap, 10,000 resamples, seed 1109) | |
| | Difference in Top-1 | Mean | 95% CI | | |
| |---|---|---| | |
| | This model β BM25, CLEAN | +0.2387 | [0.1645, 0.3091] | | |
| | This model β T0, CLEAN | +0.2573 | [0.1635, 0.3497] | | |
| | This model β BM25, ALL | +0.2263 | [0.1616, 0.2907] | | |
| | This model β T0, ALL | +0.2633 | [0.1747, 0.3499] | | |
| ALL (n=648) Top-1: this model 0.9722 Β· BM25 0.7454 Β· T0 0.7068. | |
| ### Pre-registered gates | |
| | Gate | Condition (CLEAN) | Result | | |
| |---|---|---| | |
| | G1 | Top-1 vs BM25, bootstrap CI lower bound > 0 | pass | | |
| | G2 | same-issuer wrong-link rate β€ 0.05 and below BM25 and T0 | pass (0.0246) | | |
| | G3 | NONE precision and recall β₯ 0.80 | pass (1.0 / 0.9308) | | |
| | G4 | linked accuracy β₯ 0.90 | pass (0.9946) | | |
| | G5 | ALL Top-1 β₯ BM25 | pass (0.9722 vs 0.7454) | | |
| | G6 | natural-NONE recall β₯ 0.80, evaluated only if n β₯ 30 | **UNDERPOWERED** (n=25) β excluded from the verdict | | |
| ### By filing year (CLEAN Top-1) | |
| | Year | n | This model | BM25 | T0 | | |
| |---|---|---|---|---| | |
| | 2022 | 100 | 0.97 | 0.57 | 0.68 | | |
| | 2023 | 67 | 0.9552 | 0.7463 | 0.7612 | | |
| | 2024 | 140 | 0.9857 | 0.8143 | 0.7286 | | |
| | 2025 | 109 | 0.9817 | 0.7798 | 0.7064 | | |
| | 2026 | 83 | 0.988 | 0.747 | 0.7229 | | |
| ### NONE by stratum (CLEAN) | |
| | Stratum | n | This model | BM25 | T0 | | |
| |---|---|---|---|---| | |
| | NONE_SYNTH (true original removed from the set) | 105 | 0.9333 | 0.6 | 0.7714 | | |
| | NONE_NATURAL (true original outside the 3-year window) | 25 | 0.92 | 0.92 | 0.96 | | |
| On the 25 natural NONE cases the model is **not** better than the rule baseline (T0 0.96). With n=25 none of these | |
| numbers separates the arms. | |
| ## Limitations β do not overstate | |
| - **Verdict scope.** PASS_V24 is the verdict of `prereg_E_v2_4` on sealed4 only. It does not certify other filing types, | |
| other years, or customer documents (contracts, spreadsheets). | |
| - **G6 UNDERPOWERED.** Only 25 natural NONE cases exist in sealed4 (target β₯ 30). Most NONE cases (105 of 130 in CLEAN) | |
| were built by removing the true original from the candidate set. Whether the NONE decision generalizes to natural | |
| "no original in range" cases is **not established**. | |
| - **Gold is not human-reviewed.** The gold original comes from DART's correction chain (the filing family DART itself | |
| links when a correction is submitted). It is rule/chain based. Training labels are weaker still: a rule that matches | |
| the first-submission date written in the correction (that date is masked in the model input). On CLEAN queries where | |
| the weak rule decides, it agrees with gold 0.8868 of the time. | |
| - **Dev has almost no 2022β2023 data.** Dev = 93 queries from 11 issuers (2022: 0, 2023: 1, 2024: 26, 2025: 31, | |
| 2026: 35). Epoch and the NONE threshold were selected on that dev. The 2022β2023 sealed numbers above are therefore | |
| out-of-distribution for model selection. | |
| - **Truncation.** All checked pairs (4,000 train, 432 dev) exceed 1024 tokens and are truncated (`longest_first`). The `[μμ½]` | |
| summary line is placed first so it survives; filing text deep in the form is not seen. | |
| - **Encoding dependency (decoder v2).** OpenDART documents before mid-2022 are cp949. An earlier pipeline decoded them as | |
| UTF-8 with replacement characters, which damaged 1,427 cached documents and collapsed field extraction for 2022β2023 | |
| (the cause of the previous NO_GO on sealed3). With decoder v2, query series/principal extraction on sealed4 is | |
| 0.948β0.992 per year. If you feed text decoded another way, expect lower accuracy. | |
| - **sealed3 re-score is a report-only diagnostic.** Re-scoring the earlier H2 model (not this checkpoint) on the | |
| already-exposed sealed3 set after the decoder fix raised its CLEAN Top-1 from 0.6571 to 0.9455 on the same 385 | |
| queries. That set was exposed before, so it is evidence about the input defect, not a pass criterion. | |
| - **Candidate window.** Only originals filed within 3 years before the correction (10 most recent of the same issuer) | |
| are candidates. An older original can never be linked; the correct output is then NONE. | |
| - **Seed.** Two seeds were trained; seed 1 tied on dev (0.9085) and seed 0 was chosen by the pre-registered tie rule | |
| before sealed evaluation. Seed 1 was not evaluated on sealed4. | |
| ## Training settings (from pre-registration) | |
| | | | | |
| |---|---| | |
| | Base | `BAAI/bge-reranker-v2-m3`, full fine-tune, pointwise BCE | | |
| | Hyperparameters | max_len 1024, batch 16, 2 epochs (best epoch 1), lr 2e-5, positive weight 3 | | |
| | Negatives | all other candidates in the set (same-issuer other series + other-issuer near-date) | | |
| | Train | 1,430 queries, 205 issuers, 2022β2026 | | |
| | Dev | 93 queries, 11 issuers; selection = macro average of per-year Top-1 (0.9085) | | |
| | Sealed4 | 648 queries, 79 issuers, evaluated once | | |
| ## Data source | |
| OpenDART (Financial Supervisory Service, Korea) public disclosure filings: convertible-bond issuance decisions and | |
| their corrections, accessed through the OpenDART API. This repository contains **no** filing text, extracted fields, | |
| queries, candidate sets, or labels. **No AI-Hub data was used.** | |
| ## License | |
| Apache-2.0, matching the base model `BAAI/bge-reranker-v2-m3` (Apache-2.0). See `LICENSE` and `NOTICE`. | |