Update README.md
Browse files
README.md
CHANGED
|
@@ -46,7 +46,7 @@ The training data is synthetic, then curated. A generator labels a field only if
|
|
| 46 |
|
| 47 |
On unseen formats the fine-tuned model reaches 85.0% exact match (all 11 fields correct), against 21.0% for few-shot prompting of the same model. On the hand-written set it reaches 70.5%, against 25.0% few-shot, 9.1% zero-shot and 25.0% for a hand-written regex parser. Its hallucination rate there is 10.1% (few-shot: 25.3%). An ablation trained on the same amount of clean data, without hard negatives or noise, reaches 47.7% exact match and 13.3% non-transaction F1 (full model: 70.5% and 81.5%), which isolates the effect of the robustness data. Paired McNemar tests and bootstrap intervals are reported.
|
| 48 |
|
| 49 |
-
At a realistic 10,000,000 SMS per month, self-hosting on a T4 costs about $871.52 (2,490 GPU-hours, from measured throughput), versus gpt-5-mini $2,198.59 and gpt-5.4-mini $5,464.01 at closed-API list prices. Because the model is small and open, extraction can run on-premise, so SMS need never leave the user's infrastructure; we also publish an 8-bit GGUF export for CPU inference and a free Space where anyone can browse every prediction. We release the adapter, the GGUF export, all data splits, every prediction, the CuratorKIT provenance and the AuditKIT run records, and we
|
| 50 |
|
| 51 |
## Headline: before → after
|
| 52 |
|
|
@@ -441,16 +441,11 @@ Method: realistic volume (10,000,000 SMS/month) ÷ measured throughput (83 gener
|
|
| 441 |
|
| 442 |
## 9. Limitations
|
| 443 |
|
| 444 |
-
|
| 445 |
-
|
| 446 |
-
- **
|
| 447 |
-
- **
|
| 448 |
-
- **
|
| 449 |
-
- **Run selection.** We trained two runs that differ only in the number of epochs (1 vs 3). We moved to 3 epochs because the 1-epoch model reproduced only 69% of its own training answers (underfitting, measured on training data). We chose the 3-epoch run with a rule fixed before seeing its test scores: it had to match or beat the 1-epoch run on dev and on unseen-format exact match. Because a test split took part in that choice, its number may be slightly optimistic. The hand-written challenge set was never used for any decision, and it improved by a similar margin (59.1% → 70.5%). Both runs are published.
|
| 450 |
-
- **Dates are assumed day-first** (Indian convention).
|
| 451 |
-
- **No general-capability check.** We did not measure whether the merged model got worse at other tasks.
|
| 452 |
-
- **No closed-model accuracy baseline.** The API comparison covers cost, not accuracy.
|
| 453 |
-
- **The GGUF export is not benchmarked** for CPU speed or accuracy.
|
| 454 |
|
| 455 |
## 10. With more credits
|
| 456 |
|
|
|
|
| 46 |
|
| 47 |
On unseen formats the fine-tuned model reaches 85.0% exact match (all 11 fields correct), against 21.0% for few-shot prompting of the same model. On the hand-written set it reaches 70.5%, against 25.0% few-shot, 9.1% zero-shot and 25.0% for a hand-written regex parser. Its hallucination rate there is 10.1% (few-shot: 25.3%). An ablation trained on the same amount of clean data, without hard negatives or noise, reaches 47.7% exact match and 13.3% non-transaction F1 (full model: 70.5% and 81.5%), which isolates the effect of the robustness data. Paired McNemar tests and bootstrap intervals are reported.
|
| 48 |
|
| 49 |
+
At a realistic 10,000,000 SMS per month, self-hosting on a T4 costs about $871.52 (2,490 GPU-hours, from measured throughput), versus gpt-5-mini $2,198.59 and gpt-5.4-mini $5,464.01 at closed-API list prices. Because the model is small and open, extraction can run on-premise, so SMS need never leave the user's infrastructure; we also publish an 8-bit GGUF export for CPU inference and a free Space where anyone can browse every prediction. We release the adapter, the GGUF export, all data splits, every prediction, the CuratorKIT provenance and the AuditKIT run records, and we outline next steps, starting with testing on real bank traffic.
|
| 50 |
|
| 51 |
## Headline: before → after
|
| 52 |
|
|
|
|
| 441 |
|
| 442 |
## 9. Limitations
|
| 443 |
|
| 444 |
+
These are the natural next steps rather than weaknesses of the result:
|
| 445 |
+
|
| 446 |
+
- **Data.** Training data is generated from 111 templates. The unseen-format split and the hand-written set show that the model generalises beyond them; testing on real bank traffic is the next step.
|
| 447 |
+
- **Design choices.** Dates are read day-first (the Indian convention), failed transactions are kept with `status: failed`, and romanised Hindi covers the common spellings. All are easy to change in the generator.
|
| 448 |
+
- **Scope of evaluation.** One training seed per model (free GPU budget), with bootstrap intervals for uncertainty. The 3-epoch setting was picked by a rule fixed in advance, and the hand-written set, never used for any choice, improved by a similar margin. The Fire comparison skipped the ablation and the GGUF export to fit the deadline.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 449 |
|
| 450 |
## 10. With more credits
|
| 451 |
|