sandeeprdy1729 commited on
Commit
9d22b9a
·
verified ·
1 Parent(s): 29d58aa

benchmarks update (v1): ParseBench/olmOCR-Bench measured sections

Browse files
Files changed (1) hide show
  1. README.md +31 -0
README.md CHANGED
@@ -51,6 +51,37 @@ final policy. Every P0 defect found in the v1 audit is addressed; the audit trai
51
  - Real financial tables from FinTabNet.c (`bsmock/FinTabNet.c`, HTML ground truth, CDLA-Permissive-2.0).
52
  - Compute: Kaggle free tier (T4), LoRA, 1583 SFT optimizer steps + 303 GRPO groups. Targets: 6330 (~533 tokens mean).
53
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
54
  ## Results (read the caveats)
55
  All numbers below were measured by us on **held-out synthetic data and a seeded ParseBench-proxy sample on a T4 with Transformers** (no vLLM).
56
  They are **not** the official ParseBench score and are **not comparable** to H100/vLLM numbers anywhere below. Headline deltas carry bootstrap 95% CIs.
 
51
  - Real financial tables from FinTabNet.c (`bsmock/FinTabNet.c`, HTML ground truth, CDLA-Permissive-2.0).
52
  - Compute: Kaggle free tier (T4), LoRA, 1583 SFT optimizer steps + 303 GRPO groups. Targets: 6330 (~533 tokens mean).
53
 
54
+ <!-- THRISUL-BENCH:BEGIN -->
55
+ ## Official benchmark results (thrisul-parser)
56
+
57
+ Measured 2026-10-31 on Kaggle 2×T4 (Transformers, fp16, greedy) by this benchmark notebook. Run tag: `v1`. Coverage: ParseBench 60 pages/dimension (official total 2,078) · olmOCR-Bench 105 pages (official total = all unique bench pages) · OmniDocBench: packaged · NVIDIA suites: IndicVisionBench=pending, MOSCAR=pending.
58
+
59
+ > Sampling disclosure: these runs cover a **seeded stratified subset** unless the config covered every page. Partial coverage is labeled — never quoted as the full-suite official score. Base-model deltas on identical pages are in `results.json`.
60
+
61
+ ### ParseBench (official evaluator, sampled)
62
+
63
+ | group | score |
64
+ |---|---|
65
+ | chart | 0.206 |
66
+ | table | 0.168 |
67
+ | text_formatting | 0.037 |
68
+ | text_content | 0.491 |
69
+ | text | 0.091 |
70
+ | overall_rule_avg | 0.145 |
71
+ | **mean of table/chart/text** | **0.155** |
72
+
73
+
74
+ ### Published context (ParseBench leaderboard head, self-reported)
75
+
76
+ - LlamaParse Cost Effective: 78.02
77
+ - LlamaParse Agentic: 83.07
78
+ - LlamaParse Agentic Plus: 86.61
79
+ - OpenAI GPT-5 Mini (Reasoning Minimal): 46.83
80
+ - OpenAI GPT-5 Mini (Reasoning Medium): 51.52
81
+ - Anthropic Haiku 4.5 (Disable Thinking): 45.17
82
+ - Anthropic Haiku 4.5 (Thinking): 53.12
83
+ - Google Gemini 3 Flash (Thinking Minimal): 71.04
84
+ <!-- THRISUL-BENCH:END -->
85
  ## Results (read the caveats)
86
  All numbers below were measured by us on **held-out synthetic data and a seeded ParseBench-proxy sample on a T4 with Transformers** (no vLLM).
87
  They are **not** the official ParseBench score and are **not comparable** to H100/vLLM numbers anywhere below. Headline deltas carry bootstrap 95% CIs.