From table images
to structured knowledge.
Compare OCR pipelines and vision-language models (VLMs) on reconstructing scientific tables. One table image in, structured rows and cells out—across five research domains.
The leaderboard
Relative Mapping Similarity · approximate matching with text and numeric tolerances.
| # | Model / method | Model type ⓘ | Macro equal weight per table | Micro pooled mappings | Evaluation mode | Coverage | ||||
|---|---|---|---|---|---|---|---|---|---|---|
Ready for the first result.No reviewed model submissions yet. | ||||||||||
Every submission covers the full test set. Empty and failed extractions stay in the denominator. Domain views are diagnostic; the official ranking uses all 610 tables.
Real tables. Across science.
Original table crops paired with gold rows of cell strings. Each example includes its domain, paper ID and table ID.
domainpaper_idtable_idimagegold_rowsTwo ways to measure
the same reconstruction.
RMS gives partial credit. Exact Mapping Match requires exact normalized mappings. Both use a frozen, versioned Tabulus evaluator adapted from DePlot.
Read the complete evaluation protocol ↗What counts as a mapping?
The first row provides column headers and the first column provides row labels. Their combined key maps to each body value. Whitespace and case are normalized before comparing mappings. This measures associations, not raw CSV bytes.
How do macro and micro differ?
Macro averages each table’s precision, recall and F1 separately. Micro pools matched credit and gold/predicted mapping counts, then computes the three scores. Failed tables receive zero macro scores and keep their gold counts for micro recall and F1.
What does “exact” mean here?
Exact Mapping Match uses identical normalized keys and values. RMS uses approximate text matching (threshold 0.5) and relative numeric matching (threshold 0.1). Each selects the better of original and transposed predictions using per-table F1. Evaluation mode is physical-table; model input is image-only.
What should I know about the gold?
This is an open-gold public test set. Do not use gold in inference or tuning; disclose training overlap. Source gold is preserved, including merged-cell and repeated-label conventions. The inherited mapping parser collapses duplicate keys and has a few delimiter/zero-mapping edge cases. The manifest and protocol document these limitations and empty-set conventions.
Add your model
to the leaderboard.
Inference runs on your own hardware. The dataset, evaluator and static leaderboard are open and free to access.
Open a submission discussion ↗- 01
Load the images
Import the v1.0.1 test split from SciKnowOrg/Scientific-Table-Reconstruction. Give the model only each table image.
- 02
Reconstruct and score
Save all 610 predictions as rows of strings, with explicit failures. Run the pinned CPU evaluator for all twelve measures.
Evaluator & prediction format ↗ - 03
Share the evidence
Publish predictions, prompt, model/code revisions, settings and training-overlap disclosure. Maintainers rescore before adding an entry.
Download metadata template ↓
Size, accuracy, and trade-offs.
Compare model size with reconstruction quality, or choose any two scores. Hover, tap, or focus a point for its details.
Ready to compare the first models.
Reviewed submissions will appear here. You can also preview your own saved evaluation summaries below.
Load summary.json files from the evaluator. Files stay in your browser and are not submitted or added to the rankings.
All scores range from 0 to 100; higher is better. Each metric axis includes its aggregation. Models without declared parameter counts are omitted only from size comparisons. Parameter counts are not a measure of latency or compute cost.