File size: 8,609 Bytes
251b517 9fe20a2 251b517 9fe20a2 251b517 9fe20a2 2d64baa 9fe20a2 2d64baa 9fe20a2 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 | # Scientific Table Reconstruction evaluation
Version 1 evaluates image-to-table reconstruction on all 610 physical crops from 253 papers. Only each crop image is supplied to the model. Paper and table IDs are routing identifiers; domain is for stratified reporting. Gold must not be used in inference or tuning. This is an open-gold benchmark: disclose possible training overlap. There is no hidden test set.
## Install and run
Download this Space repository, enter evaluation/, and install the requirements in a dedicated Python environment.
pip install -r requirements.txt
python bench.py --predictions predictions.jsonl --metadata metadata.json --out my-results
The evaluator downloads the immutable dataset tag v1.0.1 by default. A local dataset directory with manifest.json and data/*.parquet can be passed with --dataset. Output must be a new or empty directory.
Import the dataset:
from datasets import load_dataset
tables = load_dataset("SciKnowOrg/Scientific-Table-Reconstruction", revision="v1.0.1", split="test")
for table in tables:
image = table["image"]
# Call your model with image only, then parse into rows of strings.
Create one JSON object per table, one line each, with exactly four fields:
{"paper_id":"P1","table_id":"page_005_table_001","status":"ok","rows":[["Material","Value"],["A","12"]]}
Use the exact public IDs and preserve every prediction, including headers and row labels. Every one of the 610 IDs is required. On failure, submit status "error" or "empty" with rows []. Explicitly empty predictions are scored as failures. Duplicate, unknown, missing IDs and non-string cells are rejected. A table may contain at most 10,000 rows and 1,000,000 cells; oversized responses must be reported as errors. Do not submit just the subset that worked.
Copy metadata.example.json to metadata.json and fill the required fields. Include immutable model/code revisions, full prompt, parser, generation settings and training-overlap disclosure. The required evaluation_mode is "physical-table"; input_mode is "image-only". The dataset fingerprint must match.
## Metrics and denominators
Scores are percentages (0–100). Both RMS and Exact Mapping Match report precision, recall and F1, with macro and micro aggregation.
The frozen metric is a standalone copy of Tabulus's DePlot-derived RMS implementation. Source and implementation SHA-256 hashes are recorded in the dataset manifest. No installed Tabulus checkout is imported. The shared mapping representation uses the first row as column headers, the first column as row labels, and maps their combined key to each body value. Whitespace is normalized, strings are lowercased and ragged grids padded for scoring; the released gold strings and CSV row widths remain unchanged.
RMS uses approximate normalized key/value credit, with text threshold 0.5 and relative-number threshold 0.1. Exact Mapping Match requires identical normalized mapping keys and values, without fuzzy or numeric tolerance. Neither metric is raw CSV byte equality or coordinate-wise cell accuracy. Prediction orientation is selected independently per metric: original versus transposed, highest per-table F1; the inherited tie rule selects transposed.
For a table, G and P are gold and selected predicted mapping counts, and C is matched credit. Precision=C/P, recall=C/G and F1=2C/(P+G), with the inherited empty-set conventions below. C is fractional for RMS and integer for exact mapping.
Macro precision, recall and F1 are the arithmetic averages of their respective per-table scores over ALL 610 crops, including failed tables with zero scores. Macro F1 is not the harmonic mean of macro precision and recall. Micro scores use the sums of C, G and P across all 610 crops before applying the formulas. Per-domain aggregates use the same rules on that domain's complete population.
Empty/error predictions receive zero macro contribution and retain their gold denominator for micro recall/F1. Inherited micro empty-set conventions: when P=0 and G>0, precision=100, recall=F1=0; when G=0 and P>0, precision=0, recall=100, F1=0; both empty yields 100 for all three. Thus zero-denominator precision is a convention, never evidence of successful extraction. Coverage and pooled counts accompany every score.
Header/row-label key collisions collapse to the last value in the inherited mapping dictionary. Literal " | " cell content can affect this parser. Zero-mapping gold tables also exist. These limitations are versioned and documented in manifest.json; do not compare this protocol to metrics with different structural canonicalization. Expert improvements require a new protocol/version, not silent gold edits.
Default ranking is macro RMS F1 across all 610 crops. All twelve measures and domain views remain available. These are reconstruction scores, not DOI, bibliography, citation resolution or Step 7 scores.
## Contribute a result
1. Run your model on all test images and save the raw predictions, parser, prompt and complete prediction JSONL.
2. Run bench.py and publish predictions, metadata.json, summary.json and per_table.jsonl alongside a reproducible inference recipe in a model/dataset repository.
3. Open a Community discussion in the leaderboard Space with those links and the immutable dataset/model/code revisions.
4. Maintainers independently rerun scoring from the submitted predictions, check provenance and overlap disclosures, then update data/leaderboard.json with the validated summary and evidence URL.
The static website performs no inference and accepts no unreviewed scores into its public rankings. Running models and scoring happen on the contributor's or maintainer's computer. The site has no paid hardware, persistent server, secrets or backend storage. Official entries are empty until submissions are reviewed. Gold/self-copy sanity checks must never be presented as model results.
To add a reviewed result after independently running bench.py:
python publish_result.py --summary my-results/summary.json --evidence-url https://huggingface.co/... --board ../data/leaderboard.json
This updates only the local board file; maintainers review and commit that change to the Space.
## Model type and comparison plots
Set model_type in submission metadata to one of:
- VLM: a general-purpose vision-language model.
- specialist: a neural model specialized for OCR, documents or table reconstruction.
- classical: a traditional OCR engine/pipeline.
- hybrid: a pipeline combining distinct model types; describe its components.
- unspecified: type is unknown or not yet supplied.
This descriptive classification appears beside the model name and determines the same point color in the plot. It does not alter scoring. Existing summaries without this field display as unspecified.
Optionally provide parameter_count_b as a positive JSON number, in billions of total model parameters, or null when unknown. For a mixture-of-experts model, use total parameters, not active parameters, and explain the choice. For multi-model pipelines, disclose the components and counting convention in your evidence. Do not infer size from a model name. Proprietary or classical models with unknown counts remain comparable on score-versus-score plots.
The bottom-of-page plotter supports model size (logarithmic or linear) versus any of the twelve scores, or any pair of scores, with domain and model-type filters, keyboard/touch point selection, and SVG export. Parameter counts are declared metadata, not latency or inference-cost measurements.
Load saved summary.json files for private local comparison. They must match this dataset fingerprint, evaluation mode and complete population. Files are parsed only in the browser, never uploaded. Local points are outlined and marked unreviewed; they do not enter the public leaderboard. A new file with the same local submission ID replaces that local preview. Clearing previews removes them from the page.
## Reproducibility and licenses
bench.py verifies the content fingerprint and frozen scoring implementation against the release manifest. It records hashes of canonicalized predictions and the metric implementation, dependency versions, sufficient per-table statistics, aggregate denominators and domain breakdowns.
Source: https://github.com/SciKnowOrg/tabulus
DePlot metric: https://github.com/google-research/google-research/tree/master/deplot
RMS is adapted under Apache License 2.0 (metric/LICENSE-APACHE-2.0). Other code is under the accompanying MIT LICENSE. Dataset licensing is stated in its dataset card.
|