Update the model card

99116d2 verified about 1 month ago

9.01 kB

---
license: other
license_name: nutrient-commercial
pipeline_tag: image-text-to-text
language:
  - en
tags:
  - chart-understanding
  - chart-to-table
  - structured-extraction
  - document-ai
  - vision-language-model
metrics:
  - f1
model-index:
  - name: Nutrient Chart Parsing
    results:
      - task:
          type: image-text-to-text
          name: Chart parsing
        dataset:
          type: nutrientdocs/chart-parsing-benchmark
          name: Chart Parsing Benchmark
          split: test
        metrics:
          - type: f1
            name: Mean per-chart cell F1 (3% tolerance)
            value: 0.888
            verified: false
          - type: f1
            name: Mean per-chart cell F1 (strict)
            value: 0.797
            verified: false
---
# Nutrient Chart Parsing · _commercial_

**Read a chart image and return its data and structure as one JSON object.** The Nutrient chart-parsing
model takes a rendered chart (bar, line, pie, scatter, area, box, bubble, heatmap, histogram, treemap,
funnel, radar, violin, stem, step, candlestick, combo) and returns the title, chart type, data table,
value representation, encoding, axes, series, legend, and data-label state in a single pass. Values are
what a reader can recover from the pixels: printed labels verbatim, shares for pie and donut charts,
axis-scaled estimates otherwise.

The runnable weights are commercial and are not downloadable from this repository. This public page is the
product specification and scorecard; see [License and access](#license-and-access) to evaluate or deploy it.

- 🧪 **Try it:** [`nutrientdocs/chart-parsing-demo`](https://huggingface.co/spaces/nutrientdocs/chart-parsing-demo)
- 🏆 **Leaderboard:** [`nutrientdocs/chart-parsing-leaderboard`](https://huggingface.co/spaces/nutrientdocs/chart-parsing-leaderboard)
- 📊 **Benchmark:** [`nutrientdocs/chart-parsing-benchmark`](https://huggingface.co/datasets/nutrientdocs/chart-parsing-benchmark)

## Headline result

Systems are ranked by **mean per-chart cell F1 at 3% tolerance** on the frozen 2,500-chart benchmark:
every chart's data cells are matched by row identity and column, a value counts when it is
within 3% of the chart's value range (3 percentage points on share charts, exact when the chart prints
data labels), and the per-chart F1 is averaged so every chart counts once. **Strict F1** demands values
within 0.1%. Every entry sees the same charts and the same prompt.

| System | F1 · 3% | Strict F1 | F1 · 5% | Shape | Type | Title | sec/chart |
|---|---:|---:|---:|---:|---:|---:|---:|
| **Nutrient Chart Parsing** | **0.888** | 0.797 | 0.897 | 0.963 | 0.983 | 0.989 | 0.27 |
| Gemini 3.8 Flash | 0.710 | 0.640 | 0.713 | 0.796 | 0.960 | 0.998 | — |
| Claude Opus 5 | 0.697 | 0.617 | 0.703 | 0.792 | 0.967 | 0.998 | — |
| GPT-6 Astra | 0.675 | 0.628 | 0.676 | 0.802 | 0.964 | 0.997 | — |
| Gemini 3.5 Flash Lite | 0.663 | 0.600 | 0.672 | 0.771 | 0.947 | 0.992 | — |
| Claude Sonnet 5 | 0.657 | 0.586 | 0.664 | 0.741 | 0.973 | 0.998 | — |
| GPT-5.6 Terra | 0.630 | 0.569 | 0.634 | 0.709 | 0.960 | 0.998 | — |
| Gemini 3.1 Pro | 0.628 | 0.567 | 0.633 | 0.726 | 0.950 | 0.996 | — |
| Claude Haiku 4.5 | 0.570 | 0.469 | 0.596 | 0.708 | 0.943 | 0.995 | — |
| GPT-5.6 Sol | 0.566 | 0.514 | 0.569 | 0.658 | 0.954 | 0.997 | — |
| Qwen3.8-27B | 0.524 | 0.445 | 0.531 | 0.625 | 0.793 | 0.826 | 2.80 |
| Qwen3-VL-4B-Instruct | 0.518 | 0.443 | 0.536 | 0.623 | 0.908 | 0.984 | 0.30 |
| Qwen3.5-4B | 0.169 | 0.143 | 0.171 | 0.200 | 0.290 | 0.312 | 1.00 |

The Nutrient model reaches **0.888**, ahead of the strongest cloud model here, **Gemini 3.8 Flash (0.710)**,
and it recovers the right table shape on **96.3%** of charts against 80% for the best cloud model. Its lead
is in the data cells, not in the labels: cloud models read titles and chart types about as well. Seconds per
chart are measured on one NVIDIA A100 80GB with vLLM at 64-way concurrency and compare only within that
setup; provider APIs give no comparable latency. Full per-field and per-type breakdowns are on the
[leaderboard](https://huggingface.co/spaces/nutrientdocs/chart-parsing-leaderboard).

### Result by chart type

Mean per-chart cell F1 for the Nutrient model at the three tolerance levels, with the number of benchmark
charts of each type.

| Chart type | n | Strict | 3% | 5% |
|---|---:|---:|---:|---:|
| line | 552 | 0.831 | 0.924 | 0.928 |
| bar | 463 | 0.857 | 0.940 | 0.946 |
| pie | 249 | 0.762 | 0.765 | 0.765 |
| scatter | 235 | 0.819 | 0.859 | 0.860 |
| area | 224 | 0.822 | 0.889 | 0.912 |
| box | 171 | 0.753 | 0.980 | 0.991 |
| bubble | 164 | 0.665 | 0.750 | 0.761 |
| heatmap | 100 | 0.868 | 0.878 | 0.898 |
| histogram | 54 | 0.875 | 0.927 | 0.946 |
| treemap | 54 | 0.341 | 0.961 | 0.994 |
| combo | 51 | 0.784 | 0.937 | 0.945 |
| funnel | 49 | 0.953 | 0.953 | 0.953 |
| radar | 46 | 0.549 | 0.651 | 0.682 |
| violin | 33 | 0.691 | 0.958 | 0.980 |
| stem | 25 | 0.772 | 0.844 | 0.846 |
| step | 15 | 0.983 | 0.983 | 0.983 |
| candlestick | 15 | 0.693 | 0.807 | 0.813 |

Per-field accuracy on the same set: chart type 0.983, title 0.989, x and y axis labels 0.987 and 0.986,
axis scales 0.975 and 0.985, legend visibility 0.990, data-label visibility 0.989, column types 0.989,
column roles 0.988, axis bounds 0.835. Set F1 over encoding pairs 0.974 and over series pairs 0.993.

## Output

One JSON object per chart. `data.rows` is the table; every other key describes how the chart shows it.

```json
{
  "title": "Programming Language Popularity",
  "chartType": "pie",
  "data": {
    "columns": [
      {"name": "Language", "type": "categorical", "role": "category", "unit": null},
      {"name": "Popularity", "type": "numeric", "role": "value", "unit": null}
    ],
    "rows": [["Python", 21.6], ["Java", 19.4], ["C++", 17.7]]
  },
  "valueRepresentation": {"mode": "percentage_of_total", "format": "0.0%"},
  "encoding": {"category": "Language", "value": "Popularity", "color": "Language"},
  "axes": null,
  "series": [{"name": "Popularity", "field": "Popularity"}],
  "legend": {"visible": false, "position": null, "entries": null},
  "dataLabels": {"visible": true, "format": "percentage"}
}
```

- `chartType` is one of `bar, line, area, pie, scatter, bubble, histogram, box, violin, heatmap, treemap, funnel, radar, candlestick, step, stem, combo`.
- Column `type` is `categorical`, `numeric`, or `temporal`; column `role` is one of `category, value, series, x, y, z, size, color`.
- `valueRepresentation.mode` is `percentage_of_total` for pie, donut, treemap, and 100%-stacked charts, otherwise `absolute`.
- `axes` holds `{label, unit, scale, min, max}` per axis with the visible limits, or `null` for charts without axes.
- Any field the image does not support is `null`.

The `data` block converts to CSV, HTML, or a spreadsheet with plain code; the model is not asked to emit
those formats.

## Intended use and limits

- **Use it for:** turning rendered chart images (exports from plotting libraries, report figures, dashboard
  screenshots) into a data table and a description of how the chart shows it, for indexing, analysis,
  re-plotting, or conversion to spreadsheets.
- **Values are visual estimates.** When a chart prints no data labels the model reads values off the axis
  scale; expect errors within a few percent of the value range, and check the strict versus 3% columns
  above for how much that costs per chart type. Pie and donut charts return shares of the total, never the
  underlying absolute values.
- **Weaker chart types:** bubble (size legends are hard to invert), radar, and candlestick charts, and axis
  bounds in general (0.747 field accuracy). Pie charts read printed shares well but estimating unlabeled
  wedges is imprecise.
- **Validate downstream.** The output is well-formed JSON in practice, but parse it defensively and check
  that `data.rows` has the width `data.columns` declares before feeding it to automated pipelines.

## License and access

The model weights are offered under a **commercial Nutrient license** and are not published here. The
benchmark charts derive from ChartGen-200K, CC BY 4.0.
The benchmark is public and reproducible through
[`nutrientdocs/chart-parsing-benchmark`](https://huggingface.co/datasets/nutrientdocs/chart-parsing-benchmark).

> ### 📩 Get access
>
> Nutrient Chart Parsing is commercial. To evaluate or deploy it on-prem, with your documents staying in
> your infrastructure, **contact Nutrient:
> [nutrient.io/contact-sales](https://www.nutrient.io/contact-sales/).**

## About the author

<a href="https://nutrient.io/">
  <img src="https://avatars2.githubusercontent.com/u/1527679?v=3&s=200" height="80" />
</a>

This project is maintained and funded by [Nutrient](https://nutrient.io/) - The deterministic document infrastructure enterprises run their highest-stakes workflows on: replayable output, clear exceptions, and full audit trails on the messy, regulated documents where AI alone breaks.

	---
	license: other
	license_name: nutrient-commercial
	pipeline_tag: image-text-to-text
	language:
	- en
	tags:
	- chart-understanding
	- chart-to-table
	- structured-extraction
	- document-ai
	- vision-language-model
	metrics:
	- f1
	model-index:
	- name: Nutrient Chart Parsing
	results:
	- task:
	type: image-text-to-text
	name: Chart parsing
	dataset:
	type: nutrientdocs/chart-parsing-benchmark
	name: Chart Parsing Benchmark
	split: test
	metrics:
	- type: f1
	name: Mean per-chart cell F1 (3% tolerance)
	value: 0.888
	verified: false
	- type: f1
	name: Mean per-chart cell F1 (strict)
	value: 0.797
	verified: false
	---

	# Nutrient Chart Parsing · _commercial_

	Read a chart image and return its data and structure as one JSON object. The Nutrient chart-parsing
	model takes a rendered chart (bar, line, pie, scatter, area, box, bubble, heatmap, histogram, treemap,
	funnel, radar, violin, stem, step, candlestick, combo) and returns the title, chart type, data table,
	value representation, encoding, axes, series, legend, and data-label state in a single pass. Values are
	what a reader can recover from the pixels: printed labels verbatim, shares for pie and donut charts,
	axis-scaled estimates otherwise.

	The runnable weights are commercial and are not downloadable from this repository. This public page is the
	product specification and scorecard; see [License and access](#license-and-access) to evaluate or deploy it.

	- 🧪 Try it: [`nutrientdocs/chart-parsing-demo`](https://huggingface.co/spaces/nutrientdocs/chart-parsing-demo)
	- 🏆 Leaderboard: [`nutrientdocs/chart-parsing-leaderboard`](https://huggingface.co/spaces/nutrientdocs/chart-parsing-leaderboard)
	- 📊 Benchmark: [`nutrientdocs/chart-parsing-benchmark`](https://huggingface.co/datasets/nutrientdocs/chart-parsing-benchmark)

	## Headline result

	Systems are ranked by mean per-chart cell F1 at 3% tolerance on the frozen 2,500-chart benchmark:
	every chart's data cells are matched by row identity and column, a value counts when it is
	within 3% of the chart's value range (3 percentage points on share charts, exact when the chart prints
	data labels), and the per-chart F1 is averaged so every chart counts once. Strict F1 demands values
	within 0.1%. Every entry sees the same charts and the same prompt.

	\| System \| F1 · 3% \| Strict F1 \| F1 · 5% \| Shape \| Type \| Title \| sec/chart \|
	\|---\|---:\|---:\|---:\|---:\|---:\|---:\|---:\|
	\| Nutrient Chart Parsing \| 0.888 \| 0.797 \| 0.897 \| 0.963 \| 0.983 \| 0.989 \| 0.27 \|
	\| Gemini 3.8 Flash \| 0.710 \| 0.640 \| 0.713 \| 0.796 \| 0.960 \| 0.998 \| — \|
	\| Claude Opus 5 \| 0.697 \| 0.617 \| 0.703 \| 0.792 \| 0.967 \| 0.998 \| — \|
	\| GPT-6 Astra \| 0.675 \| 0.628 \| 0.676 \| 0.802 \| 0.964 \| 0.997 \| — \|
	\| Gemini 3.5 Flash Lite \| 0.663 \| 0.600 \| 0.672 \| 0.771 \| 0.947 \| 0.992 \| — \|
	\| Claude Sonnet 5 \| 0.657 \| 0.586 \| 0.664 \| 0.741 \| 0.973 \| 0.998 \| — \|
	\| GPT-5.6 Terra \| 0.630 \| 0.569 \| 0.634 \| 0.709 \| 0.960 \| 0.998 \| — \|
	\| Gemini 3.1 Pro \| 0.628 \| 0.567 \| 0.633 \| 0.726 \| 0.950 \| 0.996 \| — \|
	\| Claude Haiku 4.5 \| 0.570 \| 0.469 \| 0.596 \| 0.708 \| 0.943 \| 0.995 \| — \|
	\| GPT-5.6 Sol \| 0.566 \| 0.514 \| 0.569 \| 0.658 \| 0.954 \| 0.997 \| — \|
	\| Qwen3.8-27B \| 0.524 \| 0.445 \| 0.531 \| 0.625 \| 0.793 \| 0.826 \| 2.80 \|
	\| Qwen3-VL-4B-Instruct \| 0.518 \| 0.443 \| 0.536 \| 0.623 \| 0.908 \| 0.984 \| 0.30 \|
	\| Qwen3.5-4B \| 0.169 \| 0.143 \| 0.171 \| 0.200 \| 0.290 \| 0.312 \| 1.00 \|

	The Nutrient model reaches 0.888, ahead of the strongest cloud model here, Gemini 3.8 Flash (0.710),
	and it recovers the right table shape on 96.3% of charts against 80% for the best cloud model. Its lead
	is in the data cells, not in the labels: cloud models read titles and chart types about as well. Seconds per
	chart are measured on one NVIDIA A100 80GB with vLLM at 64-way concurrency and compare only within that
	setup; provider APIs give no comparable latency. Full per-field and per-type breakdowns are on the
	[leaderboard](https://huggingface.co/spaces/nutrientdocs/chart-parsing-leaderboard).

	### Result by chart type

	Mean per-chart cell F1 for the Nutrient model at the three tolerance levels, with the number of benchmark
	charts of each type.

	\| Chart type \| n \| Strict \| 3% \| 5% \|
	\|---\|---:\|---:\|---:\|---:\|
	\| line \| 552 \| 0.831 \| 0.924 \| 0.928 \|
	\| bar \| 463 \| 0.857 \| 0.940 \| 0.946 \|
	\| pie \| 249 \| 0.762 \| 0.765 \| 0.765 \|
	\| scatter \| 235 \| 0.819 \| 0.859 \| 0.860 \|
	\| area \| 224 \| 0.822 \| 0.889 \| 0.912 \|
	\| box \| 171 \| 0.753 \| 0.980 \| 0.991 \|
	\| bubble \| 164 \| 0.665 \| 0.750 \| 0.761 \|
	\| heatmap \| 100 \| 0.868 \| 0.878 \| 0.898 \|
	\| histogram \| 54 \| 0.875 \| 0.927 \| 0.946 \|
	\| treemap \| 54 \| 0.341 \| 0.961 \| 0.994 \|
	\| combo \| 51 \| 0.784 \| 0.937 \| 0.945 \|
	\| funnel \| 49 \| 0.953 \| 0.953 \| 0.953 \|
	\| radar \| 46 \| 0.549 \| 0.651 \| 0.682 \|
	\| violin \| 33 \| 0.691 \| 0.958 \| 0.980 \|
	\| stem \| 25 \| 0.772 \| 0.844 \| 0.846 \|
	\| step \| 15 \| 0.983 \| 0.983 \| 0.983 \|
	\| candlestick \| 15 \| 0.693 \| 0.807 \| 0.813 \|

	Per-field accuracy on the same set: chart type 0.983, title 0.989, x and y axis labels 0.987 and 0.986,
	axis scales 0.975 and 0.985, legend visibility 0.990, data-label visibility 0.989, column types 0.989,
	column roles 0.988, axis bounds 0.835. Set F1 over encoding pairs 0.974 and over series pairs 0.993.

	## Output

	One JSON object per chart. `data.rows` is the table; every other key describes how the chart shows it.

	```json
	{
	"title": "Programming Language Popularity",
	"chartType": "pie",
	"data": {
	"columns": [
	{"name": "Language", "type": "categorical", "role": "category", "unit": null},
	{"name": "Popularity", "type": "numeric", "role": "value", "unit": null}
	],
	"rows": [["Python", 21.6], ["Java", 19.4], ["C++", 17.7]]
	},
	"valueRepresentation": {"mode": "percentage_of_total", "format": "0.0%"},
	"encoding": {"category": "Language", "value": "Popularity", "color": "Language"},
	"axes": null,
	"series": [{"name": "Popularity", "field": "Popularity"}],
	"legend": {"visible": false, "position": null, "entries": null},
	"dataLabels": {"visible": true, "format": "percentage"}
	}
	```

	- `chartType` is one of `bar, line, area, pie, scatter, bubble, histogram, box, violin, heatmap, treemap, funnel, radar, candlestick, step, stem, combo`.
	- Column `type` is `categorical`, `numeric`, or `temporal`; column `role` is one of `category, value, series, x, y, z, size, color`.
	- `valueRepresentation.mode` is `percentage_of_total` for pie, donut, treemap, and 100%-stacked charts, otherwise `absolute`.
	- `axes` holds `{label, unit, scale, min, max}` per axis with the visible limits, or `null` for charts without axes.
	- Any field the image does not support is `null`.

	The `data` block converts to CSV, HTML, or a spreadsheet with plain code; the model is not asked to emit
	those formats.

	## Intended use and limits

	- Use it for: turning rendered chart images (exports from plotting libraries, report figures, dashboard
	screenshots) into a data table and a description of how the chart shows it, for indexing, analysis,
	re-plotting, or conversion to spreadsheets.
	- Values are visual estimates. When a chart prints no data labels the model reads values off the axis
	scale; expect errors within a few percent of the value range, and check the strict versus 3% columns
	above for how much that costs per chart type. Pie and donut charts return shares of the total, never the
	underlying absolute values.
	- Weaker chart types: bubble (size legends are hard to invert), radar, and candlestick charts, and axis
	bounds in general (0.747 field accuracy). Pie charts read printed shares well but estimating unlabeled
	wedges is imprecise.
	- Validate downstream. The output is well-formed JSON in practice, but parse it defensively and check
	that `data.rows` has the width `data.columns` declares before feeding it to automated pipelines.

	## License and access

	The model weights are offered under a commercial Nutrient license and are not published here. The
	benchmark charts derive from ChartGen-200K, CC BY 4.0.
	The benchmark is public and reproducible through
	[`nutrientdocs/chart-parsing-benchmark`](https://huggingface.co/datasets/nutrientdocs/chart-parsing-benchmark).

	> ### 📩 Get access
	>
	> Nutrient Chart Parsing is commercial. To evaluate or deploy it on-prem, with your documents staying in
	> your infrastructure, **contact Nutrient:
	> [nutrient.io/contact-sales](https://www.nutrient.io/contact-sales/).**

	## About the author

	<a href="https://nutrient.io/">
	<img src="https://avatars2.githubusercontent.com/u/1527679?v=3&s=200" height="80" />
	</a>

	This project is maintained and funded by [Nutrient](https://nutrient.io/) - The deterministic document infrastructure enterprises run their highest-stakes workflows on: replayable output, clear exceptions, and full audit trails on the messy, regulated documents where AI alone breaks.