Archive the run after the AutoScientist Challenge

a3ad1fb verified 2 months ago

5.79 kB

---
license: llama3.3
base_model: meta-llama/Llama-3.3-70B-Instruct
library_name: peft
pipeline_tag: text-generation
tags:
  - lora
  - peft
  - chart-question-answering
  - data-visualization
  - misleading-charts
datasets:
  - rodriguescarson/adaption-chart-qa-with-axis-tricks
language:
  - en
---
> **Archived intermediate checkpoint.** This is the v5 run from the AutoScientist
> Challenge 2026: held-out 78, on-data 85. The adapter that was actually submitted,
> held-out 85.9, is at [polychart-shown-is-not-supported-lora](https://huggingface.co/rodriguescarson/polychart-shown-is-not-supported-lora).
> This copy is kept only so the run history survives the challenge.

# PolyChart — Shown Is Not Supported (LoRA)

A LoRA adapter for `Llama-3.3-70B-Instruct` that reads charts under a **measure-then-claim**
contract: a claim is made only when the underlying values support it, and the distance
between what a chart *shows* and what its data *supports* is reported as a graded quantity
rather than a yes/no label.

Trained on Adaption's AutoScientist for the AutoScientist Challenge 2026.

## Scores

| | Base | This adapter |
|---|---|---|
| On-dataset win rate | 15 | **85** |
| **Held-out category win rate** | 22 | **78** |

**Read 78, not 85.** The two measure different things and this is the most important
caveat on this model. The on-dataset figure is computed on the training distribution; the
held-out figure is computed on data the model never saw. In the study behind this adapter
they sometimes moved in *opposite* directions — one matched pair went 87.2 → 81.0
on-dataset while going 66.3 → 77.3 held-out. Optimising the displayed number can select
the weaker model.

## What is novel here: severity as a continuous target

Every released misleading-chart dataset we know of, including Misviz (ACL 2026), asks a
yes/no question: is this chart misleading? This model is trained on **Tufte's Lie Factor as
a continuous, signed target** — −1.0 where an inverted axis reverses a real trend, 0.017
where an expanded axis flattens one, 4.69 where a truncated baseline invents one. A binary
label collapses all three into the same bit.

Training composition is failure-weighted. Scoring the base model cell-by-cell over
question-type × distortion located the hard cells — continuous severity estimation, precise
value extraction under a distorted axis, and mechanism naming — and the 2,129-row dataset
was weighted toward them (severity 27%), with already-solved cells removed.

That targeting is measurable: the *uniform* version of this dataset on a 109B base
(Llama-4-Scout) reached 76 held-out; this failure-weighted version on a smaller 70B base
reached 78. Capacity sets the ceiling; composition decides where in that ceiling you land.

## Training

| | |
|---|---|
| Base | `Llama-3.3-70B-Instruct` |
| Method | LoRA, r=64, alpha=128 |
| Target modules | all linear |
| Epochs | 3 |
| Rows | 2,129, failure-weighted |
| Eval loss | 0.944 → 0.861 (no overfit) |

Every row passed an independent verifier sharing no code with the generator. It caught nine
silent failures in our own pipeline before shipping, four of them inside its own scaffolding.

## A retraction

**An earlier version of this card claimed "curation beats scale," on the basis that
expanding a seed from 1,882 to 21,882 rows dropped the held-out category from 76 to 72.
That claim is withdrawn.**

It rested on a single unpaired comparison. Repeating the test properly — paired within seed,
across 13 seeds and 176 fine-tuning jobs — reversed the conclusion: augmentation improved
the held-out score on **13 of 13 seeds, mean +14 points, two-sided sign test p = 0.0002**,
and the platform's advertised "+16 points from crossing 20,000 datapoints" reproduced at
**+17.4**.

Held-out difficulty spans 44.5 to 85.9 across those seeds, roughly three times the effect
being measured, so any cross-seed comparison mostly reports which seeds landed in which arm.
That is precisely the error the original claim made.

What survives is narrower and is stated above: curation decides *where within a base's
ceiling* you land, and expansion helps most where the seed under-covers its category. The
two are not competing prescriptions. A model whose thesis is *shown is not supported* owes
that test to its own claims first, which is why the retraction is recorded here rather than
quietly deleted.

## Intended use and limits

Chart-interpretation QA and misleading-encoding detection. Not a general-purpose assistant.

- One task family, one platform, one base model.
- Held-out evaluations fire on roughly a third of jobs and cannot be triggered on demand,
  so several seeds contribute one draw per arm against a metric with sd 3–5 noise.
  Direction is reported rather than a pooled effect size.
- The evaluation domain is assigned by the platform, not chosen. Across six purpose-built
  datasets, five were scored against `general` and none against its own topic — so a
  held-out number is interpretable only alongside the domain it was computed against.
- Inherits the biases and failure modes of `Llama-3.3-70B-Instruct`.

## Use

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = "meta-llama/Llama-3.3-70B-Instruct"
model = AutoModelForCausalLM.from_pretrained(base, device_map="auto")
model = PeftModel.from_pretrained(model, "rodriguescarson/polychart-shown-is-not-supported-lora")
tok = AutoTokenizer.from_pretrained(base)
```

## Links

- Client + platform findings: https://github.com/rodriguescarson/adaption-autoscientist-api
- Live Lie Factor demo: https://lie-factor.carsonrodrigues.com
- Datasets: https://huggingface.co/rodriguescarson

## License

Llama 3.3 Community License, inherited from the base model.

	---
	license: llama3.3
	base_model: meta-llama/Llama-3.3-70B-Instruct
	library_name: peft
	pipeline_tag: text-generation
	tags:
	- lora
	- peft
	- chart-question-answering
	- data-visualization
	- misleading-charts
	datasets:
	- rodriguescarson/adaption-chart-qa-with-axis-tricks
	language:
	- en
	---

	> Archived intermediate checkpoint. This is the v5 run from the AutoScientist
	> Challenge 2026: held-out 78, on-data 85. The adapter that was actually submitted,
	> held-out 85.9, is at [polychart-shown-is-not-supported-lora](https://huggingface.co/rodriguescarson/polychart-shown-is-not-supported-lora).
	> This copy is kept only so the run history survives the challenge.

	# PolyChart — Shown Is Not Supported (LoRA)

	A LoRA adapter for `Llama-3.3-70B-Instruct` that reads charts under a measure-then-claim
	contract: a claim is made only when the underlying values support it, and the distance
	between what a chart shows and what its data supports is reported as a graded quantity
	rather than a yes/no label.

	Trained on Adaption's AutoScientist for the AutoScientist Challenge 2026.

	## Scores

	\| \| Base \| This adapter \|
	\|---\|---\|---\|
	\| On-dataset win rate \| 15 \| 85 \|
	\| Held-out category win rate \| 22 \| 78 \|

	Read 78, not 85. The two measure different things and this is the most important
	caveat on this model. The on-dataset figure is computed on the training distribution; the
	held-out figure is computed on data the model never saw. In the study behind this adapter
	they sometimes moved in opposite directions — one matched pair went 87.2 → 81.0
	on-dataset while going 66.3 → 77.3 held-out. Optimising the displayed number can select
	the weaker model.

	## What is novel here: severity as a continuous target

	Every released misleading-chart dataset we know of, including Misviz (ACL 2026), asks a
	yes/no question: is this chart misleading? This model is trained on **Tufte's Lie Factor as
	a continuous, signed target** — −1.0 where an inverted axis reverses a real trend, 0.017
	where an expanded axis flattens one, 4.69 where a truncated baseline invents one. A binary
	label collapses all three into the same bit.

	Training composition is failure-weighted. Scoring the base model cell-by-cell over
	question-type × distortion located the hard cells — continuous severity estimation, precise
	value extraction under a distorted axis, and mechanism naming — and the 2,129-row dataset
	was weighted toward them (severity 27%), with already-solved cells removed.

	That targeting is measurable: the uniform version of this dataset on a 109B base
	(Llama-4-Scout) reached 76 held-out; this failure-weighted version on a smaller 70B base
	reached 78. Capacity sets the ceiling; composition decides where in that ceiling you land.

	## Training

	\| \| \|
	\|---\|---\|
	\| Base \| `Llama-3.3-70B-Instruct` \|
	\| Method \| LoRA, r=64, alpha=128 \|
	\| Target modules \| all linear \|
	\| Epochs \| 3 \|
	\| Rows \| 2,129, failure-weighted \|
	\| Eval loss \| 0.944 → 0.861 (no overfit) \|

	Every row passed an independent verifier sharing no code with the generator. It caught nine
	silent failures in our own pipeline before shipping, four of them inside its own scaffolding.

	## A retraction

	**An earlier version of this card claimed "curation beats scale," on the basis that
	expanding a seed from 1,882 to 21,882 rows dropped the held-out category from 76 to 72.
	That claim is withdrawn.**

	It rested on a single unpaired comparison. Repeating the test properly — paired within seed,
	across 13 seeds and 176 fine-tuning jobs — reversed the conclusion: augmentation improved
	the held-out score on 13 of 13 seeds, mean +14 points, two-sided sign test p = 0.0002,
	and the platform's advertised "+16 points from crossing 20,000 datapoints" reproduced at
	+17.4.

	Held-out difficulty spans 44.5 to 85.9 across those seeds, roughly three times the effect
	being measured, so any cross-seed comparison mostly reports which seeds landed in which arm.
	That is precisely the error the original claim made.

	What survives is narrower and is stated above: curation decides *where within a base's
	ceiling* you land, and expansion helps most where the seed under-covers its category. The
	two are not competing prescriptions. A model whose thesis is shown is not supported owes
	that test to its own claims first, which is why the retraction is recorded here rather than
	quietly deleted.

	## Intended use and limits

	Chart-interpretation QA and misleading-encoding detection. Not a general-purpose assistant.

	- One task family, one platform, one base model.
	- Held-out evaluations fire on roughly a third of jobs and cannot be triggered on demand,
	so several seeds contribute one draw per arm against a metric with sd 3–5 noise.
	Direction is reported rather than a pooled effect size.
	- The evaluation domain is assigned by the platform, not chosen. Across six purpose-built
	datasets, five were scored against `general` and none against its own topic — so a
	held-out number is interpretable only alongside the domain it was computed against.
	- Inherits the biases and failure modes of `Llama-3.3-70B-Instruct`.

	## Use

	```python
	from peft import PeftModel
	from transformers import AutoModelForCausalLM, AutoTokenizer

	base = "meta-llama/Llama-3.3-70B-Instruct"
	model = AutoModelForCausalLM.from_pretrained(base, device_map="auto")
	model = PeftModel.from_pretrained(model, "rodriguescarson/polychart-shown-is-not-supported-lora")
	tok = AutoTokenizer.from_pretrained(base)
	```

	## Links

	- Client + platform findings: https://github.com/rodriguescarson/adaption-autoscientist-api
	- Live Lie Factor demo: https://lie-factor.carsonrodrigues.com
	- Datasets: https://huggingface.co/rodriguescarson

	## License

	Llama 3.3 Community License, inherited from the base model.