datacard-ci / PROJECT_GUIDE.md

Add v0.2 claim compiler with frozen corpus and comparative evaluation

9bd89bf verified about 1 month ago

5.29 kB

# DataCard CI — dataset claim compiler

## v0.2 experiment

Use `datacard_ci_v2_colab.ipynb` for the new 1,200-example training workflow,
development-based checkpoint selection and a fresh 100-example test comparing
the base model, v0.1 and v0.2. Training is disabled by default and runs one complete pass on a GPU
by default (at most two passes are supported). The pinned v0.1 comparison adapter requires repository read access
while the Hugging Face repository is private. See `DATASET_V2.md` for provenance,
limitations and evaluation policy. These are AI-authored synthetic labels, not
independent human annotations.

After a completed run, `predict_v2.py` loads the saved adapter and translates one
quote plus a split/column schema into an unapproved JSON proposal. The supported
checks cover exact row counts, within-split ID uniqueness and cross-split ID
overlap. `real_data_demo.py` exercises the verifier on attributed UCI Wine Quality
fixtures with artificial IDs and deliberate corruption. It does not measure the
model's performance on real documentation.

## Original v0.1 starter

Local prototype for translating individual dataset claims into reviewed checks.
MIT licensed. No publishing, paid API, account token, Drive mount, or hosted service required.

## Start in Google Colab

Upload `datacard_ci_colab.ipynb` using **File → Upload notebook**. Run all cells
on **CPU** first. The notebook contains its own project source, so no GitHub repo
or separate file upload is required. It runs the tests, creates an evidence report,
installs pinned Hugging Face packages, and runs a tiny adapter smoke test.
PyTorch comes from Colab's runtime and is deliberately not replaced.
The setup removes Colab's optional `torchao` package from the temporary runtime:
its preinstalled version conflicts with PEFT, and plain LoRA does not require it.
Training runs in a fresh subprocess to avoid stale imports after package updates.

The smoke test makes a tiny random Qwen3 model, performs two LoRA optimizer steps,
checks that weights changed, then saves and reloads the adapter and compares logits.
**This tests training infrastructure, not useful language understanding.**

For a short pretrained-model fine-tuning experiment, select a **T4 GPU**,
rerun the setup cells after the runtime changes, then set `RUN_TRAINING = True`
in the last training cell. It downloads `Qwen/Qwen3-0.6B`, records the exact Hub
revision, compares baseline and tuned outputs, and trains for 80 optimizer steps (two passes over training data).
This uses LoRA with float32 weights and float16 mixed precision with batch size 1, accumulation 4, and a 512-token
limit; no quantization or custom CUDA extension. GPU availability is not guaranteed.
Never buy compute for the initial experiment. Keep `RUN_TRAINING = False` for CPU.

Outputs stay in the runtime under `artifacts/`. Download the archive using the final
cell before ending the session. No automatic Hub upload or telemetry integration.

## Local use

Python 3.10+; the verifier has no third-party dependencies.

```sh
python3 -m unittest -v
python3 -m pip install -e .
```

Training additionally requires PyTorch and `requirements-colab.txt`.

```python
from datacard_ci import demo_inputs, verify
report = verify(*demo_inputs())
print([r['status'] for r in report['results']])
# FAIL, FAIL, FAIL, ABSTAIN, ABSTAIN — deliberately broken demo data
```

## What this version does

- Row count, unique column values, and disjoint values across two CSV splits.
- Exact source character spans, original card text, and SHA-256 of scanned bytes.
- Explicit unapproved proposals, manual review, and aggregate evidence without raw IDs.
- Original synthetic training data: 160 training and 40 held-out examples.
- All 40 held-out examples used for quick before/after generation evaluation.
- Two operator families plus count checks; no generated code execution.

## Limits

This is a starter, not the complete product in the original proposal. It does not
fetch Hugging Face datasets, extract claims automatically, prove legal/social facts,
implement CI/badges, or authenticate a maintainer's approval. A human selects quotes
and reviews classifications and parameters. A malicious or mistaken classification
can still misrepresent prose; the verifier checks the approved specification only.

`full_supplied_files` means all supplied CSV bytes, **not** proof of remote dataset
completeness, origin, or revision. Remote revision is null in verifier reports.
Inputs are limited to 10 splits, 2 MB and 20,000 rows per split. Empty splits are
inconclusive (ERROR), including zero-row count claims. Values are literal strings:
`01` differs from `1`; no case or whitespace normalization. Blank IDs are ERROR.
Malformed CSV is rejected before a report is created. Input prose is preserved in
reports, so use synthetic/public data in shareable runs. The synthetic benchmark is
small and repetitive; its scores cannot establish real-world reliability.

## Sources

- [Colab resource limits](https://research.google.com/colaboratory/faq.html)
- [Qwen3-0.6B model card](https://huggingface.co/Qwen/Qwen3-0.6B)
- [PEFT LoRA](https://huggingface.co/docs/peft/developer_guides/lora)

Run `python3 build_notebook.py` after changing source files to rebuild the notebook.

	# DataCard CI — dataset claim compiler

	## v0.2 experiment

	Use `datacard_ci_v2_colab.ipynb` for the new 1,200-example training workflow,
	development-based checkpoint selection and a fresh 100-example test comparing
	the base model, v0.1 and v0.2. Training is disabled by default and runs one complete pass on a GPU
	by default (at most two passes are supported). The pinned v0.1 comparison adapter requires repository read access
	while the Hugging Face repository is private. See `DATASET_V2.md` for provenance,
	limitations and evaluation policy. These are AI-authored synthetic labels, not
	independent human annotations.

	After a completed run, `predict_v2.py` loads the saved adapter and translates one
	quote plus a split/column schema into an unapproved JSON proposal. The supported
	checks cover exact row counts, within-split ID uniqueness and cross-split ID
	overlap. `real_data_demo.py` exercises the verifier on attributed UCI Wine Quality
	fixtures with artificial IDs and deliberate corruption. It does not measure the
	model's performance on real documentation.

	## Original v0.1 starter

	Local prototype for translating individual dataset claims into reviewed checks.
	MIT licensed. No publishing, paid API, account token, Drive mount, or hosted service required.

	## Start in Google Colab

	Upload `datacard_ci_colab.ipynb` using File → Upload notebook. Run all cells
	on CPU first. The notebook contains its own project source, so no GitHub repo
	or separate file upload is required. It runs the tests, creates an evidence report,
	installs pinned Hugging Face packages, and runs a tiny adapter smoke test.
	PyTorch comes from Colab's runtime and is deliberately not replaced.
	The setup removes Colab's optional `torchao` package from the temporary runtime:
	its preinstalled version conflicts with PEFT, and plain LoRA does not require it.
	Training runs in a fresh subprocess to avoid stale imports after package updates.

	The smoke test makes a tiny random Qwen3 model, performs two LoRA optimizer steps,
	checks that weights changed, then saves and reloads the adapter and compares logits.
	This tests training infrastructure, not useful language understanding.

	For a short pretrained-model fine-tuning experiment, select a T4 GPU,
	rerun the setup cells after the runtime changes, then set `RUN_TRAINING = True`
	in the last training cell. It downloads `Qwen/Qwen3-0.6B`, records the exact Hub
	revision, compares baseline and tuned outputs, and trains for 80 optimizer steps (two passes over training data).
	This uses LoRA with float32 weights and float16 mixed precision with batch size 1, accumulation 4, and a 512-token
	limit; no quantization or custom CUDA extension. GPU availability is not guaranteed.
	Never buy compute for the initial experiment. Keep `RUN_TRAINING = False` for CPU.

	Outputs stay in the runtime under `artifacts/`. Download the archive using the final
	cell before ending the session. No automatic Hub upload or telemetry integration.

	## Local use

	Python 3.10+; the verifier has no third-party dependencies.

	```sh
	python3 -m unittest -v
	python3 -m pip install -e .
	```

	Training additionally requires PyTorch and `requirements-colab.txt`.

	```python
	from datacard_ci import demo_inputs, verify
	report = verify(*demo_inputs())
	print([r['status'] for r in report['results']])
	# FAIL, FAIL, FAIL, ABSTAIN, ABSTAIN — deliberately broken demo data
	```

	## What this version does

	- Row count, unique column values, and disjoint values across two CSV splits.
	- Exact source character spans, original card text, and SHA-256 of scanned bytes.
	- Explicit unapproved proposals, manual review, and aggregate evidence without raw IDs.
	- Original synthetic training data: 160 training and 40 held-out examples.
	- All 40 held-out examples used for quick before/after generation evaluation.
	- Two operator families plus count checks; no generated code execution.

	## Limits

	This is a starter, not the complete product in the original proposal. It does not
	fetch Hugging Face datasets, extract claims automatically, prove legal/social facts,
	implement CI/badges, or authenticate a maintainer's approval. A human selects quotes
	and reviews classifications and parameters. A malicious or mistaken classification
	can still misrepresent prose; the verifier checks the approved specification only.

	`full_supplied_files` means all supplied CSV bytes, not proof of remote dataset
	completeness, origin, or revision. Remote revision is null in verifier reports.
	Inputs are limited to 10 splits, 2 MB and 20,000 rows per split. Empty splits are
	inconclusive (ERROR), including zero-row count claims. Values are literal strings:
	`01` differs from `1`; no case or whitespace normalization. Blank IDs are ERROR.
	Malformed CSV is rejected before a report is created. Input prose is preserved in
	reports, so use synthetic/public data in shareable runs. The synthetic benchmark is
	small and repetitive; its scores cannot establish real-world reliability.

	## Sources

	- [Colab resource limits](https://research.google.com/colaboratory/faq.html)
	- [Qwen3-0.6B model card](https://huggingface.co/Qwen/Qwen3-0.6B)
	- [PEFT LoRA](https://huggingface.co/docs/peft/developer_guides/lora)

	Run `python3 build_notebook.py` after changing source files to rebuild the notebook.