DataCard CI — dataset claim compiler

v0.2 experiment

Use datacard_ci_v2_colab.ipynb for the new 1,200-example training workflow, development-based checkpoint selection and a fresh 100-example test comparing the base model, v0.1 and v0.2. Training is disabled by default and runs one complete pass on a GPU by default (at most two passes are supported). The pinned v0.1 comparison adapter requires repository read access while the Hugging Face repository is private. See DATASET_V2.md for provenance, limitations and evaluation policy. These are AI-authored synthetic labels, not independent human annotations.

After a completed run, predict_v2.py loads the saved adapter and translates one quote plus a split/column schema into an unapproved JSON proposal. The supported checks cover exact row counts, within-split ID uniqueness and cross-split ID overlap. real_data_demo.py exercises the verifier on attributed UCI Wine Quality fixtures with artificial IDs and deliberate corruption. It does not measure the model's performance on real documentation.

Original v0.1 starter

Local prototype for translating individual dataset claims into reviewed checks. MIT licensed. No publishing, paid API, account token, Drive mount, or hosted service required.

Start in Google Colab

Upload datacard_ci_colab.ipynb using File → Upload notebook. Run all cells on CPU first. The notebook contains its own project source, so no GitHub repo or separate file upload is required. It runs the tests, creates an evidence report, installs pinned Hugging Face packages, and runs a tiny adapter smoke test. PyTorch comes from Colab's runtime and is deliberately not replaced. The setup removes Colab's optional torchao package from the temporary runtime: its preinstalled version conflicts with PEFT, and plain LoRA does not require it. Training runs in a fresh subprocess to avoid stale imports after package updates.

The smoke test makes a tiny random Qwen3 model, performs two LoRA optimizer steps, checks that weights changed, then saves and reloads the adapter and compares logits. This tests training infrastructure, not useful language understanding.

For a short pretrained-model fine-tuning experiment, select a T4 GPU, rerun the setup cells after the runtime changes, then set RUN_TRAINING = True in the last training cell. It downloads Qwen/Qwen3-0.6B, records the exact Hub revision, compares baseline and tuned outputs, and trains for 80 optimizer steps (two passes over training data). This uses LoRA with float32 weights and float16 mixed precision with batch size 1, accumulation 4, and a 512-token limit; no quantization or custom CUDA extension. GPU availability is not guaranteed. Never buy compute for the initial experiment. Keep RUN_TRAINING = False for CPU.

Outputs stay in the runtime under artifacts/. Download the archive using the final cell before ending the session. No automatic Hub upload or telemetry integration.

Local use

Python 3.10+; the verifier has no third-party dependencies.

python3 -m unittest -v
python3 -m pip install -e .

Training additionally requires PyTorch and requirements-colab.txt.

from datacard_ci import demo_inputs, verify
report = verify(*demo_inputs())
print([r['status'] for r in report['results']])
# FAIL, FAIL, FAIL, ABSTAIN, ABSTAIN — deliberately broken demo data

What this version does

Row count, unique column values, and disjoint values across two CSV splits.
Exact source character spans, original card text, and SHA-256 of scanned bytes.
Explicit unapproved proposals, manual review, and aggregate evidence without raw IDs.
Original synthetic training data: 160 training and 40 held-out examples.
All 40 held-out examples used for quick before/after generation evaluation.
Two operator families plus count checks; no generated code execution.

Limits

This is a starter, not the complete product in the original proposal. It does not fetch Hugging Face datasets, extract claims automatically, prove legal/social facts, implement CI/badges, or authenticate a maintainer's approval. A human selects quotes and reviews classifications and parameters. A malicious or mistaken classification can still misrepresent prose; the verifier checks the approved specification only.

full_supplied_files means all supplied CSV bytes, not proof of remote dataset completeness, origin, or revision. Remote revision is null in verifier reports. Inputs are limited to 10 splits, 2 MB and 20,000 rows per split. Empty splits are inconclusive (ERROR), including zero-row count claims. Values are literal strings: 01 differs from 1; no case or whitespace normalization. Blank IDs are ERROR. Malformed CSV is rejected before a report is created. Input prose is preserved in reports, so use synthetic/public data in shareable runs. The synthetic benchmark is small and repetitive; its scores cannot establish real-world reliability.

Sources

Run python3 build_notebook.py after changing source files to rebuild the notebook.