docs/specs/0010-example-corpora.md

Spec 0010 - Bundled example corpora (CAMEL)

Status: implemented 2026-09-06 PI request (2026-08-19, via Slack): make the lab's CAMEL corpus the example data on the site so visitors can try CCR without uploading anything. Corpus details supplied by Aliah Zewail 2026-09-06.

Problem

The site only accepted uploaded files, so a visitor with no data of their own could not run anything, and lab members re-uploaded the same file to demo it.

Contract

example_corpora.py catalogues bundled corpora: id, file, text column, language, row count, description, citation, source URL, and what preprocessing was applied. GET /api/example-corpora lists them (default first). POST /api/projects/{id}/corpora/from-example copies one into a project through the normal ingest path, so everything downstream is unchanged.

The corpus row records example_id, and a run on it writes a corpus_source block into run metadata carrying the citation, source URL and preprocessing note. Results produced on someone else's corpus stay attributable in the same file as the scores.

Step 1 offers the example above the upload control, with the citation and a link to the full dataset visible, and selects the declared text column on load.

Update 2026-09-07: the full corpus is offered too

CCR_MAX_ROWS is raised to 100,000 (the code default all along; the three docs that stated 20,000 and 50,000 are corrected), so the 57,174-row corpus is now ingestible. Both are offered: the 999-text sample stays the DEFAULT because it finishes in seconds, and the full corpus sits beside it for real analyses, labelled with its cost.

The full corpus lives in object storage under examples/, not in the repo: 38 MB would ride in the Docker image and the Space repo on every deploy. It is NOT fetched from Hugging Face at runtime, because that dataset returns 401 - fetching it would mean shipping lab credentials in a public app.

Selection downloads the master once to parse, then makes the user's copy with a server-side copy_object, so 38 MB does not travel back out through the app per visitor. Measured against the live bucket: 7.0s fetch, 0.7s parse of 57,174 x 64, 1.6s copy. storage.delete() refuses the examples/ prefix outright, because every user copy derives from one master and retention runs unattended.

The server-side copy is an optimisation, not a requirement: R2 CopyObject is not available on every bucket (the lab's account answers NoSuchKey for an object head_object resolves in the same session, while the dev account copies it fine), so the code falls back to uploading the local copy it already downloaded to parse. Each deployment needs its own upload of the master file - dev and prod are different Cloudflare accounts, so examples/camel_full.csv exists once per account.

What ships bundled, and why the sample is the default

CAMEL is 57,174 texts. The platform ships a 999-text sample, because the full corpus cannot be the bundled default:

• it exceeds CCR_MAX_ROWS (50,000 on both deployments), so ingest would refuse it outright;
• it is 38 MB against a sample_data/ directory of 156 KB, and that file would ride in the Docker image and the Space repo;
• a demo has to finish in seconds on a 2 vCPU host.

The sample is drawn proportionally by source platform with a fixed seed (scripts/build_camel_sample.py), so it keeps the corpus's platform mix rather than drowning the smaller sources. Its word distribution tracks the full corpus (median 25 vs 26 words, mean 69.0 vs 71.5). Only text and source_platform are kept; text is verbatim, whitespace trimmed. Usernames were already removed by the corpus authors. The full corpus, with the 25 construct annotations, is linked from the corpus card and the guide.

Loading an example is exempt from the anonymous upload caps. Those bound what a stranger can push into the server; this is a file we ship at a size we chose, and letting a visitor try the platform without their own data is the point of the feature. The per-day run cap still applies.

Presentation (revised 2026-09-07)

The first cut rendered every example as an expanded card, which took most of Step 1 and pushed the upload control below the fold - the wrong hierarchy, since most researchers bring their own data. It also presented CAMEL twice, as two "corpora", when it is one dataset in two sizes.

Now: one labelled row holding a dropdown, an info button and the action. Sizes are options in that dropdown, so CAMEL reads as one dataset. Provenance (description, text column, preprocessing, citation, dataset link) moved into a modal behind the info button, where it is available without occupying the step.

The entry that loads first is preselected, not default: the PI asked for CAMEL to be "available and ready-to-use", which is a starting point rather than a claim that this is THE corpus, and nothing is badged as a default.

Presentation (revised again, 2026-09-07)

The dropdown row still sat above the upload control and still read as the first thing to do. The PI's framing was the reverse: a small "use sample data" entry point, from which a visitor sees the two available sizes and can read about the data if they want to.

Now the upload row comes first. Under it, one muted line ("No data of your own? Use sample data") expands into a two-row list: name, size, language, a one-line fit note, About, Use. About opens the provenance dialog, which carries its own Use button so a reader can decide from there. The panel closes once a corpus is loaded, and the selects above show it. Picking a size already loaded in the project selects that copy instead of copying the corpus again; the API supports this by returning example_id on corpus rows. Nothing is preselected, since a list needs no default; preselected now only fixes the order.

Presentation, final shape (2026-09-07)

Step 1 shows one quiet line, "No data of your own? Use sample data", which opens a picker: a dropdown naming each corpus and its size, a circular info button, and Use. Which corpus to run is a choice, so it is a dropdown rather than a stack of cards - adding a dataset later costs one more option instead of another block of the step. The selected entry's one-line summary sits under the row; everything else (description, text column, preprocessing, citation, dataset mark, link) lives in the info dialog.

An entry the instance cannot run is disabled in the dropdown and labelled "unavailable here", with the reason under the row.

Non-goals

• Serving the full 57k corpus from the platform. It stays on Hugging Face, which is a better host for a 38 MB dataset than a Space repo.
• Streaming or paging a corpus larger than CCR_MAX_ROWS. If the lab wants full-corpus runs, that is the job-duration work already on the ROADMAP, not this.

Tests

• every catalogued example exists, parses, has its declared text column and row count, and carries a citation and source URL;
• exactly one default, listed first;
• an ANONYMOUS visitor can load the example and complete a run on it (the whole point of the feature), and the picker lands on the right text column;
• the run's metadata carries the CAMEL citation and source URL;
• an unknown example id is a 404.

Deviations (filled after implementation)

None.