README.md

metadata

title: Data-use annotation review
emoji: 🏷️
colorFrom: blue
colorTo: green
sdk: static
pinned: false

human_labeling

Fork of analysis/v24_sample_review/audit_deck.html as a hostable annotation pipeline. Same blind KEEP/DROP interaction, but the queue comes from the Hub instead of an 11MB embedded ROWS array.

• Source: rafmacalaba/datause-ner (viewer) for passages; scores are singlepass only — the infer head in rafmacalaba/gliner-datause-catchall-singlepass (sigmoid(head([start; end; mean; ±64 window])) on inference-native features). No v3/encode-head scores anywhere in this tool.
• Default queue: probe_candidates/pool — the only config with an embedded passage. probe_splits train/val/holdout rows have no passage (surface + word offsets only), so they annotate surface-only unless joined to a chunk corpus (see caveat below).
• Annotations land in human_labeling/annotations/rulings.jsonl (one JSON object per ruling). On static hosting (github.io) there is no writer: the app exports a JSON file the annotator drops into that folder (PR). On Spaces the bundled app.py accepts POST /api/rulings and appends server-side.

Quick start (local)

uv run python human_labeling/build_gliner_queue.py            # Luna spans + singlepass scores
uv run python human_labeling/app.py  # serves index.html + /api/queue + POST /api/rulings

Data source: the gliner config of rafmacalaba/datause-ner (tokenized passages + catch-all entity spans with camp2 Luna traceability). The queue maps each Luna verdict onto its span, rescores with the singlepass head (probe score) and the GLiNER proposer (extractor score @0.1), and the UI reviews keep/drop with both signals. Retired experiments (pool, passage grouping, gliner2/human473 queues) live only in git history.

Hosting

• github.io (static): publish human_labeling/ as-is. index.html fetches ./queue_gliner.json (built by build_gliner_queue.py, committed). Rulings persist in localStorage; Export downloads rulings-<annotator>.json, which is committed under annotations/ and reconciled with merge.py.
• Hugging Face Spaces (SDK: Docker, port 7860): same folder plus app.py. The frontend feature-detects POST /api/rulings: if it 200s, rulings stream to annotations/rulings.jsonl on disk; if not, it falls back to export. Dockerfile provided.

Files

file	purpose
index.html	annotation UI (passage view, per-mention ruling, luna×model filter)
build_gliner_queue.py	builds + scores queue_gliner.json from the gliner config (singlepass probe + extractor on MPS)
queue_gliner.json	served queue: 225 passages / 522 Luna-labeled spans, round-robin over origins, multi-mention first
probe_labels.py	shared decision rule + per-origin singlepass thresholds
app.py	stdlib server: static + GET /api/queue + POST /api/rulings
merge.py	annotations/rulings*.jsonl → annotations/adjudicated.jsonl (majority per key)
annotations/	ruling storage; every file is JSONL, one ruling per line
Dockerfile	Spaces deploy (Docker SDK)
build_gliner2_queue.py, build_pool_queue.py, rescore_singlepass.py, build_passage_queue.py, build_gold_queue.py	retired experiments (kept for reference; not served)

Caveat: probe_splits has no context

probe_train/val/holdout.jsonl rows are key, origin, corpus_id, w_start, w_end, surface, label, source, head_score, split — no passage text. The underlying chunk text lives in per-corpus repos (rafmacalaba/fcv-extractions-meta[-tiered]), and general_prwp (the majority origin) has no published chunk source at all (same gap as demo/server.py: HOLDOUT_META_SOURCES). So:

• build_pool_queue.py --config probe_candidates (default): full passage, blind ruling quality = audit_deck.
• build_pool_queue.py --config probe_splits: surface-only triage; items are flagged "ctx_missing": true and the UI shows the surface + scores without a context card. Fine for obvious keeps/drops, not for adjudication.

Bands (decision labels)

Every queue item carries band, tagged from the singlepass score with the bundle's per-origin best-F1 thresholds (probe_labels.py):

band	rule
keep	head_score >= per-origin threshold — model-flagged keep, quick confirm
confusion	DROP_FLOOR < head_score < threshold — the review zone (was "mid")
drop	head_score <= 0.05 — confident reject, quick confirm
unscored	score missing (out-of-grid span)

Per-origin thresholds (published thresholds.json): fcv_pads_east_africa 0.5 · general_prwp 0.4 · jad_paddy_docs 0.1 · jdc_operational 0.6 · refugee_pads 0.5 · reliefweb 0.4; fallback global_best 0.5.

Ruling schema

{"key": "sample:fcv_pads_east_africa:002082:7:1:0", "ruling": "DATA_MENTION",
 "annotator": "rm", "ts": "2026-09-08T12:00:00Z", "note": "",
 "head_score": 0.61, "origin": "fcv_pads_east_africa", "queue": "probe_candidates/pool"}
`ruling` is `DATA_MENTION` (keep) or `NON_MENTION` (drop) — same labels as the deck export.
## review.html (current)

Multi-span review over `rafmacalaba/data-use-ner`: `gliner` holdout (gold 473),
`to_annotate` pool, per-user `annotate_{paddy,aj,aivin,rafael}[_part2]` sets
(240 passages each, mutually exclusive, stratified by origin × probe tag).
Assignment manifests live in `assignment_manifest.json` (verified equal to
the Hub configs — queue files are append-only; never resampled in place).

### For annotators

1. Open your link (or pick your name on the landing page) — your file loads
   automatically. Type your name if asked; it locks on first ruling.
2. Judge each highlighted mention in its window: `→` next mention,
   `k` keep, `d` drop, `space` skip, `[`/`]` slide the window,
   `-`/`+` resize it. Optional freeform note per ruling.
3. Before ruling you see only `probe: easy` or `probe: confusion`. After
   ruling, Luna's verdict + reasoning appear, with a DISAGREE flag (also a
   toast) when you differ.
4. Every 10% you'll be nudged to **export** a checkpoint (`e` or the button,
   header and footer). Send `rulings-<name>-<queue>.jsonl` to rafael/aivin.
   To resume later, **import** the file back — you jump to the first
   unreviewed mention.

### For coordinators

Drop returned files into `human_labeling/annotations/` and run `merge.py` →
majority verdict per span `key` (ties → REVIEW) in `annotations/adjudicated.jsonl`.
Public app: https://rafmacalaba-data-use-annotate.static.hf.space/