scaffold: structure + READMEs + baseline report (code/data/checkpoints pending)

README.md

@@ -0,0 +1,41 @@
+# Jev topic classifier — local replacement
+
+Replaces the hosted **Jev** step of the school-email digest pipeline: classify each item into **one of 56 topics** so the fixed rules table can decide show/hide.
+
+**Chosen approach** (per the baseline report): `Qwen/Qwen3-Embedding-0.6B` + **LoRA (rank 16)** + a **prototype head** (56 learned vectors; score = cosine similarity × learned scale), running on **CPU** — ~0.2 s/item, ~2.2 GiB RAM, 0 GPU.
+
+> **STATUS: SCAFFOLD.** This repo is ready to receive the code, data, and weights from the owner. Nothing owner-side has been pushed yet. The numbers in `docs/report_claude_v1.md` are **Claude-written and unverified** — treat them as claims to confirm against the code and a re-run of `evaluate.py`, not as established facts.
+
+## Intended layout
+```
+jev-topic-head/
+├── README.md
+├── docs/report_claude_v1.md     # baseline claims (Claude-written, unverified)
+├── config.yaml                  # all hyperparameters          ← to push
+├── data/                        # train/val/test + taxonomy + human labels   ← to push
+├── src/                         # train_head.py, evaluate.py, rules_table.json ← to push
+├── checkpoints/                 # LoRA adapter + head prototypes (seed 0)   ← to push
+└── eval/                        # per-item test dump + confusion matrix    ← to push
+```
+
+## What unblocks verification (priority order)
+1. **`src/train_head.py` + `config.yaml`** — confirm the actual training matches the report's claims (checklist below).
+2. **`eval/per_item_test.jsonl`** — recompute top-1 accuracy, macro-F1, and show/hide without re-running the model.
+3. **`checkpoints/` + `data/`** — reproduce end to end and re-confirm the CPU latency/RAM.
+
+## Claims to verify against the code
+- [ ] LoRA on attention + feed-forward layers, **rank 16 / alpha 32 / dropout 0.05**; rest frozen
+- [ ] Head = **56 prototype vectors**, score = cosine(item, proto) × learned scale (scale init **20**)
+- [ ] Prototypes initialized from the **embedding of each topic description** (14 topics have no training item)
+- [ ] Loss = softmax cross-entropy, **label smoothing 0.05**, single-label (not multi-label)
+- [ ] AdamW: LoRA lr **3e-4** (wd 0.01), head lr **2e-3** (no wd); warmup 10% → linear decay to 5%
+- [ ] Batch **16**, ≤ **12 epochs**, early stop on **validation macro-F1**, **bf16**
+- [ ] Class-balanced sampling (weight **1/√count**), hard topic pairs ×2, **no augmentation / no synthetic items**
+- [ ] **3 seeds**; macro-F1 spanned **0.49–0.59** across seeds; shipped run = **seed 0**
+- [ ] "87.6%" = **top-1 topic accuracy vs Jev's fresh answer** on test (92/105); **macro-F1 0.572**; show/hide 95/105 (90.5%)
+
+## Headline caveat
+All scores are measured **against Jev**, not ground truth. Jev matches itself only 97.1% of the time and matched a human label 11/20 on one 20-item set. Only **31 items** have independent human labels, and the test split was used for decisions (treated as dev data). A fresh, unopened 100-item set is the intended final benchmark.
+
+## Reproduction
+_TBD once `src/` lands — will document exact commands (env, base-model path, seed, config) to reproduce train + eval._

checkpoints/README.md

@@ -0,0 +1,15 @@
+# checkpoints/
+
+Trained weights for the chosen head.
+
+## Files to push
+| path | what it is |
+|---|---|
+| `lora/` | LoRA adapter (`.safetensors`). Report: rank 16, alpha 32. |
+| `head/` | 56 prototype vectors + per-topic learned scale (`.safetensors` or `.npy`) |
+
+## Notes to include
+- Exact file names and **tensor shapes** (e.g. prototypes `56 × <dim>`, scale `56`).
+- Which **seed** (report says the shipped run = **seed 0**).
+- Embedding dim of the base model (expected 1024 for the 0.6B — confirm).
+- How to load them in `train_head.py` / `evaluate.py`.

data/README.md

@@ -0,0 +1,23 @@
+# data/
+
+The labeled set used for train / validation / test. JSONL, one item per line.
+
+## Files to push
+| file | expected |
+|---|---|
+| `train.jsonl` | 279 first-occurrence items |
+| `validation.jsonl` | 85 items |
+| `test.jsonl` | 105 items |
+| `topic_taxonomy.json` | the 56 topics + their descriptions (used to seed prototypes + zero-shot) |
+| `human_labels.jsonl` | the 31 independently human-labeled items (gold, not Jev) |
+
+## Suggested JSONL schema (confirm against the actual code)
+```json
+{"id": "...", "email_id": "...", "text": "<item text>", "jev_topic": "<topic>", "human_topic": "<topic or null>"}
+```
+
+## Rules the data must respect
+- Items from the **same email stay in the same split** (leakage guard). Repeated-event items also stay together.
+- **Sports senders ≈ 50%** of items — keep the split mix as-is.
+- Do **not** strip, truncate, or paraphrase `text`; eval must reproduce on the exact inputs (some run 843–1,923 tokens).
+- For this version, repeated items were left out of training — keep it that way.

docs/report_claude_v1.md

@@ -0,0 +1,193 @@
+# Baseline report (Claude-written, unverified)
+
+> Captured verbatim from the owner's Claude Artifact, 3 October 2026. This is the **claims baseline**. Every number below should be confirmed against `src/` + a re-run of `evaluate.py`. Do not cite these as verified results until then.
+
+---
+
+# Not final — Test report, 3 October 2026
+_Written in Simplified Technical English (ASD-STE100)._
+
+## Choosing a local replacement for the Jev classifier
+What we tried, what we saw, and why we chose a small trained model that runs on the CPU.
+
+## Summary
+Jev is a hosted AI model. It sorts each school email item into one of 56 topics.
+We want to do this job on our own PC. We do not want a paid fallback.
+We tried more than 15 models and methods. Many of them gave the same answer as Jev for fewer than 7 items in 10.
+We chose a small model called Qwen3-Embedding-0.6B with a trained head. It runs on the CPU and uses no GPU memory. It takes 0.2 seconds for each item.
+It gives the same show or hide result as Jev for 93 items in 100 (176 of 190 items).
+One bigger system scored higher (185 of 190). It needs 9.3 GiB of GPU memory. We cannot spare that memory.
+The choice is not final. We checked the model against Jev's saved answers. We did not check it against enough human answers yet.
+
+## 1. The job
+A family reads a daily digest of school email. The system must decide which items to show. Jev does one step of that work.
+
+1. Email arrives — School and team messages.
+2. Reader — A large AI model splits the email into items. An item has a title, a date and a short summary.
+3. Classifier (this report) — Today this is Jev. It picks a topic, a child, and whether the parent must act.
+4. Rules table — Each topic has a rule: show, hide, or "depends". The rules are fixed. They do not use AI.
+5. Daily digest — The family sees the items that pass the rules.
+
+Only step 3 changes. The rules in step 4 stay the same. So the topic name matters less than the show or hide result. We measured both.
+
+### Words we use
+| Word | Meaning |
+|---|---|
+| Item | One entry from an email. One email can give many items. |
+| Topic | The kind of item. There are 56 topics, for example "youth sports" or "school lunch". |
+| Same as Jev | The model gave the same answer that Jev gave. For topics this is top-1 accuracy against Jev. It does not prove the answer is correct. Jev can be wrong. |
+| Show or hide result | The final result after the rules table. The family sees the item or does not see it. |
+| Important item dropped | Jev would show the item. The model would hide it. This is the worst error. |
+| Train, validation, test | Three separate groups of items. We teach the model on the train group. We tune it on the validation group. We score it one time on the test group. |
+| CPU, GPU | The CPU is the main chip. The GPU is the graphics chip. Our GPU has 16 GiB of memory. A large text model and the games share it. |
+| Macro-F1 | A score from 0 to 1 that gives each topic the same weight. Rare topics count as much as common topics. |
+| Confidence | A number from 0 to 1. It shows how sure the model is. We send items with low confidence to a person. |
+
+## 2. How we tested
+### The data
+We used 1,168 items that the live system stored. Jev's saved answers came with them. We kept the first item of each repeated event: 469 items. We cut them into three groups.
+
+- Train 279
+- Validation 85
+- Test 105
+
+Items from the same email stay in the same group. Items of the same repeated event also stay together. This stops the model from seeing the answer in advance.
+Half of the items come from sports leagues. We checked results with and without them.
+Jev's saved answers are the "teacher". They are not the truth. A person labelled a small set of items. Only 31 items have independent human labels.
+
+### The stages
+- Stage 1. Screen (80 items): We wrote the pass rules before the test. No model met them. All models failed the rules.
+- Stage 2. Train and combine: We trained small models on Jev's saved answers. We also joined two models in a cascade.
+- Stage 3. Count show or hide results: The family cares about the result after the rules table. We scored that result, not only the topic name.
+- Stage 4. Other model types: We tested models that work like Jev: Clef, Intern and small Jev-style models.
+
+### The methods
+| Method | How it works |
+|---|---|
+| Zero-shot | We use the model as it is. We give it the topic descriptions. We do not train it. |
+| Embedding plus simple head | A model turns the item text into numbers. A small layer picks the topic from the numbers. |
+| Trained head (our choice) | Same as above. We also change a small part of the model, with a method called LoRA. We start the head from the topic descriptions. We use 3 random seeds. |
+| Fine-tuned GLiNER | We train the old Jev-like model on our items. It picks from a short list of 10 topics. |
+| Cascade | Step 1: an embedding model makes a short list of 10 topics. Step 2: a larger model (Qwen3.5-4B) picks one topic from the list. |
+
+### Rules we kept
+- We tuned settings on the validation group only.
+- We scored the test group one time for each finalist.
+- We stopped training early when the validation score stopped rising.
+- We used class-balanced training. Hard topic pairs (e.g. league schedules vs league fundraising) got extra examples.
+- We ran each trained model with 3 seeds. The scores changed by a large amount between seeds. So small differences are noise.
+
+### How we trained the chosen head
+We did not use a plain linear probe on frozen vectors. We changed a small part of the model itself. This method is called LoRA. We also trained a topic head on top.
+
+| Part | What we did |
+|---|---|
+| Base model | Qwen3-Embedding-0.6B. It turns text into a list of numbers. |
+| Model change | LoRA adapters on the attention and feed-forward layers. Rank 16, alpha 32, dropout 0.05. The rest of the model stays frozen. |
+| Topic head | One learned vector (a prototype) for each of the 56 topics. The score for a topic is the cosine similarity between the item vector and the prototype, times a learned scale. The scale starts at 20. |
+| Start of the head | Each prototype starts from the embedding of the topic description. So a topic with no training item still has a sensible place. 14 of the 56 topics have no training item. |
+| Loss | Softmax cross-entropy with label smoothing 0.05. Each item has one topic. It is not a multi-label task. |
+| Optimizer | AdamW. Learning rate 3e-4 for the LoRA weights (weight decay 0.01). Learning rate 2e-3 for the head (no weight decay). Warm-up for 10% of the steps, then a straight decay to 5% of the rate. |
+| Batch and epochs | Batch of 16. At most 12 epochs. We stop early when the validation macro-F1 stops rising. Training uses bf16 precision. |
+| Topic balance | We pick training items with weight 1 divided by the square root of the topic count. Rare topics get picked more often. Hard topic pairs get weight times 2: league schedules, league fundraising, 5K runs, and similar pairs. We used no augmentation and no synthetic items. |
+| Training data | 279 first-occurrence items. We left out repeated items for this version. The labels are Jev's fresh answers (the teacher). Only a few topic labels come from a person. |
+| Seeds | 3 random seeds. The macro-F1 changed from 0.49 to 0.59 between seeds. The CPU version uses the run with seed 0. |
+| Child and action | The head also gives a child answer and an action answer. These are weak: the child answer matches Jev for 51 to 65 items in 100. |
+
+### How we counted 87.6%
+87.6% is top-1 accuracy of the topic against Jev's fresh answer. The head picks one topic. The score counts the items where it picks the same topic as Jev.
+It is on the test group: 92 of 105 items.
+The macro-F1 on the same group is 0.572. Macro-F1 gives each topic the same weight. It is lower because rare topics are hard.
+The show or hide result matches Jev for 95 of 105 items (90.5%) on the test group.
+
+## 3. What we saw
+### All models, in the order we tried them
+| Model or method | Same topic as Jev | Speed and memory | Result |
+|---|---|---|---|
+| GLiNER2.5-Decide 340M, as shipped | 24.1% | About 0.5 s on GPU | Dropped — picked one topic too often; one sports stream hid the other errors. |
+| GLiNER2.5-Decide 1B | 23.8% | - | Dropped |
+| GLiClass-instruct-large | 1.2% | - | Dropped — scores are all the same; cannot use topic descriptions. |
+| BGE-small embeddings, zero-shot | 29.4% | CPU | Dropped |
+| Embedding plus reranker | 56 to 59% | - | Dropped — but the short list of 10 topics holds Jev's topic for 93.4% of items. Used in the cascade. |
+| Word-count model (TF-IDF) | 65 to 67% | CPU | Dropped |
+| Embedding plus simple head | 65 to 67% | CPU | Dropped |
+| BGE-small, trained | Lower than Qwen3 | CPU | Dropped |
+| Qwen3.5-4B picks from all 56 topics | 57.5% to 75.3% | About 7 GiB GPU | Dropped alone — picks the child right in only 45.9%; works well behind a short list. |
+| GLiNER-340M, fine-tuned, 10-topic list | 81.9% | 3.1 GiB GPU, or 1.3 s on CPU | Not chosen — no better than our choice; uses GPU memory. |
+| Cascade: embedding list, then Qwen3.5-4B | 87.6% | 0.41 s, 9.3 GiB GPU | Kept as helper — best quality; too much GPU memory. |
+| Qwen3-Embedding-0.6B with trained head | 87.6% | 0.20 s on CPU, 2.2 GiB RAM, 0 GPU | **Chosen** |
+| Clef-flash 9B (Cloudflare) | 84.8% | 1.47 s, 8.3 GiB GPU | Dropped — slow; drops more important items. |
+| Intern-Decision-4B | 79.0% | 0.8 s, 3.7 to 9 GiB GPU; 21 s on CPU | Parked — not better than our choice. |
+| Jev-Style 0.8B | 88% * | 0.55 s on GPU, 7.6 s on CPU | Dropped |
+| Kev-0.8B | 86% * | 0.53 s on GPU, 10 s on CPU | Dropped |
+| Kev-4B | Not scored | 46 s on CPU | Dropped — too slow. |
+| Open-Jev-DeBERTa | - | - | Cannot run — 512-token limit; our questions need 843 to 1,923 tokens. |
+
+Scores are for the test group unless stated. Early rows use the whole data set or the validation group, so compare them only as a rough guide. * These two models were scored on a different set, with a show-or-hide measure. Our choice scored 93% on that same measure.
+
+### The finalists, side by side
+All numbers use the same 190 items (validation and test together). Higher is better, except "lower is better" rows.
+
+| Metric | Trained head | Cascade | Fine-tuned GLiNER | Clef-flash | Intern (bf16) | Jev, saved answer |
+|---|---|---|---|---|---|---|
+| Same topic as Jev, val / test | 75.3 / 87.6% | 88.2 / 87.6% | 84.7 / 81.9% | 77.6 / 84.8% | 80.0 / 79.0% | 96.5 / 97.1% |
+| Same show or hide result, val / test | 83.5 / 90.5% | 92.9 / 94.3% | 91.8 / 90.5% | 85.9 / 92.4% | 84.7 / 89.5% | 97.7 / 98.1% |
+| Same final result (of 190) | 176 | 185 | 181 | 176 | 174 | 189 |
+| Important items dropped (lower better) | 7 | 3 | 5 | 11 | 8 | 0 |
+| Noise shown (lower better) | 7 | 2 | 4 | 3 | 8 | 1 |
+| Time for one item (lower better) | 0.20 s | 0.18 to 0.41 s | 0.09 s (GPU) | 1.47 s | 0.80 s | - |
+| GPU memory (lower better) | 0 | 9.3 GiB | 3.1 GiB | 8.3 GiB | 9.05 GiB | - |
+| Computer RAM | 2.2 GiB | - | - | 10.5 GiB | 8.9 GiB | - |
+
+"Jev, saved answer" compares Jev's stored answer with a fresh Jev answer. It shows the top of the scale: even Jev does not repeat itself 100%.
+
+The test group has 105 items. A difference of 2 or 3 points is 2 or 3 items. That is inside the noise.
+
+### GPU memory each model needs
+The GPU has 16 GiB. The desktop uses part of it. The main text model uses 6.7 to 11.4 GiB, depending on the model. A classifier that needs 9 GiB does not fit beside it. Trained head 0 · Fine-tuned GLiNER 3.1 · Clef-flash 8.3 · Intern (bf16) 9.05 · Cascade 9.3. (Scale 0–16 GiB; red = too large to fit beside the main text model.)
+
+### What the errors look like
+The head is right when it is sure. Its middle confidence is 0.71 to 0.79 when it is right. It is 0.34 to 0.38 when it is wrong.
+Of the 24 errors, only 1 has a confidence of 0.9 or more. 19 have a confidence below 0.5.
+So we can send low-confidence items to a person. (Exploratory numbers on development data; not a frozen setting.)
+
+| Confidence limit | Items decided by the model (of 190) | Of those, same as Jev | Items sent to a person |
+|---|---|---|---|
+| 0.3 | 153 | 144 | 37 |
+| 0.5 | 123 | 119 | 67 |
+| 0.7 | 93 | 91 | 97 |
+
+### Two facts that make the scores look better than they are
+- The head scored 90.5% on the test group and 83.5% on the validation group. The test mix was easy. The average for both groups is 87.4%.
+- Sports senders are about half of the items. The head is nearly always right on them. On the other items it matches Jev for 78 items in 100.
+
+## 4. Why we chose the trained head
+1. Does it match Jev well? Yes: trained head, cascade. Close: Clef, fine-tuned GLiNER, Intern. No: all the others.
+2. Does it fit in GPU memory? The GPU is shared with the main text model and games. The cascade needs 9.3 GiB. It does not fit. The head needs 0 GiB.
+3. Is it fast? The head takes 0.2 s on the CPU. Clef takes 1.47 s. Intern on the CPU takes 21 s.
+
+Result: Qwen3-Embedding-0.6B with a trained head. It matches Jev nearly as well as the best system, needs no GPU memory, is fast, and keeps the cascade as a helper for doubtful items.
+
+| Reason | Detail |
+|---|---|
+| It uses no GPU memory | The main text model and the games use the GPU. The head runs on the CPU. |
+| It is fast | 0.20 s for each item. The 95th slowest item takes 0.27 s. |
+| It is nearly as good as the cascade | Same topic score on the test group (87.6%). On the final show-or-hide result it is 9 items behind out of 190 (176 vs 185). |
+| Its errors show themselves | Most errors have low confidence. A person can check those items. |
+| It is simple | One small model on the CPU. No second model server. |
+| The others have a clear problem | The cascade needs too much memory. Clef is slow and drops more important items. Intern is not better. The small Jev-style models score lower. One model cannot run. |
+
+## 5. What we do not know
+- All scores compare a model with Jev. Jev is not always right. In one check on 20 items, Jev matched the person for 11. The set was chosen because the models disagreed, so this is not a general score.
+- Only 31 items have independent human labels. We need more before we can claim the head is correct.
+- The test group already helped us make decisions. We treat it as development data. A fresh group of 100 items is still unopened.
+- The head drops 7 important items out of 190 in this test. We do not know the rate on new data.
+- The head only knows the 56 topics it learned. A new topic needs new training.
+
+### Next steps
+1. Label the fresh group of 100 items.
+2. Fix the head, the rules, and the confidence limit together. Then score once on the fresh group.
+3. Pack the head as a service. Today it is test code only.
+4. Run it beside Jev for a period, with no effect on the digest, before we switch.
+
+_Source: the project spike report, sections 3c to 3s, and the current-state page in the same repository. This report contains counts only. It contains no email text._

eval/README.md

@@ -0,0 +1,20 @@
+# eval/
+
+Raw outputs so we can recompute the headline scores **without re-running the model**.
+
+## Files to push
+| file | what it is |
+|---|---|
+| `per_item_test.jsonl` | one line per test item: model topic, Jev topic, confidence, show/hide for each |
+| `confusion.json` | 56×56 confusion matrix over the test split |
+
+## Suggested schema for `per_item_test.jsonl`
+```json
+{"id": "...", "model_topic": "...", "jev_topic": "...", "confidence": 0.71, "model_show_hide": "show", "jev_show_hide": "show", "human_topic": null}
+```
+
+## Numbers to reproduce from this
+- top-1 topic accuracy vs Jev (report: 92/105 = 87.6%)
+- macro-F1 (report: 0.572)
+- show/hide agreement (report: 95/105 = 90.5%)
+- confidence split: of 24 errors, 19 < 0.5 confidence, only 1 ≥ 0.9

src/README.md

@@ -0,0 +1,19 @@
+# src/
+
+The training + evaluation code. **This is the primary thing we want to read** — it's the only way to confirm what the Claude-written report actually did.
+
+## Files to push
+| file | what it is |
+|---|---|
+| `train_head.py` | LoRA + prototype-head training |
+| `evaluate.py` | metrics: top-1 accuracy, macro-F1, show/hide, confidence buckets |
+| `rules_table.json` | the fixed step-4 show/hide rules (no AI) |
+| `config.yaml` | all hyperparameters (see the checklist in the root README) |
+
+## Also include so it runs offline
+- `requirements.txt` (or `environment.yml`) — torch version, bf16 support, `transformers`/`peft` versions
+- a short **how-to-run**: base model path (`Qwen/Qwen3-Embedding-0.6B`), seed flag, config flag, output paths
+- any preprocessing that turns a raw item into the model input (concatenation order, tokenization, max length)
+
+## What we'll check in `train_head.py` (vs the root README checklist)
+LoRA rank/alpha/dropout + which modules · prototype head + cosine×scale + scale init 20 · prototype seeding from description embeddings · loss + label smoothing · the two AdamW groups + warmup/decay · early-stop on val macro-F1 · class-balanced sampling + hard-pair weighting · seed handling.