File size: 4,414 Bytes
a6f7197 4e75514 a6f7197 4f8fc41 4e75514 a6f7197 4e75514 4f8fc41 4e75514 4f8fc41 4e75514 4f8fc41 4e75514 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 | ---
base_model: Qwen/Qwen3-Embedding-0.6B
base_model_relation: adapter
library_name: peft
pipeline_tag: feature-extraction
license: apache-2.0
pretty_name: Jev topic head — Qwen3-Embedding-0.6B (LoRA + prototype head)
tags:
- embeddings
- text-classification
- distillation
- knowledge-distillation
- generated-code
- status:scaffold
---
# Jev topic classifier — local replacement
> 📄 **Review & open questions live here:** https://huggingface.co/spaces/Kodep/jev-review — our read of the spike report, the numbers that matter, and the questions to confirm in the code.
> 🔒 **Data is kept private & separate:** https://huggingface.co/datasets/Kodep/jev-topic-data — public code/weights, but the raw item text (school-email-derived) is gated. The `data/` folder in this repo is only a pointer.
Replaces the hosted **Jev** step of the school-email digest pipeline: classify each item into **one of 56 topics** so the fixed rules table can decide show/hide.
**Chosen approach** (per the baseline report): `Qwen/Qwen3-Embedding-0.6B` + **LoRA (rank 16)** + a **prototype head** (56 learned vectors; score = cosine similarity × learned scale), running on **CPU** — ~0.2 s/item, ~2.2 GiB RAM, 0 GPU.
> **STATUS: SCAFFOLD.** This repo is ready to receive the code, data, and weights from the owner. Nothing owner-side has been pushed yet. The card metadata above is provisional until the real LoRA adapter + head weights land. The numbers in `docs/report_claude_v1.md` are **Claude-written and unverified** — treat them as claims to confirm against the code and a re-run of `evaluate.py`, not as established facts.
## Intended layout
```
jev-topic-head/
├── README.md
├── docs/report_claude_v1.md # baseline claims (Claude-written, unverified)
├── config.yaml # all hyperparameters ← to push
├── data/ # → pointer to private repo Kodep/jev-topic-data (not stored here)
├── src/ # train_head.py, evaluate.py, rules_table.json ← to push
├── checkpoints/ # LoRA adapter + head prototypes (seed 0) ← to push
└── eval/ # per-item test dump + confusion matrix ← to push
```
Data, the topic taxonomy, and the human ("gold") labels live in the **private** dataset repo [`Kodep/jev-topic-data`](https://huggingface.co/datasets/Kodep/jev-topic-data). To pull them for a run (collaborators only): `hf download Kodep/jev-topic-data --repo-type dataset --local-dir data`.
## What unblocks verification (priority order)
1. **`src/train_head.py` + `config.yaml`** — confirm the actual training matches the report's claims (checklist below).
2. **`eval/per_item_test.jsonl`** — recompute top-1 accuracy, macro-F1, and show/hide without re-running the model.
3. **`checkpoints/` + data (private repo)** — reproduce end to end and re-confirm the CPU latency/RAM.
## Claims to verify against the code
- [ ] LoRA on attention + feed-forward layers, **rank 16 / alpha 32 / dropout 0.05**; rest frozen
- [ ] Head = **56 prototype vectors**, score = cosine(item, proto) × learned scale (scale init **20**)
- [ ] Prototypes initialized from the **embedding of each topic description** (14 topics have no training item)
- [ ] Loss = softmax cross-entropy, **label smoothing 0.05**, single-label (not multi-label)
- [ ] AdamW: LoRA lr **3e-4** (wd 0.01), head lr **2e-3** (no wd); warmup 10% → linear decay to 5%
- [ ] Batch **16**, ≤ **12 epochs**, early stop on **validation macro-F1**, **bf16**
- [ ] Class-balanced sampling (weight **1/√count**), hard topic pairs ×2, **no augmentation / no synthetic items**
- [ ] **3 seeds**; macro-F1 spanned **0.49–0.59** across seeds; shipped run = **seed 0**
- [ ] "87.6%" = **top-1 topic accuracy vs Jev's fresh answer** on test (92/105); **macro-F1 0.572**; show/hide 95/105 (90.5%)
## Headline caveat
All scores are measured **against Jev**, not ground truth. Jev matches itself only 97.1% of the time and matched a human label 11/20 on one 20-item set. Only **31 items** have independent human labels, and the test split was used for decisions (treated as dev data). A fresh, unopened 100-item set is the intended final benchmark.
## Reproduction
_TBD once `src/` lands — will document exact commands (env, base-model path, seed, config) to reproduce train + eval._
|