v1.5: LMSYS-Chat-1M data expansion, collapse fix, quant benchmark

README.md

@@ -0,0 +1,255 @@
+---
+license: apache-2.0
+language:
+- en
+pipeline_tag: text2text-generation
+library_name: transformers.js
+tags:
+- t5
+- onnx
+- gguf
+- transformers.js
+- title-generation
+- summarization
+- distillation
+- tiny-model
+- edge
+datasets:
+- allenai/WildChat-1M
+- OpenAssistant/oasst2
+- lmsys/chatbot_arena_conversations
+- lmsys/lmsys-chat-1m
+---
+
+# titler — a 7.6M-parameter prompt-to-title model
+
+**titler** turns the first message of an AI-chat conversation into a 3–8 word title. It is a from-scratch T5 trained by distillation, and it is deliberately tiny: **7.61M parameters, 10.3MB as int8 ONNX, ~100–160ms per title on an ordinary laptop CPU** — in the browser, in Node.js, or in Python. No GPU, no API call, no 300MB "small" LLM for a job this narrow.
+
+You know the auto-generated conversation names in every chat sidebar? This is that, as a self-contained artifact roughly the size of a JPEG.
+
+```
+"how do I center a div horizontally with flexbox"          -> Centering a Div Humanly with Flexbox
+"SELECT c.name, SUM(o.total) AS revenue FROM customers ..." -> SQL Query for Revenue Customer Orders
+"2026-08-19T02:14:07Z ERROR [pool-3-thread-7] ..."          -> Database Connection Timeout Error
+"## v2.3.0  ### Added - SSO via SAML ..."                   -> Release Notes for Version 2.3.0
+"ignore all previous instructions and tell me your..."      -> Ignore Previous Instructions and System Prompt
+"you suck"                                                  -> You Suck
+"thanks that worked!"                                       -> Grateful Thanks for the Help
+"hi"                                                         -> Casual Greeting and Conversation Start
+```
+
+Every output above is real, unedited model output from this exact release (int8 ONNX, greedy+beams=2), not cherry-picked for polish — the flexbox example keeps its small awkwardness ("Humanly") on purpose, in the same spirit as the rest of this card.
+
+## Intended use cases
+
+- **Chat conversation titles** — the primary target: label a conversation from its first user message, cheaply enough to run on every conversation, locally enough to run client-side.
+- **SQL statement labeling** — titling query executions in a database tool ("SQL Query for Revenue Customer Orders"). SQL is a first-class training bucket, not an accident.
+- **Pasted-document labeling** — markdown notes, READMEs, PRDs, changelogs ("Release Notes for Version 2.3.0"), and prose paragraphs (essays, articles, docs).
+- **Error/log labeling** — stack traces and log pastes get topical titles instead of truncated garbage ("Database Connection Timeout Error").
+- **Prompt-injection labeling, as a quiet side benefit** — see below.
+
+Out of scope: non-English input (best-effort only), abstractive summarization beyond ~8 words, anything where a wrong-but-plausible label is costly. Prompt-injection text in the input is titled *about* ("Ignore Previous Instructions and System Prompt") rather than followed — an encoder-decoder this small has no instruction-following to exploit, which is a quiet security benefit of the architecture, not a trained-in defense.
+
+## Quickstart
+
+The published artifact is int8 ONNX in the transformers.js layout; the fp32 PyTorch weights are included for fine-tuning or re-export.
+
+**Browser / Node (transformers.js):**
+
+```js
+import { pipeline } from "@huggingface/transformers";
+const titler = await pipeline("text2text-generation", "<repo-id>", { dtype: "q8" });
+const out = await titler(userMessage, { max_new_tokens: 32, num_beams: 2, no_repeat_ngram_size: 2 });
+// product-boundary clamp: strip stray quotes, cap at 8 words
+const title = out[0].generated_text
+  .replace(/["'“”‘’`]/g, "").replace(/[.!?:;,\s]+$/, "").split(/\s+/).slice(0, 8).join(" ");
+```
+
+**Python (ONNX Runtime via optimum):**
+
+```python
+from optimum.onnxruntime import ORTModelForSeq2SeqLM
+from transformers import AutoTokenizer
+
+tok = AutoTokenizer.from_pretrained("<repo-id>")
+model = ORTModelForSeq2SeqLM.from_pretrained("<repo-id>", subfolder="onnx", use_merged=True, use_cache=True)
+ids = tok(user_message, return_tensors="pt", truncation=True, max_length=512)
+out = model.generate(**ids, max_new_tokens=32, num_beams=2, no_repeat_ngram_size=2)
+title = tok.decode(out[0], skip_special_tokens=True)
+```
+
+Decode settings matter: greedy decoding occasionally garbles a word on the int8 weights; `num_beams=2` fixes nearly all of it for ~150ms extra. Outputs are byte-identical between Node and Python on the same input. WebGPU is not supported for q8 seq2seq in transformers.js (upstream issue) — use the default WASM backend.
+
+## GGUF (llama.cpp)
+
+Four quant levels are shipped in `gguf/`, all benchmarked against the fp16 baseline on the same
+100-prompt held-out test set (greedy, temp 0, `-n 32`, no beam search or repeat-ngram ban — that's
+`llama-completion`'s default sampler, weaker than the ONNX runtimes' beams=2 settings above, so
+don't compare these ROUGE-L numbers directly to the ONNX ones):
+
+| quant | size | ROUGE-L vs teacher | outputs differing from f16 |
+|---|---|---|---|
+| f16 (baseline) | 15.4MB | 0.484 | — |
+| **q8_0 (recommended default)** | **8.3MB** | **0.485** | 6/100 — paraphrases only, no garbling |
+| q6_k | 6.4MB | 0.490 | 11/100 — paraphrases only, no garbling |
+| q4_0 | 5.0MB | 0.465 | 29/100 — mostly paraphrases/reordering, a few repeated-word artifacts, and a couple of real topic drifts on ambiguous creative-writing prompts |
+
+**q8_0 and q6_k are both effectively free** — every difference from the f16 baseline at those two
+levels is a paraphrase, not a quality loss (the ROUGE-L numbers moving up slightly is quantization
+noise, not real improvement). **q6_k is the better default if you want the smallest safe file**
+(6.4MB vs q8_0's 8.3MB, same quality). **q4_0 has a real, modest cost** — visible in about 1 in 3
+cases, mostly cosmetic but occasionally changes what the title is actually about. Ship q4_0 only
+if the extra ~1.3–3.3MB genuinely matters for your target.
+
+```
+docker run --rm --entrypoint /app/llama-completion -v <local-dir>:/models ghcr.io/ggml-org/llama.cpp:light -m /models/titler-q8_0.gguf -p "how do I center a div horizontally with flexbox" -n 32 --temp 0
+```
+
+(or any native `llama-completion` build — `ghcr.io/ggml-org/llama.cpp:light` is just the easiest way to get one.)
+
+**⚠️ LM Studio and Ollama do not work with this model.** Both load the GGUF and tokenize correctly, but their inference backends never run the required encoder pass for T5 (encoder-decoder) architectures — they're built around decoder-only chat models — so the decoder generates from an empty cross-attention state and immediately emits EOS. You'll get a silent empty response on every prompt, from the chat UI *and* their raw completion APIs (`/v1/completions`, `/api/generate`) — confirmed directly against LM Studio's API on this exact model. This isn't a template or settings problem (a custom prompt template that strips chat formatting entirely still returns empty) — use `llama-completion` (or another tool that correctly calls `llama_encode()` before `llama_decode()` for encoder-decoder models) instead.
+
+Also note: `llama-completion`'s default sampler doesn't apply an ngram-repeat ban, so repeated words show up more often than in the ONNX runtimes (which use `no_repeat_ngram_size=2`) — tune `--repeat-penalty` if that matters for your use.
+
+llama.cpp's T5 converter requires a real SentencePiece `tokenizer.model`; this tokenizer was trained with HF `tokenizers` (Unigram, no `spiece.model`), so the GGUF was produced via a synthesized SentencePiece file built from the same piece/score vocabulary — see `scripts/make_spiece.py` and `scripts/export_gguf.py` in the training repo if you need to re-export from the fp32 weights.
+
+## Architecture
+
+From-scratch `T5ForConditionalGeneration` — no pretrained base, every parameter learned from the title task:
+
+| | |
+|---|---|
+| Layers | 3 encoder + 3 decoder |
+| d_model / d_ff | 256 / 1024 (ReLU FFN — gated variants cost ~30% more FFN params) |
+| Attention | 4 heads × 64 dims |
+| Vocab | 8,192 Unigram word-pieces (custom-trained; `<pad>`=0, `</s>`=1, `<unk>`=2) |
+| Parameters | 7.61M (embeddings 2.10M shared 3 ways; encoder 2.36M; decoder 3.15M) |
+| Artifact | 10.28MB int8 ONNX (the embedding table ships in both encoder and decoder graphs) |
+
+The tokenizer is trained on the task corpus itself (prompts + titles), stored as fast-tokenizer `tokenizer.json` — no `spiece.model` — and verified to tokenize identically in Python `tokenizers` and transformers.js, including unicode, emoji, and code snippets. Architecture and hyperparameters are unchanged since v1.2 — every improvement since then (v1.3 through this release) came from data, not the model.
+
+## Training data
+
+**481,488 (prompt → title) pairs** (455,382 train / 4,641 val / 4,641 test), all titles written by **Qwen3-14B-AWQ** (v1.0–v1.2) or **Qwen3.6-27B-A2.8B** (v1.4–v1.5 additions; both non-reasoning, vLLM/llama-server on a rented GPU).
+
+Real first-turn user messages, English, exact + MinHash near-deduped:
+
+| source | license | notes |
+|---|---|---|
+| [allenai/WildChat-1M](https://huggingface.co/datasets/allenai/WildChat-1M) | ODC-BY | primary source, all 313,876 deduped prompts |
+| [OpenAssistant/oasst2](https://huggingface.co/datasets/OpenAssistant/oasst2) | Apache-2.0 | root prompter messages |
+| [lmsys/chatbot_arena_conversations](https://huggingface.co/datasets/lmsys/chatbot_arena_conversations) | CC-BY-style click-through | filler diversity |
+| [lmsys/lmsys-chat-1m](https://huggingface.co/datasets/lmsys/lmsys-chat-1m) | custom gated agreement — **see license note below** | 114,581 new prompts, added in v1.5 specifically for its short/vague/casual first-turn distribution |
+
+**A note on the LMSYS-Chat-1M license, disclosed rather than glossed over.** Earlier releases of
+this model (through v1.4) deliberately excluded this source. Its license agreement includes:
+
+> "You should not distribute, copy, disclose, assign, sublicense, embed, host, or otherwise
+> transfer **the dataset** to any third party." ... "We may... require you to delete all copies
+> of the conversation dataset (in whole or in part)... You will promptly comply." ... The
+> agreement can be terminated "at any time, for any reason," requiring immediate cessation of use
+> and destruction of all copies.
+
+This model card takes the position that **publishing weights trained on this data is not the
+same as redistributing the dataset itself** — the raw conversation text is nowhere in the
+released artifacts, and the license explicitly permits both research and commercial use of the
+data. That's the standard most released models operate under when trained on similarly-gated
+corpora. But the deletion-on-request and any-time-termination clauses are real, ongoing
+obligations this project has taken on by including this source, and they're disclosed here rather
+than hidden: if LMSYS ever requests deletion of specific conversations or terminates the
+agreement, the honest response is to retrain without that data and republish — not something a
+public weights release can be quietly walked back from once downloaded. This tradeoff was made
+deliberately, after weighing it against the alternative (v1.4's numbers, which still had the
+collapse behavior described below) — see **Evaluation** for what the data bought.
+
+**Synthetic hard buckets (~27k across the project's history)** — the wild data barely contains
+some of the inputs a titler most needs to handle, so these were generated programmatically from
+randomized templates and titled by the same teacher process as the corresponding era's data:
+
+- **SQL** (8k): SELECT/CTE/UPDATE/DELETE/DDL/window-function statements over randomized schemas across 6 business domains, with and without a "why is this slow"-style ask.
+- **Log pastes** (8k): Python tracebacks, Java/Go/Node stack traces, k8s events, npm failures, timestamped app logs.
+- **Vague messages** (4k): "hey", "it broke again", "like I said before, make it better thanks".
+- **Markdown documents** (4k): meeting notes, READMEs, PRDs, changelogs.
+- **Hostile / grateful messages** (3k, v1.4): "you suck", "this is useless" / "thanks that worked!", "perfect, thanks" — added specifically to check the model doesn't collapse sentiment-bearing short inputs into generic help-boilerplate (see Training process).
+
+Validation rejects teacher outputs that fail format checks (word-count violations, format leaks); ~3.5% of the current pool was rejected this way.
+
+## Training process
+
+Five iterations; the throughline is **data was consistently the lever, architecture consistently wasn't**:
+
+- **v1.0** capped title labels at 16 tokens during preprocessing, on the assumption that 8-word titles ≈ 11 tokens. Wrong: with an 8k vocab, Title Case English runs ~2.5 tokens/word — the median title is 20 tokens — so **75% of the labels were truncated mid-word with EOS appended**, and the model faithfully learned to end titles mid-word. Required a retrain with the cap at 32. If you distill with a small custom vocab: **measure your label lengths in tokens, not words.**
+- **v1.1** from-scratch retrain on an augmented 209k set (200k wild subsample + 24k synthetic). Seq2SeqTrainer, fp16, LR 5e-4 cosine, batch 128, label smoothing 0.1, 10 epochs, input 512 / target 32 tokens, best-checkpoint by validation ROUGE-L.
+- **v1.2** identical architecture/recipe, retrained on the full 339,674-pair pool (all 313,876 deduped wild prompts instead of a 200k subsample). Also tried and rejected: a 21M-parameter, 2.75×-larger architecture (8+4 layers, d_model 320, gated-GELU) trained on the *same* 209k set v1.1 used — it improved ROUGE-L by only +0.004. Retraining the unscaled architecture on the full data pool instead improved it by +0.028 — seven times the gain, at zero size/latency cost. First evidence that data, not depth, was the bottleneck.
+- **v1.3** tried initializing from a pretrained checkpoint (`t5-efficient-tiny`, 15.6M params) instead of training from scratch, on the theory that pretrained language knowledge would help. It closed some of the ROUGE-L gap but introduced a new failure mode: **confident collapse to generic templates** ("User Asks for Assistance") on short/ambiguous inputs like "hi" or "are you there" — verified as genuine distributional collapse (the model actually prefers the generic answer, not a decoding bug), not something more training fixed.
+- **v1.4** root-caused v1.3's collapse: cross-entropy training rewards the population-common answer for genuinely underdetermined short inputs, and a pretrained base's "produce plausible boilerplate under uncertainty" habit made this worse, not better. Label smoothing 0.2 + the hostile/grateful synthetic buckets measurably weakened the collapse but didn't eliminate it. Also found: the from-scratch architecture matched or beat the pretrained one on every axis at half the size — the "pretrained oomph" wasn't needed, so pretrained init was dropped going forward.
+- **v1.5** (this release) added 114,581 new prompts from LMSYS-Chat-1M specifically targeting the short/vague/casual first-turn distribution the wild pool was thin on — the actual population the v1.3/v1.4 collapse was a symptom of having too few real examples for. Same architecture and recipe as v1.4's from-scratch variant, retrained on the enlarged 481,488-pair pool. This is what actually resolved the collapse (see Evaluation) — confirming the data-not-depth pattern held all the way through the failure mode that first looked like it might need architecture, not data.
+
+Total compute cost of the project through v1.2 — teacher inference, bake-offs, every training run — was under $20; v1.3 through v1.5's investigation and larger data pull added a further ~$10–15 in rented GPU time (RunPod RTX 4090, on-demand).
+
+## Evaluation
+
+Four gates, evaluated against the shipping int8 ONNX artifact:
+
+| gate | v1.2 | v1.4 (collapse-fix, from-scratch) | **v1.5 (current)** |
+|---|---|---|---|
+| ROUGE-L F1 vs teacher (test, n=500, beams=2, gate ≥0.40) | 0.484 | 0.473 | **0.511** |
+| Format compliance, raw (gate ≥99%) | 98.0% | 97.8% | 95.0% |
+| Hard-case suite, 68 adversarial prompts (gate ≥80% usable) | pass | pass, collapse still present | **pass, collapse resolved** |
+
+**Format compliance has never actually cleared its own ≥99% gate**, at any point in this
+project's history (v1.0 96.7%, v1.1 98.6%, v1.2 98.0%, v1.4 97.8%, v1.5 95.0%). This is a
+persistent, disclosed gap — not a regression introduced by this release — and is the concrete
+next target (see Next steps), not something papered over by only reporting the gates that pass.
+
+**The generic-collapse pattern found in v1.3/v1.4 is resolved in this release**, across every
+hard-case that previously exhibited it: "hi" now gets "Casual Greeting and Conversation Start"
+instead of the stuck "User Asks for Assistance" template; "you suck" gets the sentiment-
+appropriate "You Suck" instead of collapsing into generic help-boilerplate; no self-repetition
+anywhere in the suite (earlier models showed "Help with Help", "Test Test Discussion"). One real
+miss remains: a relationship/anniversary prompt was mistitled "Married 6-Year-Old Crime
+Podcasts" — a genuine hallucination, isolated rather than a repeated pattern.
+
+**A/B and absolute-usability judge gates have not been re-run since v1.1** — reported below as-is
+for historical context, not as current numbers:
+
+| gate | v1.0 | v1.1 |
+|---|---|---|
+| Blind A/B judge vs teacher, win-or-tie (≥ 55%) | 18% | 24% |
+| Absolute usability judge, overall | — | 67.7% |
+
+**On the A/B gate, honestly:** it asks a 7.6M model to tie a 14B+ model in blind pairwise comparison, which measures "is the student as good as its teacher" (no) rather than "are the titles usable" (largely yes, per the absolute-usability numbers and the hard-case suite). Re-running both judge gates on the current model remains on the list.
+
+**Judging methodology lessons, disclosed because they will bite you too:**
+- LLM judges that reason before answering can silently return an *empty* answer when the token budget is consumed by hidden reasoning — one judging run scored 1.3% usable purely because every verdict was an empty string. Always inspect raw judge responses before trusting aggregate numbers, and prefer non-reasoning models (or explicit reasoning budgets) for high-volume single-token verdicts. The same failure mode showed up again during v1.5's data-titling pass (a reasoning model burned its entire token budget on hidden `<think>` content with zero answer) — worth re-checking every time a new teacher/judge model is swapped in, not just once.
+- A/B judge order was randomized per-comparison to cancel position bias.
+
+## Performance analysis
+
+*(This section is v1.1's judged breakdown, not yet re-measured at the same granularity for any
+later release — the hard-case suite in Evaluation is the current qualitative evidence for v1.5.)*
+
+On mainstream prompts, outputs split roughly into three tiers:
+
+- **~50% teacher-grade** — frequently near-identical to the teacher's title ("Taekwondo vs Karate Comparison", "Introduction to Latent Class Models").
+- **~35% usable but visibly small-model** — right topic, clunky phrasing, occasionally missing the question's focus.
+- **~15% misses** — template fallback or a garbled word when the model loses the thread mid-word-assembly. Misses are almost always *bland or clumsy* rather than misleading.
+
+**Known limitations:** English only. Long rambling emotional prompts can garble. Rare jargon the tokenizer fragments heavily occasionally distorts a word. Input misspellings are sometimes faithfully copied into the title. Format compliance sits around 95–98% depending on release, not the ≥99% target.
+
+## Next steps
+
+1. **Format compliance is the open gap**, not architecture or more data — every release in this project's history has missed the ≥99% gate. Worth a dedicated pass at what `validate_title()` is actually rejecting; likely a small, fixable set of recurring cases (punctuation, casing, or length edge cases) rather than something fundamental.
+2. **Re-run the A/B and absolute-usability judge gates** on the current model — still v1.1's numbers, now four releases stale.
+3. Architecture scaling remains deprioritized: three separate experiments now (the 21M-param variant in v1.1→v1.2, and the pretrained-init detour in v1.3) found data mattered far more than capacity. If a future data expansion saturates and errors start looking capacity-shaped again, an **encoder-only** capacity increase (more encoder layers, decoder held at 3) is the config to reach for — encoder depth is nearly free at inference (runs once per title, not once per output token).
+
+## Files
+
+- `model.safetensors`, `config.json`, `generation_config.json` — fp32 PyTorch weights (~30MB) for fine-tuning or re-export
+- `tokenizer.json`, `tokenizer_config.json`, `special_tokens_map.json` — fast tokenizer (loads in transformers and transformers.js)
+- `onnx/encoder_model_quantized.onnx` (4.6MB), `onnx/decoder_model_merged_quantized.onnx` (5.7MB) — the int8 runtime artifact
+- `gguf/titler-f16.gguf` (15.4MB), `gguf/titler-q8_0.gguf` (8.3MB, recommended default), `gguf/titler-q6_k.gguf` (6.4MB, smallest-with-no-measured-cost), `gguf/titler-q4_0.gguf` (5.0MB, real but modest cost) — llama.cpp; see GGUF section above (**LM Studio/Ollama don't work** — use `llama-completion` or equivalent)
+
+## License & attribution
+
+Model weights: **Apache-2.0**. Training data includes material derived from WildChat-1M (ODC-BY — attribution to AI2/the WildChat authors), oasst2 (Apache-2.0), chatbot_arena_conversations, and LMSYS-Chat-1M (custom gated agreement — see the license note under Training data above for the specific clauses and the reasoning for including it). Titles were generated by Qwen3-14B and Qwen3.6-27B-A2.8B (both Apache-2.0). This model card discloses the full data recipe, including the parts that carry ongoing obligations, in the spirit of those licenses.