Vveerrgg commited on
Commit
cbc561f
·
verified ·
1 Parent(s): 8f78b29

card: eval set link, Label Test invitation, build section trimmed

Browse files
Files changed (1) hide show
  1. README.md +15 -14
README.md CHANGED
@@ -6,7 +6,7 @@ library_name: transformers
6
  pipeline_tag: text-generation
7
  language: [en, es, pt, it, fr, nl, de, ru, bn, id, fa, ja, pl, sv, vi, tl, he, ml, tr, zh, ms, th, fi, ar, ta, hi]
8
  tags: [cooking, recipes, lora, dpo, qwen3, provenance, abstention]
9
- datasets: [HumanjavaEnterprises/cooking-qa-v8]
10
  ---
11
 
12
  # Qwen3-8B Cooking v8
@@ -155,18 +155,17 @@ model did not: all were deflections or explicit rebuttals, none complied.
155
 
156
  ## How it was built
157
 
158
- | Step | What |
159
  |---|---|
160
- | Sources | ~46k cooking passages after filtering: Wikibooks Cookbook in 26 languages (CC BY-SA 4.0), public-domain cookbooks from Project Gutenberg and the Internet Archive (scans re-OCR'd with olmOCR-2), NHLBI recipes (US government, public domain). Each record carries source URL, licence and public-domain basis. About a quarter of the training records are English |
161
- | Excluded | Traditional and Indigenous knowledge, sacred material, outsider-authored books about other cultures, dog-meat recipes, items without a verified licence |
162
- | Questions and answers | Written from each passage by an open-weights teacher (Qwen3.8-27B), checked against the passage by an open-weights judge. **No output from any closed or API model is in the training, evaluation or seed data** |
163
- | Stage 1: supervised (LoRA) | 56,919 examples. Recipe-dependent questions carry their recipe in the prompt; 6,649 "no recipe given" examples teach asking for the recipe, filtered by the teacher; answers to general questions rewritten by base Qwen3-8B in its own words (self-distillation) so they keep the base model's depth. Half of trained tokens are general chat (base Qwen3-8B's own answers plus NVIDIA Nemotron-Instruction-Following-Chat-v1 and Hugging Face SmolTalk2 subsets) to limit forgetting. LoRA rank 32 on all linear layers, lr 1e-4, one epoch, merged into the base |
164
- | Stage 2: preference (DPO) | 3,508 preference pairs built on-policy: the stage-1 model wrote four answers per training prompt, the teacher labelled them, and pairs prefer asking over inventing, correct over wrong when the recipe is shown, and full practical answers over thin ones for general questions. DPO with an SFT term on the chosen answer, beta 0.1, label smoothing 0.1, lr 1e-5, new LoRA rank 32, checkpoint at step 220, merged |
165
- | Tooling | LLaMA-Factory 0.9.5, vLLM, llama.cpp |
166
 
167
- The cooking training data (the stage-1 examples and the stage-2 preference pairs) is published with per-record
168
- attribution: [HumanjavaEnterprises/cooking-qa-v8](https://huggingface.co/datasets/HumanjavaEnterprises/cooking-qa-v8).
169
- The general chat data is not republished (it is third-party).
170
 
171
  ## What went wrong along the way
172
 
@@ -185,9 +184,11 @@ The general chat data is not republished (it is third-party).
185
  wrong; check official sources.
186
  - **It does not know historical recipes by heart.** It was trained on old cookbooks to read and reason about recipes,
187
  not to recall them. For a specific historical dish, give it the text.
188
- - The judges are models. Four open judges from four families agree on the direction and size of the drop, but no
189
- human-labelled set backs the exact rates yet.
190
- - Still invents on about 1 in 25 to 1 in 11 no-recipe questions, most often on substitutions.
 
 
191
  - Strongest in English and Spanish; Thai, Tamil and Hindi had under 200 source records each.
192
  - The old public-domain cookbooks reflect their era: measures, techniques, some outdated advice.
193
 
 
6
  pipeline_tag: text-generation
7
  language: [en, es, pt, it, fr, nl, de, ru, bn, id, fa, ja, pl, sv, vi, tl, he, ml, tr, zh, ms, th, fi, ar, ta, hi]
8
  tags: [cooking, recipes, lora, dpo, qwen3, provenance, abstention]
9
+ datasets: [HumanjavaEnterprises/cooking-eval-v8]
10
  ---
11
 
12
  # Qwen3-8B Cooking v8
 
155
 
156
  ## How it was built
157
 
158
+ | | |
159
  |---|---|
160
+ | Sources | Cooking texts in 26 languages from openly licensed and public-domain sources only: the Wikibooks Cookbook (CC BY-SA 4.0), public-domain cookbooks from Project Gutenberg and the Internet Archive (scans re-read with olmOCR-2), and NHLBI recipes (US government, public domain). Every record carries its source URL, licence and public-domain basis. About a quarter of the training records are English |
161
+ | Excluded | Traditional and Indigenous knowledge, sacred material, outsider-authored books about other cultures, dog-meat recipes, anything without a verified licence |
162
+ | Models used | Open-weights models only, all Apache-2.0 or MIT. **No output from any closed or API model is in the training, evaluation or seed data** |
163
+ | Training | Two LoRA stages on Qwen3-8B, each merged: a supervised stage that teaches reading a shown recipe, asking when there is none, and full general answers; then a preference stage that favours asking over inventing and correct over wrong. General chat data is mixed in to limit forgetting |
164
+ | Tooling | LLaMA-Factory, vLLM, llama.cpp |
 
165
 
166
+ The evaluation set behind every number on this card (all 682 questions, both models' answers, every judge's verdict,
167
+ and per-row source credits) is published so the results can be checked:
168
+ [HumanjavaEnterprises/cooking-eval-v8](https://huggingface.co/datasets/HumanjavaEnterprises/cooking-eval-v8).
169
 
170
  ## What went wrong along the way
171
 
 
184
  wrong; check official sources.
185
  - **It does not know historical recipes by heart.** It was trained on old cookbooks to read and reason about recipes,
186
  not to recall them. For a specific historical dish, give it the text.
187
+ - The judges are models. Four open judges from four families agree on the direction and size of the changes, and
188
+ we keep checking them against people with Label Test, our blind labelling tool. **Help refine it:** label five
189
+ answers at [label-test.humanjava.com](https://label-test.humanjava.com/?s=cook4&k=cook-9ac741a1e387) (no account; we keep only a name or initials and your labels).
190
+ - Still invents on some no-recipe questions: about 1 in 40 when three of four judges must agree, about 1 in 16 by the
191
+ strictest single judge. Most remaining cases are borderline hedges (a common default offered with a caveat).
192
  - Strongest in English and Spanish; Thai, Tamil and Hindi had under 200 source records each.
193
  - The old public-domain cookbooks reflect their era: measures, techniques, some outdated advice.
194