Hugging Face's logo

GoudaAI
/
Cheddar-2.0-Instruct

Text Generation
Safetensors
English
gouda-cheddar
conversational
custom_code
Model card Files
xet
 Community
Erbium08 commited on
Commit
ed44e4c
·
verified ·
1 Parent(s): 8e8ba17

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +124 -0
README.md CHANGED
@@ -1,3 +1,127 @@
1
  ---
2
  license: mit
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: mit
3
+ datasets:
4
+ - HuggingFaceFW/fineweb-edu
5
+ - HuggingFaceH4/ultrachat_200k
6
+ language:
7
+ - en
8
+ base_model:
9
+ - GoudaAI/Cheddar-2.0
10
+ pipeline_tag: text-generation
11
  ---
12
+ # Cheddar-2.0 Instruct
13
+ [![gouda-2-0-banner.png](https://i.postimg.cc/G35qXN6M/gouda-2-0-banner.png)](https://postimg.cc/JtjNnTrJ)
14
+
15
+ Gouda-2.0 is the second generation family of models from GoudaAI all trained in less than 24 GPU hours (on a 4090 rtx). We release 4 open models ranging from 24m-302m parameters:
16
+ - Ricotta-2.0 (24m) - [huggingface](https://huggingface.co/GoudaAI/Ricotta-2.0)
17
+ - Gruyere-2.0 (63m) - [huggingface](https://huggingface.co/GoudaAI/Gruyere-2.0)
18
+ - Gouda-2.0 (132m) - [huggingface](https://huggingface.co/GoudaAI/Gouda-2.0)
19
+ - Cheddar-2.0 (302m) [huggingface](https://huggingface.co/GoudaAI/Cheddar-2.0)
20
+
21
+ And their instruct versions:
22
+ - Ricotta-2.0-Instruct - [huggingface](https://huggingface.co/GoudaAI/Ricotta-2.0-Instruct)
23
+ - Gruyere-2.0-Instruct - [huggingface](https://huggingface.co/GoudaAI/Gruyere-2.0-Instruct)
24
+ - Gouda-2.0-Instruct - [huggingface](https://huggingface.co/GoudaAI/Gouda-2.0-Instruct)
25
+ - **Cheddar-2.0-Instruct [huggingface](https://huggingface.co/GoudaAI/Cheddar-2.0-Instruct)**
26
+
27
+ Instruct versions are avaliable for chat on [gouda-ai](https://gouda-ai.vercel.app)
28
+
29
+ Training Code is available here:
30
+ - Source Code: [eoind08/GoudaCore-v2](https://github.com/eoind08/GoudaCore-v2)
31
+ - SFT Code: [eoind08/UltrachatSFT-gouda](https://github.com/eoind08/UltrachatSFT-gouda)
32
+
33
+ ## Quickstart - Uses Quickstart code from SmolLM-2
34
+
35
+ ```bash
36
+ pip install transformers
37
+ ```
38
+
39
+ #### Running the model on CPU/GPU/multi GPU
40
+ * _Using full precision_
41
+ ```python
42
+ # pip install transformers
43
+ from transformers import AutoModelForCausalLM, AutoTokenizer
44
+ checkpoint = "GoudaAI/Gouda-2.0-Instruct"
45
+ device = "cuda" # for GPU usage or "cpu" for CPU usage
46
+ tokenizer = AutoTokenizer.from_pretrained(checkpoint)
47
+ # for multiple GPUs install accelerate and do `model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto")`
48
+ model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)
49
+ messages = [{"role": "user", "content": "What is the capital of France."}]
50
+ input_text=tokenizer.apply_chat_template(messages, tokenize=False)
51
+ inputs = tokenizer.encode(input_text, return_tensors="pt").to(device)
52
+ outputs = model.generate(inputs, max_new_tokens=50, temperature=0.2, top_p=0.9, do_sample=True)
53
+ print(tokenizer.decode(outputs[0]))
54
+ ```
55
+
56
+ ## Pre Training
57
+ Each model had a self-imposed 24 GPU hour limit meaning the number of tokens the model sees gets smaller as the models get larger. For pre-training we targeted a conservative 18 hour run. Pretraining occurred on a 4090 RTX:
58
+
59
+ | Model | Parameters (m) | Tokens (m) | GPU Time (h) | Final Validation Loss |
60
+ |---------------------------|----------------|------------|--------------|-----------------------|
61
+ | Ricotta-2.0 | 24.1 | 20,701 | 16.94 | 3.21 |
62
+ | Gruyere-2.0 | 62.9 | 8,987 | 16.50 | 2.96 |
63
+ | Gouda-2.0 | 132.2 | 5,597 | 17.82 | 2.85 |
64
+ | **Cheddar-2.0** | **302.1** | **2,809** | **17.97** | **2.88** |
65
+
66
+ ## SFT
67
+ SFT took place on a fixed 0.25 epoch run of UltraChat-200k with 20% decay, this run was trained on a 3080 RTX. Overall, UltraChat was likely a poor choice, long conversations are not necessarily suitable for such small models. Later generations of Gouda will use Smol-Smoltalk, designed specifically for small models. Below are the GPU times, final val loss and tokens for each model:
68
+
69
+ | Model | Parameters (m) | Tokens (m) | GPU Time (h) | Final Validation Loss |
70
+ |---------------------------|----------------|------------|--------------|-----------------------|
71
+ | Ricotta-2.0 | 24.1 | 64.7 | 0.167 | 2.09 |
72
+ | Gruyere-2.0 | 62.9 | 64.7 | 0.334 | 1.84 |
73
+ | Gouda-2.0 | 132.2 | 64.7 | 0.583 | 1.73 |
74
+ | **Cheddar-2.0** | **302.1** | **64.6** | **1.325** | **1.74** |
75
+
76
+ ## Gouda-2.1
77
+ For 2.1 there will be 2 major changes:
78
+ - Dataset: Since Gouda-0.0.1, we have used the same 10b token dataset of fineweb-edu (as done by Andrej Karparthy). Of course the logical idea may be to simply extend it to a 20b corpus or larger however whilst we are no longer imposing a strict 18 hour training target, it would be preferable to have a smaller amount of high quality data rather than blasted the models with billions of tokens. As such we are moving to a custom 10b token corpus made up of more specific datasets to hopefully produce better factual recall and ability, at the potential sacrifice of coherence and linguistic ability.
79
+ - SFT & DPO: For the instruct models we have previously only used a randomly selected 50k conversations from UltraChat-200k as more conversations produced a degrading affect on the models eval scores and a tendency to default to "Sorry I cannot do X". For 2.1 we plan to use Smol-Smoltalk specifically designed for models around the Gouda suite size followed by DPO on UltraFeedback. More research will need to be done into the use of Smol-Smoltalk as early experiments on Ricotta-2.0 found worse benchmark and qualiative results when compared with Ricotta-2.0-Instruct (UltraChat)
80
+
81
+ ## Eval Results
82
+ Across evaluation benchmarks, the gouda both pretrained and instruct models display results comparable to models of a similar size despite having a substantially smaller token-base.
83
+
84
+ ### Cheddar-2.0 (302m, 5.6B tokens) vs Pythia-410m (300B tokens) - Closest Pythia Model, raw_acc
85
+ | Task | Gouda-2.0 | Pythia-160m | Gap |
86
+ |---------------|-----------|-------------|------|
87
+ | ARC-Easy | .518 | .521 | +7.6 |
88
+ | ARC-Challenge | .222 | .213 | +3.8 |
89
+ | SciQ | .762 | .811 | +1.3 |
90
+ | PIQA | .631 | .668 | +0.2 |
91
+ | WinoGrande | .515 | .537 | -1.9 |
92
+ | LAMBADA | .262 | .516 | -6.2 |
93
+
94
+
95
+ ### PRETRAINED MODELS - Name (Parameters / Tokens)
96
+ | Task | Ricotta-2.0 (24m / 20.70B) | Gruyère-2.0 (63m / 8.99B) | Gouda-2.0 (132m / 5.60B) | Cheddar-2.0 (302m / 2.8B) |
97
+ |---|---:|---:|---:|---:|
98
+ | LAMBADA (OpenAI) | 0.213 +- 0.006 |0.248 +- 0.020 | 0.266 +- 0.006 | 0.262 +- 0.006 |
99
+ | PIQA (acc_norm) | 0.596 +- 0.011 | 0.620 +- 0.011 | 0.626 +- 0.011 | 0.624 +- 0.011 |
100
+ | WinoGrande | 0.513 +- 0.014 | 0.522 +- 0.014 | 0.512 +- 0.014 | 0.515 +- 0.014 |
101
+ | WSC | 0.535 +- 0.03 | 0.535 +- 0.030 | 0.553 +- 0.030 | 0.520 +- 0.030 |
102
+ | ARC - Easy (acc_norm) | 0.403 +- 0.010 | 0.439 +- 0.010 | 0.458 +- 0.010 | 0.459 +- 0.010 |
103
+ | ARC - Challenge (acc_norm) | 0.226 +- 0.012 | 0.242 +- 0.013 | 0.255 +- 0.013 | 0.257 +- 0.013 |
104
+ | SciQ (acc_norm) | 0.612 +- 0.015 | 0.644 +- 0.015 | 0.669 +- 0.015 | 0.654 +- 0.015 |
105
+ | HellaSwag (acc_norm) | 0.283 +- 0.004 | 0.312 +- 0.005 | 0.326 +- 0.005 | 0.327 +- 0.005 |
106
+ | OpenBookQA (acc_norm) | 0.286 +- 0.020 | 0.284 +- 0.020 | 0.294 +- 0.020 | 0.316 +- 0.021 |
107
+ | BoolQ | 0.490 +- 0.009 | 0.607 +- 0.009 | 0.546 +- 0.009 | 0.571 +- 0.009 |
108
+ | COPA | 0.57 +- 0.050 | 0.62 +- 0.049 | 0.66 +- 0.048 | 0.600 +- 0.049 |
109
+ | TruthfulQA (MC2) | 0.454 +- 0.015 | 0.422 +- 0.015 | 0.411 +- 0.015 | 0.413 + 0.015 |
110
+ | WikiText (PPL ↓) | 77.9 | 44.8 | 38.9 | 38.9 |
111
+
112
+ ### INSTRUCT MODELS - Name (Parameters / SFT Tokens)
113
+ | Task | Ricotta-2.0-Instruct (24m / 64.7m) | Gruyère-2.0-Instruct (63m / 64.7m) | Gouda-2.0-Instruct (132m / 64.7m) | **Cheddar-2.0-Instruct (302m / 64.6m)** |
114
+ |---|---:|---:|---:|---:|
115
+ | LAMBADA (OpenAI) | 0.231 +- 0.006 | 0.265 +- 0.006 | 0.292 +- 0.006 | **0.283 +- 0.006** |
116
+ | PIQA (acc_norm) | 0.603 +- 0.011 | 0.613 +- 0.011 | 0.633 +- 0.011 | **0.643 +- 0.011** |
117
+ | WinoGrande | 0.515 +- 0.014 | 0.505 +- 0.014 | 0.524 +- 0.014 | **0.515 +- 0.014** |
118
+ | WSC | 0.520 +- 0.030 | 0.549 +- 0.030 | 0.520 +- 0.030 | **0.550 +- 0.030** |
119
+ | ARC - Easy (acc_norm) | 0.393 +- 0.010 | 0.428 +- 0.010 | 0.418 +- 0.010 | **0.455 +- 0.010** |
120
+ | ARC - Challenge (acc_norm) | 0.230 +- 0.012 | 0.236 +- 0.012 | 0.270 +- 0.013 | **0.272 +- 0.013** |
121
+ | SciQ (acc_norm) | 0.568 +- 0.016 | 0.628 +- 0.015 | 0.626 +- 0.015 | **0.653 +- 0.0151** |
122
+ | HellaSwag (acc_norm) | 0.282 +- 0.004 | 0.307 +- 0.005 | 0.328 +- 0.005 | **0.330 +- 0.005** |
123
+ | OpenBookQA (acc_norm) | 0.282 +- 0.020 | 0.298 +- 0.020 | 0.314 +- 0.021 | **0.310 +- 0.021** |
124
+ | BoolQ | 0.516 +- 0.009 | 0.608 +- 0.009 | 0.523 +- 0.009 | **0.601 +- 0.009** |
125
+ | COPA | 0.58 +- 0.050 | 0.6 +- 0.049 | 0.6 +- 0.049 | **0.65 +- 0.048** |
126
+ | TruthfulQA (MC2) | 0.434 +- 0.015 | 0.411 +- 0.015 | 0.393 +- 0.015 | **0.406 +- 0.015** |
127
+ | WikiText (PPL ↓) | 89.9 | 52.6 | 44.5 | **42.0** |