Cheddar-2.0 Instruct

Gouda-2.0 is the second generation family of models from GoudaAI all trained in less than 24 GPU hours (on a 4090 rtx). We release 4 open models ranging from 24m-302m parameters:

Ricotta-2.0 (24m) - huggingface
Gruyere-2.0 (63m) - huggingface
Gouda-2.0 (132m) - huggingface
Cheddar-2.0 (302m) huggingface

And their instruct versions:

Ricotta-2.0-Instruct - huggingface
Gruyere-2.0-Instruct - huggingface
Gouda-2.0-Instruct - huggingface
Cheddar-2.0-Instruct huggingface

Instruct versions are avaliable for chat on gouda-ai

Training Code is available here:

Source Code: eoind08/GoudaCore-v2
SFT Code: eoind08/UltrachatSFT-gouda

Quickstart - Uses Quickstart code from SmolLM-2

pip install transformers

Running the model on CPU/GPU/multi GPU

Using full precision

# pip install transformers
from transformers import AutoModelForCausalLM, AutoTokenizer
checkpoint = "GoudaAI/Cheddar-2.0-Instruct"
device = "cuda" # for GPU usage or "cpu" for CPU usage
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
# for multiple GPUs install accelerate and do `model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto")`
model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)
messages = [{"role": "user", "content": "What is the capital of France."}]
input_text=tokenizer.apply_chat_template(messages, tokenize=False)
inputs = tokenizer.encode(input_text, return_tensors="pt").to(device)
outputs = model.generate(inputs, max_new_tokens=50, temperature=0.2, top_p=0.9, do_sample=True)
print(tokenizer.decode(outputs[0]))

Pre Training

Each model had a self-imposed 24 GPU hour limit meaning the number of tokens the model sees gets smaller as the models get larger. For pre-training we targeted a conservative 18 hour run. Pretraining occurred on a 4090 RTX:

Model	Parameters (m)	Tokens (m)	GPU Time (h)	Final Validation Loss
Ricotta-2.0	24.1	20,701	16.94	3.21
Gruyere-2.0	62.9	8,987	16.50	2.96
Gouda-2.0	132.2	5,597	17.82	2.85
Cheddar-2.0	302.1	2,809	17.97	2.88

SFT

SFT took place on a fixed 0.25 epoch run of UltraChat-200k with 20% decay, this run was trained on a 3080 RTX. Overall, UltraChat was likely a poor choice, long conversations are not necessarily suitable for such small models. Later generations of Gouda will use Smol-Smoltalk, designed specifically for small models. Below are the GPU times, final val loss and tokens for each model:

Model	Parameters (m)	Tokens (m)	GPU Time (h)	Final Validation Loss
Ricotta-2.0	24.1	64.7	0.167	2.09
Gruyere-2.0	62.9	64.7	0.334	1.84
Gouda-2.0	132.2	64.7	0.583	1.73
Cheddar-2.0	302.1	64.6	1.325	1.74

Gouda-2.1

For 2.1 there will be 2 major changes:

Dataset: Since Gouda-0.0.1, we have used the same 10b token dataset of fineweb-edu (as done by Andrej Karparthy). Of course the logical idea may be to simply extend it to a 20b corpus or larger however whilst we are no longer imposing a strict 18 hour training target, it would be preferable to have a smaller amount of high quality data rather than blasted the models with billions of tokens. As such we are moving to a custom 10b token corpus made up of more specific datasets to hopefully produce better factual recall and ability, at the potential sacrifice of coherence and linguistic ability.
SFT & DPO: For the instruct models we have previously only used a randomly selected 50k conversations from UltraChat-200k as more conversations produced a degrading affect on the models eval scores and a tendency to default to "Sorry I cannot do X". For 2.1 we plan to use Smol-Smoltalk specifically designed for models around the Gouda suite size followed by DPO on UltraFeedback. More research will need to be done into the use of Smol-Smoltalk as early experiments on Ricotta-2.0 found worse benchmark and qualiative results when compared with Ricotta-2.0-Instruct (UltraChat)

Eval Results

Across evaluation benchmarks, the gouda both pretrained and instruct models display results comparable to models of a similar size despite having a substantially smaller token-base.

Cheddar-2.0 (302m, 5.6B tokens) vs Pythia-410m (300B tokens) - Closest Pythia Model, raw_acc

Task	Gouda-2.0	Pythia-160m	Gap
ARC-Easy	.518	.521	-0.3
ARC-Challenge	.222	.213	+0.9
SciQ	.762	.811	-4.9
PIQA	.631	.668	-3.7
WinoGrande	.515	.537	-2.2
LAMBADA	.262	.516	-25.4

PRETRAINED MODELS - Name (Parameters / Tokens)

Task	Ricotta-2.0 (24m / 20.70B)	Gruyère-2.0 (63m / 8.99B)	Gouda-2.0 (132m / 5.60B)	Cheddar-2.0 (302m / 2.8B)
LAMBADA (OpenAI)	0.213 +- 0.006	0.248 +- 0.020	0.266 +- 0.006	0.262 +- 0.006
PIQA (acc_norm)	0.596 +- 0.011	0.620 +- 0.011	0.626 +- 0.011	0.624 +- 0.011
WinoGrande	0.513 +- 0.014	0.522 +- 0.014	0.512 +- 0.014	0.515 +- 0.014
WSC	0.535 +- 0.03	0.535 +- 0.030	0.553 +- 0.030	0.520 +- 0.030
ARC - Easy (acc_norm)	0.403 +- 0.010	0.439 +- 0.010	0.458 +- 0.010	0.459 +- 0.010
ARC - Challenge (acc_norm)	0.226 +- 0.012	0.242 +- 0.013	0.255 +- 0.013	0.257 +- 0.013
SciQ (acc_norm)	0.612 +- 0.015	0.644 +- 0.015	0.669 +- 0.015	0.654 +- 0.015
HellaSwag (acc_norm)	0.283 +- 0.004	0.312 +- 0.005	0.326 +- 0.005	0.327 +- 0.005
OpenBookQA (acc_norm)	0.286 +- 0.020	0.284 +- 0.020	0.294 +- 0.020	0.316 +- 0.021
BoolQ	0.490 +- 0.009	0.607 +- 0.009	0.546 +- 0.009	0.571 +- 0.009
COPA	0.57 +- 0.050	0.62 +- 0.049	0.66 +- 0.048	0.600 +- 0.049
TruthfulQA (MC2)	0.454 +- 0.015	0.422 +- 0.015	0.411 +- 0.015	0.413 + 0.015
WikiText (PPL ↓)	77.9	44.8	38.9	38.9

INSTRUCT MODELS - Name (Parameters / SFT Tokens)

Task	Ricotta-2.0-Instruct (24m / 64.7m)	Gruyère-2.0-Instruct (63m / 64.7m)	Gouda-2.0-Instruct (132m / 64.7m)	Cheddar-2.0-Instruct (302m / 64.6m)
LAMBADA (OpenAI)	0.231 +- 0.006	0.265 +- 0.006	0.292 +- 0.006	0.283 +- 0.006
PIQA (acc_norm)	0.603 +- 0.011	0.613 +- 0.011	0.633 +- 0.011	0.643 +- 0.011
WinoGrande	0.515 +- 0.014	0.505 +- 0.014	0.524 +- 0.014	0.515 +- 0.014
WSC	0.520 +- 0.030	0.549 +- 0.030	0.520 +- 0.030	0.550 +- 0.030
ARC - Easy (acc_norm)	0.393 +- 0.010	0.428 +- 0.010	0.418 +- 0.010	0.455 +- 0.010
ARC - Challenge (acc_norm)	0.230 +- 0.012	0.236 +- 0.012	0.270 +- 0.013	0.272 +- 0.013
SciQ (acc_norm)	0.568 +- 0.016	0.628 +- 0.015	0.626 +- 0.015	0.653 +- 0.0151
HellaSwag (acc_norm)	0.282 +- 0.004	0.307 +- 0.005	0.328 +- 0.005	0.330 +- 0.005
OpenBookQA (acc_norm)	0.282 +- 0.020	0.298 +- 0.020	0.314 +- 0.021	0.310 +- 0.021
BoolQ	0.516 +- 0.009	0.608 +- 0.009	0.523 +- 0.009	0.601 +- 0.009
COPA	0.58 +- 0.050	0.6 +- 0.049	0.6 +- 0.049	0.65 +- 0.048
TruthfulQA (MC2)	0.434 +- 0.015	0.411 +- 0.015	0.393 +- 0.015	0.406 +- 0.015
WikiText (PPL ↓)	89.9	52.6	44.5	42.0