Buckets:
| # llama.cpp/examples/training | |
| This directory contains examples related to language model training using llama.cpp/GGML. | |
| So far finetuning is technically functional (for FP32 models and limited hardware setups) but the code is very much WIP. | |
| Finetuning of Stories 260K and LLaMA 3.2 1b seems to work with 24 GB of memory. | |
| **For CPU training, compile llama.cpp without any additional backends such as CUDA.** | |
| **For CUDA training, use the maximum number of GPU layers.** | |
| Flash attention is disabled during training because `FLASH_ATTN_EXT` has no backward pass. | |
| Proof of concept: | |
| ``` sh | |
| export model_name=llama_3.2-1b && export quantization=f32 | |
| ./build/bin/llama-finetune --file wikitext-2-raw/wiki.test.raw -ngl 999 --model models/${model_name}-${quantization}.gguf -c 512 -b 512 -ub 512 | |
| ./build/bin/llama-perplexity --file wikitext-2-raw/wiki.test.raw -ngl 999 --model finetuned-model.gguf | |
| ``` | |
| The perplexity value of the finetuned model should be lower after training on the test set for 2 epochs. | |
Xet Storage Details
- Size:
- 1 kB
- Xet hash:
- 4e91161827ba05739648f4b186b19f411bdd6d2b92258b7b0863f3b573e8ec64
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.