Gryphe commited on
Commit
f5ccece
·
verified ·
1 Parent(s): 4d0a528

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -27,7 +27,7 @@ Normally when I finetune a model I train as much of it as possible, loading ever
27
 
28
  This specific tensor has a massive influence on a model's writing style, something I first discovered building MythoMax years ago. Gemma 31B is a VRAM-hungry monster, so the question became: how do I have the maximum impact with the minimum hardware requirements?
29
 
30
- The answer: freeze everything else. All 60 transformer layers, all the attention heads, all the MLPs — completely untouched. Only `lm_head` trains, which means VRAM requirements drop dramatically, training completes in a single overnight run on consumer hardware, and every single one of Gemma's capabilities remains fully intact. The model hasn't changed. Only the voice has, and it's done so in the best way possible. (Obligatory disclaimer: I might be biased towards my own.)
31
 
32
  I used the same data I had on me for my last Pantheon Reasoning release, with one notable exception - No instruct 24k set. 100% narrative data, certified cliché free.
33
 
 
27
 
28
  This specific tensor has a massive influence on a model's writing style, something I first discovered building MythoMax years ago. Gemma 31B is a VRAM-hungry monster, so the question became: how do I have the maximum impact with the minimum hardware requirements?
29
 
30
+ The answer: freeze everything else. All 60 transformer layers, all the attention heads, all the MLPs — completely untouched. Only `lm_head` trains, which means VRAM requirements drop dramatically, training completes in a single overnight run on consumer hardware, and every single one of Gemma's capabilities remains fully intact. The model hasn't changed. Only the voice has, and it's done so in the best way possible. (Obligatory disclaimer: I might be biased towards my own data.)
31
 
32
  I used the same data I had on me for my last Pantheon Reasoning release, with one notable exception - No instruct 24k set. 100% narrative data, certified cliché free.
33