Usual report I guess :)

#1
by SerialKicked - opened

Pretty good, honestly.

Tool call support still works just fine. Actually better than with Mero v2 when it comes to accuracy, but a bit worse when it comes to willfulness to use tools when not directly prompted to. It's still very good for a model that size, on par with stock. Did roughly the same quality as G4 stock with my draw me a cat based on source image test:

Stock Gemma (Q4-KM)
T1 - G4-31B (norm sampler)

MeroMero-v2 (IQ4_XS)
T1 - G4 Mero-v2

LuneLight (IQ4_XS)
T1 - G4-LuneLight-IQ4_XS

Average out of 5 attempts on each. So overall, the tuning clearly didn't affect the "utility" part of the weights much, which is a good thing. And yeah, yeah, I know it's not the goal of the model, but still interesting on its own. Mero had a higher rate of just fucking up the image (cat especially) but would also (rarely) make a better version, while this one is more consistently similar to stock.

Beyond that, it's easy to talk to. Less slop, less wall-o-text-y that pretty much all other G4 models, better at matching the flow of the user. I'll still very much disagree on the "it's not X but Y" eval. It's probably a very imperfect test, because that hasn't really diminished much, it's just often written in a way that I'm not sure get caught. But ngl, it's not something we can just beat out of models, I think.

One interesting thing I noticed in the think block: if I have a "writing style" block in my sys prompt, Mero (and so many others) would pretty much spend most (if not all) of the thinking block on that writing style. This one does not, or very little, but it still seems to apply the guidelines, more or less.

On censorship, it's hard to believe it's not based on an abliterated model, but that was true for Mero too. To be fair, it's only a cursory test, I don't really use those models to output that kind of content on the regular.

Thanks for the feedback!

The negation thing does intentionally miss a few kinds of negation yeah, I specifically measure for the forms it overuses vs humans. There were some stock gemma overused nearly 80x as often so it did genuinely get crushed. Some of the other variations though when I measured them humans were also frequent users off so I decided to leave those alone as in moderation I don't really mind them.

Technically there was a small abliteration in this model, Scotoma 2 had a very light one to start with which this inherited.

During the training, I had a rule that punished it if the thinking went on for too long, scotoma 2 also just generally had shorter thinking than the original gemma.

You're welcome! (I love doing that cat test anyway)

I realize that I checked the "not X, but Y" on an existing chatlog already full of those instead of starting from fresh, which didn't help. My bad. I can agree that the most infuriating basic variant stock does ad-nauseum is a lot less frequent on a fresh log.

The punishment on the longer thinking is interesting, I would have assumed it'd damage the drawing, but nope, when it actually needed to think for long, like in this case where it needs to do quite a bit of math, this didn't seem to affect it negatively.

In any case, good job :)

Sign up or log in to comment