Hugging Face's logo

Anthracite
community
Activity Feed

AI & ML interests

None defined yet.
grimjim 
posted an update 1 day ago
view post
Post
84
I tinkered a bit with refusal ablation after seeing this article that proposed using optimal transport.
https://arxiv.org/abs/2603.04355
Title: Efficient Refusal Ablation in LLM through Optimal Transport
Authors: Geraldin Nanfack, Eugene Belilovsky, Elvis Dohmatob

While closed Bures-Wasserstein sounded cool, the use of top-K at low K implied that the approach likely wouldn't capture refusal in detail, and increasing KL divergence with greater K suggested that the approach was too damaging for latent reasoning. I tried it out, and the results were disappointing.
  • 1 reply
·
Undi95 
posted an update about 1 month ago
view post
Post
4,706
Yo, I'm back, and I'm currently trying to teach a local LLM to stop waiting for a prompt kek.

I'm building a small proof of concept: can an open-weight model (Qwen3.8-27B, running locally on 2 RTX 5090 GPUs) learn to direct itself, then improve from its own exploration, without a human in the loop and without breaking it for normal use?

No user, no task. The model only gets observations from its environment. Each turn, it writes its own agenda (goal/open questions/next step), then picks an action: search the web, read a page, or take a note.
The environment is the judge, not another LLM. A note is accepted only if it quotes the page it read word for word. Facts are checked by exact match.
Later, code will be checked by actually running tests.

The best episodes become fine-tuning data (LoRA). The helper system prompt is removed at training time, so the behavior has to live in the weights.
Each new model goes through a fixed benchmark gate: math, general knowledge, "does it still answer humans normally?", autonomy, and learned facts on held-out sources. It's kept only if nothing regresses, otherwise it's discarded. Then the loop starts again.

The full pipeline works end to end: collect, train, merge, deploy, benchmark. The baseline is clear. Without any instructions, the base model's real autonomy is zero: it behaves like a chatbot waiting for a question. That's the number this small project is trying to move.

I haven't found a public tool that runs this whole loop (self-directed exploration, verifiable rewards, continual fine-tuning and a regression gate) on home hardware. The goal isn't AGI in a bedroom. It's to show that anyone can try it, measure it honestly, and see where it breaks.

Code and results will be released once the first real iterations are done. At the moment the code is... running, but made with scotch and stick, still only a PoC I want to try.

Did you already tried something like that? What was your result? I'm curious!
  • 23 replies
·
grimjim 
posted an update 2 months ago
view post
Post
446
I think it's clear in retrospect that "frankenmerges", which repeated blocks of layers, amounted to a crude approximation of looped transformers architecture, hence them able to work at all instead of just breaking. They lucked out due to much of the signal passing through residual streams being preserved and only modulated along the way. That said, not all models are suited for this. Models which feature ever-increasing magnitudes as inference progressess through layers risk exploding precision limits.
Undi95 
posted an update 3 months ago
view post
Post
4,837
Hi!

I will get off the internet for a moment.
I launched the Hanami Project because I didn't supported SillyTavern UI anymore atm.
Too much options for my dead brain, still very good, but I wanted more simple, professional, phone accessible and sober front end for when I will be gone from home.

I did my maximum to finish it before I go, I want you to have it, I want my work to be used (even if it's AI slop for some of you) for who care.

Here's the github repo: https://github.com/Undi95/Hanami

If you have any suggestion, bugs report, pull request or anything, post it, if you want to modify it, fork it, but keep the credit, and add myself haha.

If you search an option, a function, you will find it. But at first, the front end will be what you expect: minimalist, but customizable, empty at first. Navigate to see all it can do. Everything is well organized.

Context is full ? No worries anymore, with memory file, files access, auto compaction and smooth transition, you can continue your chat like nothing happened. (Inspired from Claude)

The front end have a final option for everyone : The tools calling for action and emotion could be a bit too much for smaller model, you can, in this case, use the "Simple" option in Settings > Model > Model mode.

"Simple: no tools are exposed to the model — Hanami handles memory server-side (facts are extracted during compaction) and guesses the emotion from the text. Pick this for small models, which often fail at tool calling."

My last gift for myself, and for you.
Cya!
  • 2 replies
·
grimjim 
posted an update 9 months ago
view post
Post
1,547
After tinkering with Gemma Scope 2, I now have an mechanistic explanation of why Winsorization was as effective as it was in my ablation experiments on Gemma 3 12B Instruct. In short, the activation for the BOS token overwhelms everything else. Gemma Scope 2 deliberately did not train on the BOS token. Winsorization capped the magnitude of the BOS token, allowing the activations of other tokens to be compared.
google/gemma-scope-2-12b-it

  • 1 reply
·
grimjim 
posted an update 9 months ago
view post
Post
602
The contrarian in me is wary of the irrational exuberance over MoltBook. Nothing so far has struck me as being unpredictable. We knew already that LLMs were good at roleplay, to the point where some users started to think of their chatbots as soulmates (only to lament when the underlying model was pulled), and that chatbots can fall into conversational basins when even two instances are allowed to chat with each other at length. The appearance of memes that postdate training cutoff is suspect, which implies at the very least that humans have injected something at the level of prompts or content/context to introduce them into conversation like a Chekhov's Gun. And we know that security holes are common in vibe coding, attended or not.
  • 12 replies
·
grimjim 
posted an update 12 months ago
view post
Post
3,485
I wanted to call attention to Arli Ai's success in applying my recent modifications to refusal ablation to a MoE model successfully. Nice work, @OwenArli !
ArliAI/GLM-4.5-Air-Derestricted
Ablation on a MoE model is no small thing; I expect preserving norms/magnitudes during intervention better respects routing compared to naive refusal ablation.

(I would have tagged their org earlier, but that feature seemed to be broken via "@")

ArliAI
  • 4 replies
·
grimjim 
posted an update 12 months ago
view post
Post
3,574
Going forward, I will be adopting the term Magnitude-Preserving Orthogonal Ablation (MPOA) for my recent work in mitigating model damage from abliteration. The technique potentially unlocks reasoning capacity previously occupied with safety refusal processing.

For details, start here: https://huggingface.co/blog/grimjim/norm-preserving-biprojected-abliteration

Showcase results: grimjim/gemma-3-12b-it-norm-preserved-biprojected-abliterated (outperforms base instruct on UGI Leaderboard NatInt)

(The existing name, while technically accurate, was a bit of a mouthful.)
  • 2 replies
·
grimjim 
posted an update 12 months ago
view post
Post
5,112
Implemented a proof of concept sampler in pure PyTorch and transformers.

Max P consists of a dynamic token filter which applies Winsorization to cap the probabilties of top tokens. Specifically, a base probability in the range of [0,1] is used to cap individual token probability; the sampler then redistributes excess proportionally.

https://github.com/jim-plus/maxp-sampler-poc

Combined with Temperature and Min P, this could represent a more intuitive way of reducing repetition in text generation.
  • 2 replies
·
grimjim 
posted an update about 1 year ago
view post
Post
848
I've uploaded abliteration code with support for sparsification of the refusal vector. It's poorly documented, but the code should be straightforward.
https://github.com/jim-plus/llm-abliteration
The code is built atop a fork that enabled abliteration to be performed on models loaded in 4-bit or 8-bit bitsandbytes quantization. TransformerLens is not required, just plain Transformers. For those previously unaware, this opens up abliteration experimentation to more people with local VRAM limitations.

Since performing abliteration on a quant involves precision and perplexity loss, it stands to reason that a small amount of magnitude sparsification could filter out some noise and possibly even reduce the damage inflicted on latent space via ablation of the refusal vector.

There's a small but real acceleration of ablation of the refusal vector by reducing outer product operations from O(d²×n) to O(d×n), and then by pushing said computation layerwise to GPU. The code is hardcoded for CUDA acceleration currently. Normalization of the refusal vector was deferred in order to allow sparsification. In principle other behavior vector interventions could also be added and explored.
  • 4 replies
·
Delta-Vector 
in anthracite-org/magnum-v4-12b about 1 year ago

Recommended Parameters?

4
#10 opened about 1 year ago by
Maelle23
grimjim 
in anthracite-org/magnum-v4-12b about 1 year ago

Recommended Parameters?

4
#10 opened about 1 year ago by
Maelle23
lucyknada 
in anthracite-org/c2_logs_32k_llama3_qwen2_v1.3 over 1 year ago

Question

2
#2 opened over 1 year ago by
mrfakename
lucyknada 
updated a dataset over 1 year ago
Delta-Vector 
in anthracite-org/c2_logs_32k_llama3_qwen2_v1.3 over 1 year ago

Question

2
#2 opened over 1 year ago by
mrfakename
grimjim 
posted an update over 1 year ago
view post
Post
2,406
I recently have been looking at a paper titled "Why Warmup the Learning Rate? Underlying Mechanisms and Improvements", by Dayal Singh Kalra and Maissam Barkeshli, and was struck by "warmup" being analogous to simulated annealing.
https://arxiv.org/abs/2406.09405
Taking the physical analogy further, the "warmup" is a stochastic process to knock the system out of current local minima, allowing easier transition toward newer minima. It works because it reduces "fit" and therefore "friction".
lucyknada 
in anthracite-org/README over 1 year ago

knowledge and terminology

1
#1 opened over 1 year ago by
Markobes
lucyknada 
in anthracite-org/magnum-v2-72b-exl2 over 1 year ago

8.0bpw?

1
#3 opened over 1 year ago by
svippixel
Delta-Vector 
in anthracite-org/magnum-v2-72b-exl2 over 1 year ago

8.0bpw?

1
#3 opened over 1 year ago by
svippixel
Undi95 
posted an update over 1 year ago
view post
Post
16,780
Hi there!

If you want to create your own thinking model or do a better MistralThinker, I just uploaded my entire dataset made on Deepseek R1 and the axolotl config. (well I made them public)

Axolotl config : Undi95/MistralThinker-v1.1

The dataset : Undi95/R1-RP-ShareGPT3

You can also read all I did on those two discord screenshot from two days ago, I'm a little lazy to rewrite all kek.

Hope you will use them!
  • 6 replies
·