Hugging Face's logo
hqq's picture

hqq

sudoAimer
1 3
·

AI & ML interests

None yet

Recent Activity

reacted to Avifenesh's post with 👀 about 1 month ago
I added a small gate to the Gated DeltaNet layers of Qwen3.5 and trained it with GRPO under a length penalty. On Qwen3.5-4B it reasons 48% shorter than LoRA on MATH-500, and anytime accuracy goes from 0.786 to 0.852. A LoRA with about the same parameter count barely moves. The gate sits on beta, the per-token write strength. One matrix per layer, zero init, so it starts as a no-op. It started as a negative eigenvalue experiment. The negative range turned out not to matter. The gate did. Write-up: https://huggingface.co/blog/Avifenesh/beta-gate-qwen35 Adapters: https://huggingface.co/collections/Avifenesh/gate-qwen35-6accb63081800b2f2d583b80 Code: https://github.com/avifenesh/gdn-beta-gatehttps://huggingface.co/collections/Avifenesh/gate-qwen35
upvoted a collection 5 months ago
Gemma 4 QAT Q4_0
liked a dataset 9 months ago
pentagoniac/SemiKong_Training_Datset
View all activity

Organizations

None yet