HypLoRA Gemma-7B Math-10K

This repository provides HypLoRA adapter weights trained on Math-10K for Gemma 7B.

Code release: HypLoRA (hf-checkpoint-v branch) - please use this branch for the released adapters.

Paper: Hyperbolic Fine-Tuning for Large Language Models.

All checkpoints: GGLabYale HypLoRA collection.

Adapter details

Field Value
Hugging Face adapter ID GGLabYale/hyplora-gemma-7b-math10k
Base model google/gemma-7b
Training dataset Math-10K
Adapter method HypLoRA (lora_type: hyplora-1.0)
PEFT adapter type LORA (custom HypLoRA implementation)
Rank (r) 32
LoRA alpha (lora_alpha) 64
Adapter files adapter_config.json, adapter_model.safetensors

Setup

Use the hf-checkpoint-v code branch.

git clone --branch hf-checkpoint-v https://github.com/marlin-codes/HypLoRA.git
cd HypLoRA
bash setup.sh
conda activate hyplora

The branch's setup.sh installs pinned dependencies and applies the project's custom HypLoRA PEFT changes. See the README for environment and runtime requirements.

Evaluation with this released adapter

In the cloned codebase, open scripts/table3/hyplora/gemma-7b/eval.sh and set its --lora_weights argument to:

--lora_weights "GGLabYale/hyplora-gemma-7b-math10k"

Then execute from the model-specific script directory:

cd scripts/table3/hyplora/gemma-7b/
bash eval.sh

Keep the other evaluation settings from the corresponding script.

Evaluation results

The following are evaluation results (accuracy) comparing standard LoRA and HypLoRA (K=1) for this base-model family and training task (higher is better).

Benchmark LoRA HypLoRA (K=1)
AQuA 0.437 0.515
MAWPS 0.87 0.928
SVAMP 0.72 0.774
GSM8K 0.615 0.741
Params 0.79% 0.79%

Citation

@inproceedings{yang2025hyperbolicfinetuning,
  title     = {Hyperbolic Fine-Tuning for Large Language Models},
  author    = {Menglin Yang and Ram Samarth B B and Aosong Feng and Bo Xiong and Jihong Liu and Irwin King and Rex Ying},
  booktitle = {Advances in Neural Information Processing Systems},
  volume    = {38},
  year      = {2025},
  url       = {https://arxiv.org/abs/2410.04010}
}
Downloads last month
46
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for GGLabYale/hyplora-gemma-7b-math10k

Base model

google/gemma-7b
Adapter
(9,196)
this model

Collection including GGLabYale/hyplora-gemma-7b-math10k

Paper for GGLabYale/hyplora-gemma-7b-math10k