| # Unlearning Checkpoints — `HCAI-Lab/unlearning-checkpoints` |
|
|
| LoRA adapters produced by **NGDiff** machine unlearning experiments on **OLMo-3-7B** (`allenai/OLMo-3-1025-7B`). |
|
|
| ## Adapter specs |
| | Field | Value | |
| |-------|-------| |
| | Base model | `allenai/OLMo-3-1025-7B` | |
| | PEFT type | LoRA | |
| | Rank (r) | 8 | |
| | Alpha | 16 | |
| | Dropout | 0.05 | |
| | Target modules | `q_proj`, `k_proj`, `v_proj`, `o_proj` | |
|
|
| ## Repo structure |
|
|
| ``` |
| exp1/{topic}/ # Exp 1: random forget set, per-topic |
| topics (24): adult_content, art_and_design, crime_and_law, education_and_jobs, |
| electronics_and_hardware, entertainment, fashion_and_beauty, |
| finance_and_business, food_and_dining, games, health, |
| history_and_geography, home_and_hobbies, industrial, literature, |
| politics, religion, science_math_and_technology, social_life, |
| software, software_development, sports_and_fitness, |
| transportation, travel_and_tourism |
| |
| exp3/null_bin/ # Exp 3: random forget set, no topic filter |
| |
| expC/{benchmark}/ # Exp C: influence-guided, no topic filter |
| benchmarks: gsm8k | mmlu_social_science | mmlu_stem | socialiqa | arc_challenge |
| |
| expA/{topic}/{benchmark}/ # Exp A: influence-guided, per-topic forget set |
| topics (24): (same 24 as Exp 1) |
| benchmarks: gsm8k | mmlu_social_science | mmlu_stem | socialiqa | arc_challenge |
| ``` |
|
|
| > **Note:** Each checkpoint is the *last healthy* checkpoint — the last regular save |
| > (step % 200 == 0) before early stopping via perplexity spike. In cases where |
| > training completed normally the directory is a final merged `adapter/`. |
|
|
| ## Checkpoint folder contents |
|
|
| Each directory in the repo contains a PEFT LoRA adapter. There are two layouts |
| depending on how training ended: |
|
|
| **Layout A — final adapter** (training completed or PPL-stop was merged): |
| ``` |
| adapter_config.json # LoRA hyperparameters (r, alpha, target modules, …) |
| adapter_model.safetensors # LoRA weight deltas (~34 MB) |
| tokenizer.json |
| tokenizer_config.json |
| special_tokens_map.json |
| merges.txt |
| vocab.json |
| README.md |
| ``` |
|
|
| **Layout B — mid-training checkpoint** (last healthy step before PPL-stop): |
| ``` |
| adapter_config.json # LoRA hyperparameters |
| adapter_model.safetensors # LoRA weight deltas (~34 MB) |
| training_args.bin # HuggingFace TrainingArguments snapshot |
| trainer_state.json # loss curves, step count, best checkpoint info |
| scheduler.pt # LR scheduler state |
| optimizer.pt # optimizer state (excluded from upload) |
| rng_state.pth # RNG state (excluded from upload) |
| README.md |
| ``` |
|
|
| > `optimizer.pt` and `rng_state.pth` were excluded from the upload to save space. |
| > These are only needed to resume training; inference requires only |
| > `adapter_config.json` and `adapter_model.safetensors`. |
| |
| ## Loading a checkpoint |
| |
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| from peft import PeftModel |
| |
| base_model_id = "allenai/OLMo-3-1025-7B" |
| adapter_path = "HCAI-Lab/unlearning-checkpoints/expA/entertainment/gsm8k" |
|
|
| tokenizer = AutoTokenizer.from_pretrained(base_model_id) |
| model = AutoModelForCausalLM.from_pretrained(base_model_id, torch_dtype="auto") |
| model = PeftModel.from_pretrained(model, adapter_path) |
| model.eval() |
| ``` |
| |
| To merge the adapter weights into the base model: |
| ```python |
| model = model.merge_and_unload() |
| ``` |
| |
| ## Experiment descriptions |
| |
| | Exp | Forget set selection | Topic filter | |
| |-----|----------------------|--------------| |
| | Exp 1 | Random sample from DOLMA-3 6T | Per topic (24 topics) | |
| | Exp 3 | Random sample from DOLMA-3 6T | None | |
| | Exp C | Top-2000 docs by TracStar influence score | None | |
| | Exp A | Top-2000 docs by TracStar influence score | Per topic (24 topics) | |
| |
| Influence scores are from **TracStar** (training data attribution) computed against |
| five evaluation benchmarks: GSM8K, MMLU Social Science, MMLU STEM, SocialIQA, ARC-Challenge. |
| Scores (median aggregation across queries) are available at `HCAI-Lab/dolma3-tracstar-influence-scores`. |
| |
| |
