license: other
license_name: falcon-llm-license
license_link: https://falconllm.tii.ae/falcon-terms-and-conditions.html
base_model: tiiuae/Falcon3-3B-Instruct
base_model_relation: adapter
library_name: peft
pipeline_tag: text-generation
language:
- en
datasets:
- HuggingFaceFW/fineweb-edu
tags:
- lora
- conceptizer
Falcon3-3B-Instruct-Conceptizer
Falcon3-3B-Instruct-Conceptizer is built using artificial intelligence technology from the Technology Innovation Institute.
LoRA adapters for Falcon3-3B-Instruct from the paper Conceptizer: Next-Token Prediction Is Not All You Need (Haoran Zheng and Chen Shani).
Conceptizer adds a fixed vocabulary of units of meaning alongside the model's tokenizer, so post-training rewards different wordings of the same meaning, such as safety, protection and security, not only the exact next word. The concepts enter only the training loss; the adapters generate ordinary text.
In the paper, Conceptizer gives the lowest held-out perplexity at concept-target positions against all three baselines and improves knowledge and sentiment tasks, while staying comparable on commonsense.
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
tokenizer = AutoTokenizer.from_pretrained("tiiuae/Falcon3-3B-Instruct")
model = AutoModelForCausalLM.from_pretrained("tiiuae/Falcon3-3B-Instruct", dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "FlyinGodzilla/Falcon3-3B-Instruct-Conceptizer")
The paper used transformers 5.0 and peft 0.19. The repository holds the paper's three training seeds: seed 0 loads by default, and seeds 37 and 42 load with subfolder="seed37" or subfolder="seed42".
Training
Rank-16 LoRA post-training on FineWeb-Edu text for 50,000 steps, with next-token prediction, the concept loss and a retention loss that keeps the model close to the base model; the paper has the details. To train your own model with Conceptizer, the Conceptizer repository has the code, the concept vocabulary, the training data and the commands.
License
Distributed under the TII Falcon License; use must comply with TII's Acceptable Use Policy.