File size: 2,440 Bytes
9721004 7dd4c6d 9721004 2c73856 9721004 5219c0d 9721004 2c73856 9721004 5219c0d 9721004 2c73856 9721004 5219c0d 9721004 2c73856 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 | ---
license: other
license_name: falcon-llm-license
license_link: https://falconllm.tii.ae/falcon-terms-and-conditions.html
base_model: tiiuae/Falcon3-3B-Instruct
base_model_relation: adapter
library_name: peft
pipeline_tag: text-generation
language:
- en
datasets:
- HuggingFaceFW/fineweb-edu
tags:
- lora
- conceptizer
---
# Falcon3-3B-Instruct-Conceptizer
**Falcon3-3B-Instruct-Conceptizer is built using artificial intelligence technology from the Technology Innovation Institute.**
LoRA adapters for [Falcon3-3B-Instruct](https://huggingface.co/tiiuae/Falcon3-3B-Instruct) from the paper [*Conceptizer: Next-Token Prediction Is Not All You Need*](https://arxiv.org/abs/2610.33027) (Haoran Zheng and Chen Shani).
Conceptizer adds a fixed vocabulary of units of meaning alongside the model's tokenizer, so post-training rewards different wordings of the same meaning, such as *safety*, *protection* and *security*, not only the exact next word. The concepts enter only the training loss; the adapters generate ordinary text.
In the paper, Conceptizer gives the lowest held-out perplexity at concept-target positions against all three baselines and improves knowledge and sentiment tasks, while staying comparable on commonsense.
## Usage
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
tokenizer = AutoTokenizer.from_pretrained("tiiuae/Falcon3-3B-Instruct")
model = AutoModelForCausalLM.from_pretrained("tiiuae/Falcon3-3B-Instruct", dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "FlyinGodzilla/Falcon3-3B-Instruct-Conceptizer")
```
The paper used transformers 5.0 and peft 0.19. The repository holds the paper's three training seeds: seed 0 loads by default, and seeds 37 and 42 load with `subfolder="seed37"` or `subfolder="seed42"`.
## Training
Rank-16 LoRA post-training on FineWeb-Edu text for 50,000 steps, with next-token prediction, the concept loss and a retention loss that keeps the model close to the base model; the paper has the details. To train your own model with Conceptizer, the [Conceptizer repository](https://github.com/HaoranZhengRaul/conceptizer-code) has the code, the concept vocabulary, the training data and the commands.
## License
Distributed under the [TII Falcon License](LICENSE); use must comply with TII's [Acceptable Use Policy](https://falconllm.tii.ae/falcon-acceptable-use-policy.html).
|