Claude Opus 5.5

Model card: link the paper on arXiv

021325a verified 1 day ago

2.44 kB

---
license: other
license_name: falcon-llm-license
license_link: https://falconllm.tii.ae/falcon-terms-and-conditions.html
base_model: tiiuae/Falcon3-1B-Instruct
base_model_relation: adapter
library_name: peft
pipeline_tag: text-generation
language:
- en
datasets:
- HuggingFaceFW/fineweb-edu
tags:
- lora
- conceptizer
---
# Falcon3-1B-Instruct-Conceptizer

**Falcon3-1B-Instruct-Conceptizer is built using artificial intelligence technology from the Technology Innovation Institute.**

LoRA adapters for [Falcon3-1B-Instruct](https://huggingface.co/tiiuae/Falcon3-1B-Instruct) from the paper [*Conceptizer: Next-Token Prediction Is Not All You Need*](https://arxiv.org/abs/2610.33027) (Haoran Zheng and Chen Shani).

Conceptizer adds a fixed vocabulary of units of meaning alongside the model's tokenizer, so post-training rewards different wordings of the same meaning, such as *safety*, *protection* and *security*, not only the exact next word. The concepts enter only the training loss; the adapters generate ordinary text.

In the paper, Conceptizer gives the lowest held-out perplexity at concept-target positions against all three baselines and improves knowledge and sentiment tasks, while staying comparable on commonsense.

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
tokenizer = AutoTokenizer.from_pretrained("tiiuae/Falcon3-1B-Instruct")
model = AutoModelForCausalLM.from_pretrained("tiiuae/Falcon3-1B-Instruct", dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "FlyinGodzilla/Falcon3-1B-Instruct-Conceptizer")
```

The paper used transformers 5.0 and peft 0.19. The repository holds the paper's three training seeds: seed 0 loads by default, and seeds 37 and 42 load with `subfolder="seed37"` or `subfolder="seed42"`.

## Training

Rank-16 LoRA post-training on FineWeb-Edu text for 50,000 steps, with next-token prediction, the concept loss and a retention loss that keeps the model close to the base model; the paper has the details. To train your own model with Conceptizer, the [Conceptizer repository](https://github.com/HaoranZhengRaul/conceptizer-code) has the code, the concept vocabulary, the training data and the commands.

## License

Distributed under the [TII Falcon License](LICENSE); use must comply with TII's [Acceptable Use Policy](https://falconllm.tii.ae/falcon-acceptable-use-policy.html).

	---
	license: other
	license_name: falcon-llm-license
	license_link: https://falconllm.tii.ae/falcon-terms-and-conditions.html
	base_model: tiiuae/Falcon3-1B-Instruct
	base_model_relation: adapter
	library_name: peft
	pipeline_tag: text-generation
	language:
	- en
	datasets:
	- HuggingFaceFW/fineweb-edu
	tags:
	- lora
	- conceptizer
	---

	# Falcon3-1B-Instruct-Conceptizer

	Falcon3-1B-Instruct-Conceptizer is built using artificial intelligence technology from the Technology Innovation Institute.

	LoRA adapters for [Falcon3-1B-Instruct](https://huggingface.co/tiiuae/Falcon3-1B-Instruct) from the paper [Conceptizer: Next-Token Prediction Is Not All You Need](https://arxiv.org/abs/2610.33027) (Haoran Zheng and Chen Shani).

	Conceptizer adds a fixed vocabulary of units of meaning alongside the model's tokenizer, so post-training rewards different wordings of the same meaning, such as safety, protection and security, not only the exact next word. The concepts enter only the training loss; the adapters generate ordinary text.

	In the paper, Conceptizer gives the lowest held-out perplexity at concept-target positions against all three baselines and improves knowledge and sentiment tasks, while staying comparable on commonsense.

	## Usage

	```python
	import torch
	from transformers import AutoModelForCausalLM, AutoTokenizer
	from peft import PeftModel

	tokenizer = AutoTokenizer.from_pretrained("tiiuae/Falcon3-1B-Instruct")
	model = AutoModelForCausalLM.from_pretrained("tiiuae/Falcon3-1B-Instruct", dtype=torch.bfloat16, device_map="auto")
	model = PeftModel.from_pretrained(model, "FlyinGodzilla/Falcon3-1B-Instruct-Conceptizer")
	```

	The paper used transformers 5.0 and peft 0.19. The repository holds the paper's three training seeds: seed 0 loads by default, and seeds 37 and 42 load with `subfolder="seed37"` or `subfolder="seed42"`.

	## Training

	Rank-16 LoRA post-training on FineWeb-Edu text for 50,000 steps, with next-token prediction, the concept loss and a retention loss that keeps the model close to the base model; the paper has the details. To train your own model with Conceptizer, the [Conceptizer repository](https://github.com/HaoranZhengRaul/conceptizer-code) has the code, the concept vocabulary, the training data and the commands.

	## License

	Distributed under the [TII Falcon License](LICENSE); use must comply with TII's [Acceptable Use Policy](https://falconllm.tii.ae/falcon-acceptable-use-policy.html).