metadata
base_model: allenai/OLMo-1B-hf
datasets:
- Codemaster67/Unichem_10M
language: en
library_name: peft
license: apache-2.0
tags:
- chemistry
- smiles
- olmo
- causal-lm
- qlora
- peft
- 4bit
OLMo-7B QLoRA Adapter -- Chemistry SMILES CPT
Model Description
This is a QLoRA (Quantized LoRA) adapter trained on top of allenai/OLMo-1B-hf for chemistry SMILES language modelling using the Codemaster67/Unichem_10M dataset.
The base model was loaded in 4-bit precision (NF4 quantization via bitsandbytes with double quantization) and LoRA adapter matrices were trained on top in bfloat16.
Decoupled learning rates are used: LoRA adapters train at 2e-05,
while embed_tokens and lm_head train at 2.0000000000000003e-06 (10x smaller) to
avoid catastrophic forgetting of the base vocabulary.
QLoRA / Quantization Configuration
| Parameter | Value |
|---|---|
| Quantization | NF4 (4-bit) |
| Double Quantization | True |
| Compute dtype | bfloat16 |
| Rank (r) | 64 |
| Alpha | 128 |
| Effective Scaling | 2.0 |
| Target Modules | all-linear |
| Dropout | 0.01 |
| RSLoRA | False |
| Modules to Save | embed_tokens, lm_head |
Decoupled Learning Rates
| Parameter Group | Learning Rate |
|---|---|
| LoRA adapters | 2e-05 |
| embed_tokens + lm_head | 2.0000000000000003e-06 (x0.1) |
Training Details
| Parameter | Value |
|---|---|
| Method | QLoRA (4-bit base + LoRA adapters) |
| Epochs | 1 |
| Learning Rate (LoRA) | 2e-05 |
| Learning Rate (embed/head) | 2.0000000000000003e-06 |
| Optimizer | AdamW 8-bit |
| Batch Size (per device) | 32 |
| Gradient Accumulation | 1 |
| Max Sequence Length | 512 |
| Warmup Ratio | 0.1 |
| Weight Decay | 0.01 |
| Effective Batch Size | 32 |
| Scheduler | Cosine |
| Precision | bf16 (adapters) / 4-bit NF4 (base) |
| Gradient Checkpointing | True |
| Validation Split | 5 % |
| Packed Training Sequences | 18907 |
| Packed Validation Sequences | 995 |
Training Results
| Metric | Value |
|---|---|
| Training Loss | 1.0676 |
| Validation Loss | 0.9724 |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_use_double_quant=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
)
base_model = AutoModelForCausalLM.from_pretrained(
"allenai/OLMo-1B-hf", quantization_config=bnb_config, trust_remote_code=True
)
model = PeftModel.from_pretrained(base_model, "Codemaster67/Unichem_10M_tokens")
tokenizer = AutoTokenizer.from_pretrained("Codemaster67/Unichem_10M_tokens", trust_remote_code=True)
smiles_input = "<|start_of_smiles|>CC(=O)Oc1ccccc1C(=O)O<|end_of_smiles|>"
inputs = tokenizer(smiles_input, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=False))
Intended Use
Chemistry-domain language modelling, SMILES generation and completion, and downstream molecular property prediction via fine-tuning.
Limitations
- QLoRA adapters only; requires the base model allenai/OLMo-1B-hf loaded in 4-bit to use.
- Trained primarily on SMILES strings; natural-language instruction-following ability may degrade compared to the base OLMo checkpoint.