---
base_model: allenai/OLMo-1B-hf
datasets:
- Codemaster67/Unichem_10M
language: en
library_name: peft
license: apache-2.0
tags:
- chemistry
- smiles
- olmo
- causal-lm
- qlora
- peft
- 4bit
---

# OLMo-7B QLoRA Adapter -- Chemistry SMILES CPT

## Model Description

This is a **QLoRA (Quantized LoRA)** adapter trained on top of
[allenai/OLMo-1B-hf](https://huggingface.co/allenai/OLMo-1B-hf) for chemistry
SMILES language modelling using the
[Codemaster67/Unichem_10M](https://huggingface.co/datasets/Codemaster67/Unichem_10M) dataset.

The base model was loaded in **4-bit precision** (NF4 quantization via
bitsandbytes with double quantization) and LoRA adapter matrices were
trained on top in bfloat16.

**Decoupled learning rates** are used: LoRA adapters train at `2e-05`,
while `embed_tokens` and `lm_head` train at `2.0000000000000003e-06` (10x smaller) to
avoid catastrophic forgetting of the base vocabulary.

## QLoRA / Quantization Configuration

| Parameter | Value |
|---|---|
| **Quantization** | NF4 (4-bit) |
| **Double Quantization** | True |
| **Compute dtype** | bfloat16 |
| **Rank (r)** | 64 |
| **Alpha** | 128 |
| **Effective Scaling** | 2.0 |
| **Target Modules** | all-linear |
| **Dropout** | 0.01 |
| **RSLoRA** | False |
| **Modules to Save** | embed_tokens, lm_head |

## Decoupled Learning Rates

| Parameter Group | Learning Rate |
|---|---|
| **LoRA adapters** | 2e-05 |
| **embed_tokens + lm_head** | 2.0000000000000003e-06 (x0.1) |

## Training Details

| Parameter | Value |
|---|---|
| **Method** | QLoRA (4-bit base + LoRA adapters) |
| **Epochs** | 1 |
| **Learning Rate (LoRA)** | 2e-05 |
| **Learning Rate (embed/head)** | 2.0000000000000003e-06 |
| **Optimizer** | AdamW 8-bit |
| **Batch Size (per device)** | 32 |
| **Gradient Accumulation** | 1 |
| **Max Sequence Length** | 512 |
| **Warmup Ratio** | 0.1 |
| **Weight Decay** | 0.01 |
| **Effective Batch Size** | 32 |
| **Scheduler** | Cosine |
| **Precision** | bf16 (adapters) / 4-bit NF4 (base) |
| **Gradient Checkpointing** | True |
| **Validation Split** | 5 % |
| **Packed Training Sequences** | 18907 |
| **Packed Validation Sequences** | 995 |

## Training Results

| Metric | Value |
|---|---|
| **Training Loss** | 1.0676 |
| **Validation Loss** | 0.9724 |

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)
base_model = AutoModelForCausalLM.from_pretrained(
    "allenai/OLMo-1B-hf", quantization_config=bnb_config, trust_remote_code=True
)
model = PeftModel.from_pretrained(base_model, "Codemaster67/Unichem_10M_tokens")
tokenizer = AutoTokenizer.from_pretrained("Codemaster67/Unichem_10M_tokens", trust_remote_code=True)

smiles_input = "<|start_of_smiles|>CC(=O)Oc1ccccc1C(=O)O<|end_of_smiles|>"
inputs = tokenizer(smiles_input, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=False))
```

## Intended Use

Chemistry-domain language modelling, SMILES generation and completion,
and downstream molecular property prediction via fine-tuning.

## Limitations

- QLoRA adapters only; requires the base model [allenai/OLMo-1B-hf](https://huggingface.co/allenai/OLMo-1B-hf) loaded in 4-bit to use.
- Trained primarily on SMILES strings; natural-language instruction-following
  ability may degrade compared to the base OLMo checkpoint.