File size: 3,714 Bytes
6a884f1 3a08378 6a884f1 3a08378 6a884f1 3a08378 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 | ---
base_model: Codemaster67/Olmo-7b-spe
datasets:
- Codemaster67/Causal_lm_chemistry_1M_rows
language: en
library_name: peft
license: apache-2.0
tags:
- chemistry
- smiles
- olmo
- causal-lm
- qlora
- peft
- 4bit
---
# OLMo-7B QLoRA Adapter — Chemistry SMILES CPT
## Model Description
This is a **QLoRA (Quantized LoRA)** adapter trained on top of
[Codemaster67/Olmo-7b-spe](https://huggingface.co/Codemaster67/Olmo-7b-spe) for chemistry
SMILES language modelling using the
[Codemaster67/Causal_lm_chemistry_1M_rows](https://huggingface.co/datasets/Codemaster67/Causal_lm_chemistry_1M_rows) dataset.
The base model was loaded in **4-bit precision** (NF4 quantization via
bitsandbytes with double quantization) and LoRA adapter matrices were
trained on top in bfloat16. This is the most memory-efficient training
configuration compared to full LoRA and full fine-tuning.
The base model's tokenizer was pre-extended with ~300 SPE (SMILES Pair
Encoding) chemistry tokens plus `<|start_of_smiles|>` / `<|end_of_smiles|>`
special tokens. The `embed_tokens` and `lm_head` layers are saved as
full (non-LoRA) trainable copies via `modules_to_save` because they were
resized during tokenizer extension.
## QLoRA / Quantization Configuration
| Parameter | Value |
|---|---|
| **Quantization** | NF4 (4-bit) |
| **Double Quantization** | True |
| **Compute dtype** | bfloat16 |
| **Rank (r)** | 64 |
| **Alpha** | 128 |
| **Effective Scaling** | 2.0 |
| **Target Modules** | all-linear |
| **Dropout** | 0.01 |
| **RSLoRA** | True (rank-stabilized) |
| **Modules to Save** | embed_tokens, lm_head |
## Training Details
| Parameter | Value |
|---|---|
| **Method** | QLoRA (4-bit base + LoRA adapters) |
| **Epochs** | 1 |
| **Learning Rate** | 1e-05 |
| **Optimizer** | AdamW 8-bit |
| **Batch Size (per device)** | 32 |
| **Gradient Accumulation** | 1 |
| **Max Sequence Length** | 512 |
| **Warmup Ratio** | 0.1 |
| **Weight Decay** | 0.01 |
| **Scheduler** | Cosine |
| **Precision** | bf16 (adapters) / 4-bit NF4 (base) |
| **Gradient Checkpointing** | True |
| **Augmentation** | OFF |
| **Training Samples** | 250000 |
| **Eval Samples** | 25000 |
## Evaluation Results
| Metric | Value |
|---|---|
| **Final Eval Loss** | 0.9892120957374573 |
| **Final Eval Perplexity** | 2.6891148723675795 |
| **Training Loss** | 0.5213 |
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_use_double_quant=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
)
base_model = AutoModelForCausalLM.from_pretrained(
"Codemaster67/Olmo-7b-spe", quantization_config=bnb_config, trust_remote_code=True
)
model = PeftModel.from_pretrained(base_model, "Codemaster67/Olmo-7b_250KQlora")
tokenizer = AutoTokenizer.from_pretrained("Codemaster67/Olmo-7b_250KQlora", trust_remote_code=True)
smiles_input = "<|start_of_smiles|>CC(=O)Oc1ccccc1C(=O)O<|end_of_smiles|>"
inputs = tokenizer(smiles_input, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=False))
```
## Intended Use
Chemistry-domain language modelling, SMILES generation and completion,
and downstream molecular property prediction via fine-tuning.
## Limitations
- QLoRA adapters only; requires the base model [Codemaster67/Olmo-7b-spe](https://huggingface.co/Codemaster67/Olmo-7b-spe) loaded in 4-bit to use.
- Trained primarily on SMILES strings; natural-language instruction-following
ability may degrade compared to the base OLMo checkpoint.
- Augmentation was disabled for this run.
|