Instructions to use Codemaster67/Unichem_10M_tokens with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use Codemaster67/Unichem_10M_tokens with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("allenai/OLMo-1B-hf") model = PeftModel.from_pretrained(base_model, "Codemaster67/Unichem_10M_tokens") - Notebooks
- Google Colab
- Kaggle
|
Download README.md from Codemaster67/Unichem_10M_tokens: direct link, hf CLI and curl.
- Browser
- Download file 3.59 kB
-
https://huggingface.co/Codemaster67/Unichem_10M_tokens/resolve/main/README.md
- Command line
-
hf download hf://Codemaster67/Unichem_10M_tokens/README.md
-
curl -L -o README.md https://huggingface.co/Codemaster67/Unichem_10M_tokens/resolve/main/README.md
3.59 kB
| base_model: allenai/OLMo-1B-hf | |
| datasets: | |
| - Codemaster67/Unichem_10M | |
| language: en | |
| library_name: peft | |
| license: apache-2.0 | |
| tags: | |
| - chemistry | |
| - smiles | |
| - olmo | |
| - causal-lm | |
| - qlora | |
| - peft | |
| - 4bit | |
| # OLMo-7B QLoRA Adapter -- Chemistry SMILES CPT | |
| ## Model Description | |
| This is a **QLoRA (Quantized LoRA)** adapter trained on top of | |
| [allenai/OLMo-1B-hf](https://huggingface.co/allenai/OLMo-1B-hf) for chemistry | |
| SMILES language modelling using the | |
| [Codemaster67/Unichem_10M](https://huggingface.co/datasets/Codemaster67/Unichem_10M) dataset. | |
| The base model was loaded in **4-bit precision** (NF4 quantization via | |
| bitsandbytes with double quantization) and LoRA adapter matrices were | |
| trained on top in bfloat16. | |
| **Decoupled learning rates** are used: LoRA adapters train at `2e-05`, | |
| while `embed_tokens` and `lm_head` train at `2.0000000000000003e-06` (10x smaller) to | |
| avoid catastrophic forgetting of the base vocabulary. | |
| ## QLoRA / Quantization Configuration | |
| | Parameter | Value | | |
| |---|---| | |
| | **Quantization** | NF4 (4-bit) | | |
| | **Double Quantization** | True | | |
| | **Compute dtype** | bfloat16 | | |
| | **Rank (r)** | 64 | | |
| | **Alpha** | 128 | | |
| | **Effective Scaling** | 2.0 | | |
| | **Target Modules** | all-linear | | |
| | **Dropout** | 0.01 | | |
| | **RSLoRA** | False | | |
| | **Modules to Save** | embed_tokens, lm_head | | |
| ## Decoupled Learning Rates | |
| | Parameter Group | Learning Rate | | |
| |---|---| | |
| | **LoRA adapters** | 2e-05 | | |
| | **embed_tokens + lm_head** | 2.0000000000000003e-06 (x0.1) | | |
| ## Training Details | |
| | Parameter | Value | | |
| |---|---| | |
| | **Method** | QLoRA (4-bit base + LoRA adapters) | | |
| | **Epochs** | 1 | | |
| | **Learning Rate (LoRA)** | 2e-05 | | |
| | **Learning Rate (embed/head)** | 2.0000000000000003e-06 | | |
| | **Optimizer** | AdamW 8-bit | | |
| | **Batch Size (per device)** | 32 | | |
| | **Gradient Accumulation** | 1 | | |
| | **Max Sequence Length** | 512 | | |
| | **Warmup Ratio** | 0.1 | | |
| | **Weight Decay** | 0.01 | | |
| | **Effective Batch Size** | 32 | | |
| | **Scheduler** | Cosine | | |
| | **Precision** | bf16 (adapters) / 4-bit NF4 (base) | | |
| | **Gradient Checkpointing** | True | | |
| | **Validation Split** | 5 % | | |
| | **Packed Training Sequences** | 18907 | | |
| | **Packed Validation Sequences** | 995 | | |
| ## Training Results | |
| | Metric | Value | | |
| |---|---| | |
| | **Training Loss** | 1.0676 | | |
| | **Validation Loss** | 0.9724 | | |
| ## Usage | |
| ```python | |
| from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig | |
| from peft import PeftModel | |
| import torch | |
| bnb_config = BitsAndBytesConfig( | |
| load_in_4bit=True, | |
| bnb_4bit_use_double_quant=True, | |
| bnb_4bit_quant_type="nf4", | |
| bnb_4bit_compute_dtype=torch.bfloat16, | |
| ) | |
| base_model = AutoModelForCausalLM.from_pretrained( | |
| "allenai/OLMo-1B-hf", quantization_config=bnb_config, trust_remote_code=True | |
| ) | |
| model = PeftModel.from_pretrained(base_model, "Codemaster67/Unichem_10M_tokens") | |
| tokenizer = AutoTokenizer.from_pretrained("Codemaster67/Unichem_10M_tokens", trust_remote_code=True) | |
| smiles_input = "<|start_of_smiles|>CC(=O)Oc1ccccc1C(=O)O<|end_of_smiles|>" | |
| inputs = tokenizer(smiles_input, return_tensors="pt") | |
| outputs = model.generate(**inputs, max_new_tokens=128) | |
| print(tokenizer.decode(outputs[0], skip_special_tokens=False)) | |
| ``` | |
| ## Intended Use | |
| Chemistry-domain language modelling, SMILES generation and completion, | |
| and downstream molecular property prediction via fine-tuning. | |
| ## Limitations | |
| - QLoRA adapters only; requires the base model [allenai/OLMo-1B-hf](https://huggingface.co/allenai/OLMo-1B-hf) loaded in 4-bit to use. | |
| - Trained primarily on SMILES strings; natural-language instruction-following | |
| ability may degrade compared to the base OLMo checkpoint. | |