Instructions to use maxspin/medichat with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use maxspin/medichat with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("NousResearch/Llama-2-7b-chat-hf") model = PeftModel.from_pretrained(base_model, "maxspin/medichat") - Notebooks
- Google Colab
- Kaggle
Training procedure
The following bitsandbytes quantization config was used during training:
- load_in_8bit: True
- load_in_4bit: False
- llm_int8_threshold: 6.0
- llm_int8_skip_modules: None
- llm_int8_enable_fp32_cpu_offload: False
- llm_int8_has_fp16_weight: False
- bnb_4bit_quant_type: fp4
- bnb_4bit_use_double_quant: False
- bnb_4bit_compute_dtype: float32
Framework versions
PEFT 0.4.0
Here is the code to load the load along with llama2 chat model:
!pip install -q accelerate==0.21.0 peft==0.4.0 bitsandbytes==0.40.2 transformers==4.31.0 trl==0.4.7
import os, torch, logging from datasets import load_dataset from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, HfArgumentParser, TrainingArguments, pipeline from peft import LoraConfig, PeftModel from trl import SFTTrainer import torch from peft import PeftModel, PeftConfig from transformers import AutoModelForCausalLM, AutoTokenizer
config = PeftConfig.from_pretrained(path_to_directory_containing adapter_config.json) model = AutoModelForCausalLM.from_pretrained("NousResearch/Llama-2-7b-chat-hf", return_dict=True, load_in_8bit=True, device_map='auto') tokenizer = AutoTokenizer.from_pretrained("NousResearch/Llama-2-7b-chat-hf")
model = PeftModel.from_pretrained(model, "/content/llama-2-7b-medichat") batch = tokenizer("My friend has been feeling a little dizzy these days.What could he pissibely be suffering from ?", return_tensors='pt') batch = batch.to('cuda') output_tokens = model.generate(**batch, max_new_tokens=300) ### You can adjust the max_new_tokens output_response = tokenizer.decode(output_tokens[0], skip_special_tokens=True)
- Downloads last month
- 2