Instructions to use maxspin/medichat with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use maxspin/medichat with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("NousResearch/Llama-2-7b-chat-hf") model = PeftModel.from_pretrained(base_model, "maxspin/medichat") - Notebooks
- Google Colab
- Kaggle
|
Download README.md from maxspin/medichat: direct link, hf CLI and curl.
- Browser
- Download file 1.69 kB
-
https://huggingface.co/maxspin/medichat/resolve/main/README.md
- Command line
-
hf download hf://maxspin/medichat/README.md
-
curl -L -o README.md https://huggingface.co/maxspin/medichat/resolve/main/README.md
1.69 kB
| library_name: peft | |
| ## Training procedure | |
| The following `bitsandbytes` quantization config was used during training: | |
| - load_in_8bit: True | |
| - load_in_4bit: False | |
| - llm_int8_threshold: 6.0 | |
| - llm_int8_skip_modules: None | |
| - llm_int8_enable_fp32_cpu_offload: False | |
| - llm_int8_has_fp16_weight: False | |
| - bnb_4bit_quant_type: fp4 | |
| - bnb_4bit_use_double_quant: False | |
| - bnb_4bit_compute_dtype: float32 | |
| ### Framework versions | |
| - PEFT 0.4.0 | |
| - Here is the code to load the load along with llama2 chat model: | |
| !pip install -q accelerate==0.21.0 peft==0.4.0 bitsandbytes==0.40.2 transformers==4.31.0 trl==0.4.7 | |
| import os, torch, logging | |
| from datasets import load_dataset | |
| from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, HfArgumentParser, TrainingArguments, pipeline | |
| from peft import LoraConfig, PeftModel | |
| from trl import SFTTrainer | |
| import torch | |
| from peft import PeftModel, PeftConfig | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| config = PeftConfig.from_pretrained(path_to_directory_containing adapter_config.json) | |
| model = AutoModelForCausalLM.from_pretrained("NousResearch/Llama-2-7b-chat-hf", return_dict=True, load_in_8bit=True, device_map='auto') | |
| tokenizer = AutoTokenizer.from_pretrained("NousResearch/Llama-2-7b-chat-hf") | |
| model = PeftModel.from_pretrained(model, "/content/llama-2-7b-medichat") | |
| batch = tokenizer("My friend has been feeling a little dizzy these days.What could he pissibely be suffering from ?", return_tensors='pt') | |
| batch = batch.to('cuda') | |
| output_tokens = model.generate(**batch, max_new_tokens=300) ### You can adjust the max_new_tokens | |
| output_response = tokenizer.decode(output_tokens[0], skip_special_tokens=True) |