maxspin commited on
Commit
0dd9d90
·
1 Parent(s): 2d05d3b

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +23 -0
README.md CHANGED
@@ -18,3 +18,26 @@ The following `bitsandbytes` quantization config was used during training:
18
 
19
 
20
  - PEFT 0.4.0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
18
 
19
 
20
  - PEFT 0.4.0
21
+
22
+ - Here is the code to load the load along with llama2 chat model:
23
+
24
+ !pip install -q accelerate==0.21.0 peft==0.4.0 bitsandbytes==0.40.2 transformers==4.31.0 trl==0.4.7
25
+
26
+ import os, torch, logging
27
+ from datasets import load_dataset
28
+ from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, HfArgumentParser, TrainingArguments, pipeline
29
+ from peft import LoraConfig, PeftModel
30
+ from trl import SFTTrainer
31
+ import torch
32
+ from peft import PeftModel, PeftConfig
33
+ from transformers import AutoModelForCausalLM, AutoTokenizer
34
+
35
+ config = PeftConfig.from_pretrained(path_to_directory_containing adapter_config.json)
36
+ model = AutoModelForCausalLM.from_pretrained("NousResearch/Llama-2-7b-chat-hf", return_dict=True, load_in_8bit=True, device_map='auto')
37
+ tokenizer = AutoTokenizer.from_pretrained("NousResearch/Llama-2-7b-chat-hf")
38
+
39
+ model = PeftModel.from_pretrained(model, "/content/llama-2-7b-medichat")
40
+ batch = tokenizer("My friend has been feeling a little dizzy these days.What could he pissibely be suffering from ?", return_tensors='pt')
41
+ batch = batch.to('cuda')
42
+ output_tokens = model.generate(**batch, max_new_tokens=300) ### You can adjust the max_new_tokens
43
+ output_response = tokenizer.decode(output_tokens[0], skip_special_tokens=True)