Instructions to use Junping0645/vtuber with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use Junping0645/vtuber with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("EleutherAI/polyglot-ko-3.8b") model = PeftModel.from_pretrained(base_model, "Junping0645/vtuber") - Transformers
How to use Junping0645/vtuber with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="Junping0645/vtuber")# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("Junping0645/vtuber", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use Junping0645/vtuber with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Junping0645/vtuber" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Junping0645/vtuber", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/Junping0645/vtuber
- SGLang
How to use Junping0645/vtuber with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Junping0645/vtuber" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Junping0645/vtuber", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Junping0645/vtuber" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Junping0645/vtuber", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use Junping0645/vtuber with Docker Model Runner:
docker model run hf.co/Junping0645/vtuber
File size: 3,267 Bytes
b0b97ab 323253c b0b97ab 323253c 423e873 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 | ---
base_model: EleutherAI/polyglot-ko-3.8b
library_name: peft
pipeline_tag: text-generation
language:
- ko
tags:
- base_model:adapter:EleutherAI/polyglot-ko-3.8b
- lora
- transformers
- korean
- chatbot
---
# ๋ง๋๋ฌด โ ๋
๊ฑฐ ์ด๋ฅด์ ์ฉ AI ๋ํ ์ฑ๋ด (QLoRA ์ด๋ํฐ)
`EleutherAI/polyglot-ko-3.8b`๋ฅผ QLoRA(4bit NF4)๋ก ํ์ธํ๋ํ LoRA ์ด๋ํฐ์
๋๋ค.
๋
๊ฑฐ ์ด๋ฅด์ ๊ณผ ์งง๊ณ ๋ฐ๋ปํ ์กด๋๋ง๋ก ๋ํํ๋ฉฐ, ๋จผ์ ๊ณต๊ฐํ๊ณ ํ์์ ์๋ถ๋ฅผ ์ฑ๊ธฐ๋ ํ๋ฅด์๋๋ก ํ์ตํ์ต๋๋ค.
## ํ์ต ๋ฐ์ดํฐ
- ๋จ๋ฐ ๋ํ 11,266๊ฐ: 16๊ฐ ์นดํ
๊ณ ๋ฆฌ(๊ฑด๊ฐ/์ธ๋ก์/์์ฃผ/๋ช
์ ๋ฑ) ร 8๊ฐ ๊ฐ์ , GPT-4o-mini๋ก ์์ฑ + ์ค์ ์ด๋ฅด์ ์ธํฐ๋ทฐ ์ ์ฌ์์ ์ถ์ถํ ์ฌํฌ๋ฆฌ/๊ตฌ์ด์ฒด ๋ฐํ 1,266๊ฐ(์ฌํฌ๋ฆฌ ์
๋ ฅ ์ดํด ๋ณด๊ฐ์ฉ)
- ๋ฉํฐํด ๋ํ 1,000ํธ(๋ํ๋น ํ๊ท 7.1ํด) โ ํด๋ง๋ค ์ฌ๋ผ์ด๋ฉ ์๋์ฐ๋ก ํ์ฅํด 3,546๊ฐ ํ์ต ์ํ๋ก ๋ณํ
- ์ต์ข
train 14,072 / val 740
## ํ์ต ์ค์
- LoRA: r=8, alpha=16, dropout=0.1, target_modules=["query_key_value", "dense"] (GPT-NeoX ์ดํ
์
์์ฃผ๋ก ์ข๊ฒ ํ๊น โ ๊ณผ์ ํฉ ์ต์ )
- 4bit NF4 ์์ํ(bitsandbytes), MAX_LENGTH=768
- batch_size=2, gradient_accumulation_steps=8 (์ ํจ ๋ฐฐ์น 16), 5 epoch ์บก + EarlyStoppingCallback(patience=3)
- ๊ฒฐ๊ณผ: epoch 3.296์์ early stopping, **eval_loss 1.366**, RTX 3060 Ti(8GB)์์ 141.9๋ถ ์์, peak VRAM 3.4GB
## ์ฌ์ฉ๋ฒ
๋ฒ ์ด์ค ๋ชจ๋ธ(7.6GB)์ 4bit๋ก ๋ก๋ํ ๋ค ์ด ์ด๋ํฐ๋ฅผ ์น์ต๋๋ค.
```python
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
BASE_MODEL = "EleutherAI/polyglot-ko-3.8b"
ADAPTER = "Junping0645/Vtuber" # ์ด ๋ฆฌํฌ
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
bnb = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.float16,
bnb_4bit_use_double_quant=True,
)
base = AutoModelForCausalLM.from_pretrained(BASE_MODEL, quantization_config=bnb, device_map={"": 0})
model = PeftModel.from_pretrained(base, ADAPTER)
model.eval()
prompt = "### ์ด๋ฅด์ : ์์ฆ ๋ง์ด ์ธ๋กญ์ต๋๋ค.\n### ๋ง๋๋ฌด:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=80, do_sample=True, temperature=0.8,
top_p=0.9, repetition_penalty=1.15,
pad_token_id=tokenizer.eos_token_id, eos_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(out[0], skip_special_tokens=True))
```
### ๋ฉํฐํด(๋ํ ์ด๋ ฅ ์ ์ง)
์ด์ ํด๋ค์ ๊ฐ์ ํฌ๋งท์ผ๋ก ์ด์ด๋ถ์ด๋ฉด ๋ฌธ๋งฅ์ ๋ฐ์ํ ๋ต๋ณ์ด ๋์ต๋๋ค(ํ์ต ์ ์ฌ์ฉํ ํฌ๋งท๊ณผ ๋์ผํด์ผ ํจ):
```
### ์ด๋ฅด์ : ์์ฃผ๊ฐ ๋ค์ ์ฃผ์ ๋๋ฌ ์จ๋์.
### ๋ง๋๋ฌด: ์ ๋ง์? ๊ธฐ๋๋์๊ฒ ์ด์! ์ฆ๊ฑฐ์ด ์๊ฐ ๋ณด๋ด์๊ธธ ๋ฐ๋์.
### ์ด๋ฅด์ : ๋ช ์ด์ธ์ง ์ ๋ฌผ์ด๋ณด์
จ๋ค์, ์ด๋ฒ์ ์ด๋ฑํ๊ต ๋ค์ด๊ฐ์.
### ๋ง๋๋ฌด: ๊ทธ๋ ๊ตฐ์! ์์ฆ ์์ด๋ค์ ๋น ๋ฅด์ฃ .
### ์ด๋ฅด์ : ๊ฑ๊ฐ ์ค๋ฉด ๋ญ ํด์ฃผ๋ฉด ์ข์ํ ๊น์?
### ๋ง๋๋ฌด:
```
## ์ฐธ๊ณ
- VRAM: ํ์ต 3.4GB / ์ถ๋ก ์ฝ 3.3GB (RTX 3060 Ti 8GB ๊ธฐ์ค)
- Framework: PEFT 0.19.1, transformers 5.13.0, bitsandbytes 0.49.2, torch 2.11.0+cu128
|