Instructions to use Junping0645/vtuber with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use Junping0645/vtuber with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("EleutherAI/polyglot-ko-3.8b") model = PeftModel.from_pretrained(base_model, "Junping0645/vtuber") - Transformers
How to use Junping0645/vtuber with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="Junping0645/vtuber")# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("Junping0645/vtuber", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use Junping0645/vtuber with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Junping0645/vtuber" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Junping0645/vtuber", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/Junping0645/vtuber
- SGLang
How to use Junping0645/vtuber with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Junping0645/vtuber" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Junping0645/vtuber", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Junping0645/vtuber" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Junping0645/vtuber", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use Junping0645/vtuber with Docker Model Runner:
docker model run hf.co/Junping0645/vtuber
|
Download README.md from Junping0645/vtuber: direct link, hf CLI and curl.
- Browser
- Download file 3.27 kB
-
https://huggingface.co/Junping0645/vtuber/resolve/main/README.md
- Command line
-
hf download hf://Junping0645/vtuber/README.md
-
curl -L -o README.md https://huggingface.co/Junping0645/vtuber/resolve/main/README.md
3.27 kB
| base_model: EleutherAI/polyglot-ko-3.8b | |
| library_name: peft | |
| pipeline_tag: text-generation | |
| language: | |
| - ko | |
| tags: | |
| - base_model:adapter:EleutherAI/polyglot-ko-3.8b | |
| - lora | |
| - transformers | |
| - korean | |
| - chatbot | |
| # ๋ง๋๋ฌด โ ๋ ๊ฑฐ ์ด๋ฅด์ ์ฉ AI ๋ํ ์ฑ๋ด (QLoRA ์ด๋ํฐ) | |
| `EleutherAI/polyglot-ko-3.8b`๋ฅผ QLoRA(4bit NF4)๋ก ํ์ธํ๋ํ LoRA ์ด๋ํฐ์ ๋๋ค. | |
| ๋ ๊ฑฐ ์ด๋ฅด์ ๊ณผ ์งง๊ณ ๋ฐ๋ปํ ์กด๋๋ง๋ก ๋ํํ๋ฉฐ, ๋จผ์ ๊ณต๊ฐํ๊ณ ํ์์ ์๋ถ๋ฅผ ์ฑ๊ธฐ๋ ํ๋ฅด์๋๋ก ํ์ตํ์ต๋๋ค. | |
| ## ํ์ต ๋ฐ์ดํฐ | |
| - ๋จ๋ฐ ๋ํ 11,266๊ฐ: 16๊ฐ ์นดํ ๊ณ ๋ฆฌ(๊ฑด๊ฐ/์ธ๋ก์/์์ฃผ/๋ช ์ ๋ฑ) ร 8๊ฐ ๊ฐ์ , GPT-4o-mini๋ก ์์ฑ + ์ค์ ์ด๋ฅด์ ์ธํฐ๋ทฐ ์ ์ฌ์์ ์ถ์ถํ ์ฌํฌ๋ฆฌ/๊ตฌ์ด์ฒด ๋ฐํ 1,266๊ฐ(์ฌํฌ๋ฆฌ ์ ๋ ฅ ์ดํด ๋ณด๊ฐ์ฉ) | |
| - ๋ฉํฐํด ๋ํ 1,000ํธ(๋ํ๋น ํ๊ท 7.1ํด) โ ํด๋ง๋ค ์ฌ๋ผ์ด๋ฉ ์๋์ฐ๋ก ํ์ฅํด 3,546๊ฐ ํ์ต ์ํ๋ก ๋ณํ | |
| - ์ต์ข train 14,072 / val 740 | |
| ## ํ์ต ์ค์ | |
| - LoRA: r=8, alpha=16, dropout=0.1, target_modules=["query_key_value", "dense"] (GPT-NeoX ์ดํ ์ ์์ฃผ๋ก ์ข๊ฒ ํ๊น โ ๊ณผ์ ํฉ ์ต์ ) | |
| - 4bit NF4 ์์ํ(bitsandbytes), MAX_LENGTH=768 | |
| - batch_size=2, gradient_accumulation_steps=8 (์ ํจ ๋ฐฐ์น 16), 5 epoch ์บก + EarlyStoppingCallback(patience=3) | |
| - ๊ฒฐ๊ณผ: epoch 3.296์์ early stopping, **eval_loss 1.366**, RTX 3060 Ti(8GB)์์ 141.9๋ถ ์์, peak VRAM 3.4GB | |
| ## ์ฌ์ฉ๋ฒ | |
| ๋ฒ ์ด์ค ๋ชจ๋ธ(7.6GB)์ 4bit๋ก ๋ก๋ํ ๋ค ์ด ์ด๋ํฐ๋ฅผ ์น์ต๋๋ค. | |
| ```python | |
| import torch | |
| from peft import PeftModel | |
| from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig | |
| BASE_MODEL = "EleutherAI/polyglot-ko-3.8b" | |
| ADAPTER = "Junping0645/Vtuber" # ์ด ๋ฆฌํฌ | |
| tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL) | |
| bnb = BitsAndBytesConfig( | |
| load_in_4bit=True, | |
| bnb_4bit_quant_type="nf4", | |
| bnb_4bit_compute_dtype=torch.float16, | |
| bnb_4bit_use_double_quant=True, | |
| ) | |
| base = AutoModelForCausalLM.from_pretrained(BASE_MODEL, quantization_config=bnb, device_map={"": 0}) | |
| model = PeftModel.from_pretrained(base, ADAPTER) | |
| model.eval() | |
| prompt = "### ์ด๋ฅด์ : ์์ฆ ๋ง์ด ์ธ๋กญ์ต๋๋ค.\n### ๋ง๋๋ฌด:" | |
| inputs = tokenizer(prompt, return_tensors="pt").to(model.device) | |
| out = model.generate(**inputs, max_new_tokens=80, do_sample=True, temperature=0.8, | |
| top_p=0.9, repetition_penalty=1.15, | |
| pad_token_id=tokenizer.eos_token_id, eos_token_id=tokenizer.eos_token_id) | |
| print(tokenizer.decode(out[0], skip_special_tokens=True)) | |
| ``` | |
| ### ๋ฉํฐํด(๋ํ ์ด๋ ฅ ์ ์ง) | |
| ์ด์ ํด๋ค์ ๊ฐ์ ํฌ๋งท์ผ๋ก ์ด์ด๋ถ์ด๋ฉด ๋ฌธ๋งฅ์ ๋ฐ์ํ ๋ต๋ณ์ด ๋์ต๋๋ค(ํ์ต ์ ์ฌ์ฉํ ํฌ๋งท๊ณผ ๋์ผํด์ผ ํจ): | |
| ``` | |
| ### ์ด๋ฅด์ : ์์ฃผ๊ฐ ๋ค์ ์ฃผ์ ๋๋ฌ ์จ๋์. | |
| ### ๋ง๋๋ฌด: ์ ๋ง์? ๊ธฐ๋๋์๊ฒ ์ด์! ์ฆ๊ฑฐ์ด ์๊ฐ ๋ณด๋ด์๊ธธ ๋ฐ๋์. | |
| ### ์ด๋ฅด์ : ๋ช ์ด์ธ์ง ์ ๋ฌผ์ด๋ณด์ จ๋ค์, ์ด๋ฒ์ ์ด๋ฑํ๊ต ๋ค์ด๊ฐ์. | |
| ### ๋ง๋๋ฌด: ๊ทธ๋ ๊ตฐ์! ์์ฆ ์์ด๋ค์ ๋น ๋ฅด์ฃ . | |
| ### ์ด๋ฅด์ : ๊ฑ๊ฐ ์ค๋ฉด ๋ญ ํด์ฃผ๋ฉด ์ข์ํ ๊น์? | |
| ### ๋ง๋๋ฌด: | |
| ``` | |
| ## ์ฐธ๊ณ | |
| - VRAM: ํ์ต 3.4GB / ์ถ๋ก ์ฝ 3.3GB (RTX 3060 Ti 8GB ๊ธฐ์ค) | |
| - Framework: PEFT 0.19.1, transformers 5.13.0, bitsandbytes 0.49.2, torch 2.11.0+cu128 | |