Small-LLM1 / README.md
aaaaaaaieuyd's picture
Upload 7 files
16dce9d verified
|
Raw History Blame Contribute Delete
2.51 kB
---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
datasets:
- roneneldan/TinyStories
tags:
- tiny
- toy-model
- from-scratch
- llama
---
# Small LLM1
**์žฌ๋ฏธ๋กœ ๋งŒ๋“  ์•ฝ 5M(4.98M) ํŒŒ๋ผ๋ฏธํ„ฐ์งœ๋ฆฌ ์ดˆ์†Œํ˜• ์–ธ์–ด ๋ชจ๋ธ์ž…๋‹ˆ๋‹ค.**
> โš ๏ธ **์žฌ๋ฏธ๋กœ ๋งŒ๋“  ๋ชจ๋ธ์ด๋ผ ์„ฑ๋Šฅ์€ ๋‚ฎ์Šต๋‹ˆ๋‹ค.** ์‹ค์ œ ์šฉ๋„๋กœ ์“ฐ๊ธฐ์—๋Š” ๋ถ€์กฑํ•˜๊ณ , ์งง์€ ์˜์–ด ๋™ํ™” ๋ฌธ์žฅ์„ ์ด์–ด ์“ฐ๋Š” ์ •๋„๋งŒ ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
>
> โš ๏ธ **ํŒŒ์ธํŠœ๋‹(instruction tuning / chat tuning)์€ ํ•˜์ง€ ์•Š์•˜์Šต๋‹ˆ๋‹ค.** ์‚ฌ์ „ํ•™์Šต๋งŒ ํ•œ base ๋ชจ๋ธ์ด๋ผ ์งˆ๋ฌธ์— ๋‹ตํ•˜๊ฑฐ๋‚˜ ๋Œ€ํ™”ํ•˜์ง€ ๋ชปํ•˜๊ณ , ์ฃผ์–ด์ง„ ๋ฌธ์žฅ์„ ์ด์–ด์„œ ์“ฐ๊ธฐ๋งŒ ํ•ฉ๋‹ˆ๋‹ค.
## ๋ชจ๋ธ ์ •๋ณด
| ํ•ญ๋ชฉ | ๊ฐ’ |
|---|---|
| ๊ตฌ์กฐ | Llama ์Šคํƒ€์ผ decoder-only Transformer (์ฒ˜์Œ๋ถ€ํ„ฐ ํ•™์Šต) |
| ํŒŒ๋ผ๋ฏธํ„ฐ | 4.98M |
| ๋ ˆ์ด์–ด / hidden / heads | 6 / 256 / 8 |
| FFN ํฌ๊ธฐ | 512 |
| ์–ดํœ˜ ํฌ๊ธฐ | 4,096 (์ง์ ‘ ํ•™์Šตํ•œ byte-level BPE) |
| ์ปจํ…์ŠคํŠธ ๊ธธ์ด | 256 ํ† ํฐ |
| ํ•™์Šต ๋ฐ์ดํ„ฐ | [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) ์•ž๋ถ€๋ถ„ ์•ฝ 40๋งŒ ๊ฐœ ์ด์•ผ๊ธฐ (์•ฝ 92M ํ† ํฐ) |
| ํ•™์Šต | 6,000 step ร— 64 ร— 256 ํ† ํฐ, RTX 5070 1์žฅ์—์„œ ์•ฝ 4๋ถ„ |
| ๊ฒ€์ฆ loss | 1.77 (perplexity ์•ฝ 5.9, TinyStories ๊ฒ€์ฆ ๊ตฌ๊ฐ„) |
## ์‚ฌ์šฉ๋ฒ•
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("aaaaaaaieuyd/Small-LLM1")
tok = AutoTokenizer.from_pretrained("aaaaaaaieuyd/Small-LLM1")
x = tok("Once upon a time", return_tensors="pt")
out = model.generate(**x, max_new_tokens=80, do_sample=True, temperature=0.8, top_k=40)
print(tok.decode(out[0], skip_special_tokens=True))
```
์ถœ๋ ฅ ์˜ˆ์‹œ:
> Once upon a time, there was a little girl named Lily. She had a little tooth that was very important to her. The tooth was her special tooth, and she could always count on it to show it to her friends.
## ํ•œ๊ณ„
- ์˜์–ด ๋™ํ™”์ฒด ๋ฌธ์žฅ๋งŒ ํ•™์Šตํ•ด์„œ ํ•œ๊ตญ์–ด๋‚˜ ๋‹ค๋ฅธ ์ฃผ์ œ๋Š” ๊ฑฐ์˜ ๋ชป ๋‹ค๋ฃน๋‹ˆ๋‹ค.
- ์‚ฌ์‹ค ์ •๋ณด, ์ถ”๋ก , ์ฝ”๋“œ ๊ฐ™์€ ๊ฑด ๊ธฐ๋Œ€ํ•˜๋ฉด ์•ˆ ๋ฉ๋‹ˆ๋‹ค.
- ์•ž๋’ค๊ฐ€ ์•ˆ ๋งž๋Š” ์ด์•ผ๊ธฐ๋ฅผ ์ž์ฃผ ๋งŒ๋“ญ๋‹ˆ๋‹ค.
- ํŒŒ์ธํŠœ๋‹์„ ํ•˜์ง€ ์•Š์•„์„œ ์ง€์‹œ๋ฅผ ๋”ฐ๋ฅด์ง€ ๋ชปํ•ฉ๋‹ˆ๋‹ค.
---
**English:** Small LLM1 is a ~5M-parameter toy language model trained from scratch on TinyStories, just for fun. Performance is low, and it is a base model with **no fine-tuning** (no instruction or chat tuning). It can only continue short, simple English stories.