Update README.md
Browse files
README.md
CHANGED
|
@@ -7,9 +7,13 @@ base_model:
|
|
| 7 |
---
|
| 8 |
# KSSDS: Korean Sentence Splitter for Dialogue Systems
|
| 9 |
|
| 10 |
-
KSSDS๋ [lcw99/t5-base-korean-text-summary](https://huggingface.co/lcw99/t5-base-korean-text-summary)์ ์ธ์ฝ๋๋ฅผ ๊ธฐ๋ฐ์ผ๋ก ํ์ฌ,
|
| 11 |
-
|
| 12 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
|
| 14 |
---
|
| 15 |
|
|
@@ -17,11 +21,12 @@ KSSDS๋ [lcw99/t5-base-korean-text-summary](https://huggingface.co/lcw99/t5-bas
|
|
| 17 |
|
| 18 |
### 1. PyPI ๋๋ GitHub ์ค์น๋ฅผ ํตํ ์ฌ์ฉ (๊ถ์ฅ)
|
| 19 |
- PyPI ๋๋ GitHub๋ฅผ ํตํด ์ค์นํ๋ฉด ๋์ฑ ๋ ํธ๋ฆฌํ๊ฒ ์ฌ์ฉํ์ค ์ ์์ต๋๋ค.
|
| 20 |
-
- [GitHub repository](https://github.com/ggomarobot/KSSDS)์์ ์ค์ ๋ฐ ์ฌ์ฉ ๋ฐฉ๋ฒ์ ํ์ธํ์ธ์.
|
| 21 |
-
|
| 22 |
### 2. Hugging Face Hub ๋ค์ดํฐ๋ธ ๋ฐฉ์ ์ฌ์ฉ
|
| 23 |
|
| 24 |
-
Hugging Face Hub์์ ๋ชจ๋ธ๊ณผ T5 ์ธ์ฝ๋๋ฅผ ์ง์ ๋ค์ด๋ก๋ํ์ฌ ์ฌ์ฉํ ์ ์์ต๋๋ค.
|
|
|
|
| 25 |
|
| 26 |
```python
|
| 27 |
# ํ์ํ ๋ผ์ด๋ธ๋ฌ๋ฆฌ ๋ฐ ๋ชจ๋ ๋ถ๋ฌ์ค๊ธฐ
|
|
@@ -193,6 +198,19 @@ for idx, sentence in enumerate(split_sentences):
|
|
| 193 |
|
| 194 |
```
|
| 195 |
|
| 196 |
-
##
|
| 197 |
|
| 198 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 7 |
---
|
| 8 |
# KSSDS: Korean Sentence Splitter for Dialogue Systems
|
| 9 |
|
| 10 |
+
KSSDS๋ [lcw99/t5-base-korean-text-summary](https://huggingface.co/lcw99/t5-base-korean-text-summary)์ ์ธ์ฝ๋๋ฅผ ๊ธฐ๋ฐ์ผ๋ก ํ์ฌ,
|
| 11 |
+
AI Hub ๋ฐ์ดํฐ๋ฅผ ์ด์ฉํด fine-tuningํ ๋ชจ๋ธ์
๋๋ค.
|
| 12 |
+
|
| 13 |
+
์ด ๋ชจ๋ธ์ ํ๊ตญ์ด ๋ํ ์์คํ
์ฉ ๋ฌธ์ฅ ๋ถ๋ฆฌ๊ธฐ๋ก,
|
| 14 |
+
Whisper์ ๊ฐ์ STT ๋ชจ๋ธ์ด ์์ฑํ ํ๊ตญ์ด ํ
์คํธ๋ฅผ ๋ฌธ์ฅ ๋จ์๋ก ๋ถ๋ฆฌํ๋ ๊ฒ์ ๋ชฉํ๋ก ๋ง๋ค์ด์ก์ต๋๋ค.
|
| 15 |
+
|
| 16 |
+
์์ธํ ์ค๋ช
์ [KSSDS GitHub repository](https://github.com/ggomarobot/KSSDS)๋ฅผ ์ฐธ๊ณ ํด์ฃผ์ธ์.
|
| 17 |
|
| 18 |
---
|
| 19 |
|
|
|
|
| 21 |
|
| 22 |
### 1. PyPI ๋๋ GitHub ์ค์น๋ฅผ ํตํ ์ฌ์ฉ (๊ถ์ฅ)
|
| 23 |
- PyPI ๋๋ GitHub๋ฅผ ํตํด ์ค์นํ๋ฉด ๋์ฑ ๋ ํธ๋ฆฌํ๊ฒ ์ฌ์ฉํ์ค ์ ์์ต๋๋ค.
|
| 24 |
+
- [KSSDS GitHub repository](https://github.com/ggomarobot/KSSDS)์์ ์ค์ ๋ฐ ์ฌ์ฉ ๋ฐฉ๋ฒ์ ํ์ธํ์ธ์.
|
| 25 |
+
|
| 26 |
### 2. Hugging Face Hub ๋ค์ดํฐ๋ธ ๋ฐฉ์ ์ฌ์ฉ
|
| 27 |
|
| 28 |
+
Hugging Face Hub์์ ๋ชจ๋ธ๊ณผ T5 ์ธ์ฝ๋๋ฅผ ์ง์ ๋ค์ด๋ก๋ํ์ฌ ์ฌ์ฉํ ์ ์์ต๋๋ค.
|
| 29 |
+
์๋๋ ๋ฌธ์ฅ ๋ถ๋ฆฌ๋ฅผ ์ํํ๋ ์ ์ฒด ํ์ดํ๋ผ์ธ์ ์์ ์ฝ๋์
๋๋ค.
|
| 30 |
|
| 31 |
```python
|
| 32 |
# ํ์ํ ๋ผ์ด๋ธ๋ฌ๋ฆฌ ๋ฐ ๋ชจ๋ ๋ถ๋ฌ์ค๊ธฐ
|
|
|
|
| 198 |
|
| 199 |
```
|
| 200 |
|
| 201 |
+
## KSSDS_NO_LF ๋ชจ๋ธ ์๊ฐ
|
| 202 |
|
| 203 |
+
**KSSDS_NO_LF**๋ ablation study๋ฅผ ์ํด Length Filter๋ฅผ ์ ์ฉํ์ง ์๊ณ ํ์ต๋ ๋ชจ๋ธ์
๋๋ค.
|
| 204 |
+
์ด ๋ชจ๋ธ์ ๊ธด ํ
์คํธ๋ฅผ ์ฒ๋ฆฌํ ๋ ์๋์ ์ผ๋ก ๋ ์ธ๋ถ์ ์ธ ๋ฌธ์ฅ ๋ถ๋ฆฌ๋ฅผ ์ํํฉ๋๋ค.
|
| 205 |
+
|
| 206 |
+
์ฌ์ฉ ๋ฐฉ์์ ๋ณธ ๋ชจ๋ธ๊ณผ ๋์ผํ์ง๋ง
|
| 207 |
+
์ฐ๊ตฌ ๋ฐ ๋น๊ต ๋ชฉ์ ์ธ์๋ ์ค์ ์ฌ์ฉ์ ์ ํฉํ์ง ์์ ์ ์์ต๋๋ค.
|
| 208 |
+
|
| 209 |
+
### ์ฌ์ฉ ๋ฐฉ๋ฒ
|
| 210 |
+
์์ [Hugging Face Hub ๋ค์ดํฐ๋ธ ๋ฐฉ์ ์ฌ์ฉ](#2-hugging-face-hub-๋ค์ดํฐ๋ธ-๋ฐฉ์-์ฌ์ฉ) ์ฝ๋ ์ค๋ํซ์์ `model_name`์ `"ggomarobot/KSSDS_NO_LF"`๋ก ๊ต์ฒดํ์ฌ ์ฌ์ฉํ ์ ์์ต๋๋ค:
|
| 211 |
+
```python
|
| 212 |
+
# Load tokenizer and model
|
| 213 |
+
model_name = "ggomarobot/KSSDS_NO_LF"
|
| 214 |
+
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
| 215 |
+
model = T5ForTokenClassification.from_pretrained(model_name)
|
| 216 |
+
```
|