ggomarobot commited on
Commit
765dc24
ยท
verified ยท
1 Parent(s): c7568f4

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +26 -8
README.md CHANGED
@@ -7,9 +7,13 @@ base_model:
7
  ---
8
  # KSSDS: Korean Sentence Splitter for Dialogue Systems
9
 
10
- KSSDS๋Š” [lcw99/t5-base-korean-text-summary](https://huggingface.co/lcw99/t5-base-korean-text-summary)์˜ ์ธ์ฝ”๋”๋ฅผ ๊ธฐ๋ฐ˜์œผ๋กœ ํ•˜์—ฌ, AI Hub ๋ฐ์ดํ„ฐ๋ฅผ ์ด์šฉํ•ด fine-tuningํ•œ ๋ชจ๋ธ์ž…๋‹ˆ๋‹ค.
11
- ์ด ๋ชจ๋ธ์€ ํ•œ๊ตญ์–ด ๋Œ€ํ™” ์‹œ์Šคํ…œ ์šฉ ๋ฌธ์žฅ ๋ถ„๋ฆฌ๊ธฐ๋กœ, Whisper์™€ ๊ฐ™์€ STT ๋ชจ๋ธ์ด ์ƒ์„ฑํ•œ ํ•œ๊ตญ์–ด ํ…์ŠคํŠธ๋ฅผ ๋ฌธ์žฅ ๋‹จ์œ„๋กœ ๋ถ„๋ฆฌํ•˜๋Š” ๊ฒƒ์„ ๋ชฉํ‘œ๋กœ ๋งŒ๋“ค์–ด์กŒ์Šต๋‹ˆ๋‹ค.
12
- ์ž์„ธํ•œ ์„ค๋ช…์€ [GitHub repository](https://github.com/ggomarobot/KSSDS)๋ฅผ ์ฐธ๊ณ ํ•ด์ฃผ์„ธ์š”.
 
 
 
 
13
 
14
  ---
15
 
@@ -17,11 +21,12 @@ KSSDS๋Š” [lcw99/t5-base-korean-text-summary](https://huggingface.co/lcw99/t5-bas
17
 
18
  ### 1. PyPI ๋˜๋Š” GitHub ์„ค์น˜๋ฅผ ํ†ตํ•œ ์‚ฌ์šฉ (๊ถŒ์žฅ)
19
  - PyPI ๋˜๋Š” GitHub๋ฅผ ํ†ตํ•ด ์„ค์น˜ํ•˜๋ฉด ๋”์šฑ ๋” ํŽธ๋ฆฌํ•˜๊ฒŒ ์‚ฌ์šฉํ•˜์‹ค ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
20
- - [GitHub repository](https://github.com/ggomarobot/KSSDS)์—์„œ ์„ค์ • ๋ฐ ์‚ฌ์šฉ ๋ฐฉ๋ฒ•์„ ํ™•์ธํ•˜์„ธ์š”.
21
- -
22
  ### 2. Hugging Face Hub ๋„ค์ดํ‹ฐ๋ธŒ ๋ฐฉ์‹ ์‚ฌ์šฉ
23
 
24
- Hugging Face Hub์—์„œ ๋ชจ๋ธ๊ณผ T5 ์ธ์ฝ”๋”๋ฅผ ์ง์ ‘ ๋‹ค์šด๋กœ๋“œํ•˜์—ฌ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์•„๋ž˜๋Š” ๋ฌธ์žฅ ๋ถ„๋ฆฌ๋ฅผ ์ˆ˜ํ–‰ํ•˜๋Š” ์ „์ฒด ํŒŒ์ดํ”„๋ผ์ธ์˜ ์˜ˆ์ œ ์ฝ”๋“œ์ž…๋‹ˆ๋‹ค.
 
25
 
26
  ```python
27
  # ํ•„์š”ํ•œ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ ๋ฐ ๋ชจ๋“ˆ ๋ถˆ๋Ÿฌ์˜ค๊ธฐ
@@ -193,6 +198,19 @@ for idx, sentence in enumerate(split_sentences):
193
 
194
  ```
195
 
196
- ## ์ž์„ธํ•œ ์ •๋ณด
197
 
198
- ์ž์„ธํ•œ ์„ค์ • ๋ฐ ์‚ฌ์šฉ ๋ฐฉ๋ฒ•์€ [GitHub repository](https://github.com/ggomarobot/KSSDS)๋ฅผ ์ฐธ์กฐํ•ด์ฃผ์„ธ์š”.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
7
  ---
8
  # KSSDS: Korean Sentence Splitter for Dialogue Systems
9
 
10
+ KSSDS๋Š” [lcw99/t5-base-korean-text-summary](https://huggingface.co/lcw99/t5-base-korean-text-summary)์˜ ์ธ์ฝ”๋”๋ฅผ ๊ธฐ๋ฐ˜์œผ๋กœ ํ•˜์—ฌ,
11
+ AI Hub ๋ฐ์ดํ„ฐ๋ฅผ ์ด์šฉํ•ด fine-tuningํ•œ ๋ชจ๋ธ์ž…๋‹ˆ๋‹ค.
12
+
13
+ ์ด ๋ชจ๋ธ์€ ํ•œ๊ตญ์–ด ๋Œ€ํ™” ์‹œ์Šคํ…œ ์šฉ ๋ฌธ์žฅ ๋ถ„๋ฆฌ๊ธฐ๋กœ,
14
+ Whisper์™€ ๊ฐ™์€ STT ๋ชจ๋ธ์ด ์ƒ์„ฑํ•œ ํ•œ๊ตญ์–ด ํ…์ŠคํŠธ๋ฅผ ๋ฌธ์žฅ ๋‹จ์œ„๋กœ ๋ถ„๋ฆฌํ•˜๋Š” ๊ฒƒ์„ ๋ชฉํ‘œ๋กœ ๋งŒ๋“ค์–ด์กŒ์Šต๋‹ˆ๋‹ค.
15
+
16
+ ์ž์„ธํ•œ ์„ค๋ช…์€ [KSSDS GitHub repository](https://github.com/ggomarobot/KSSDS)๋ฅผ ์ฐธ๊ณ ํ•ด์ฃผ์„ธ์š”.
17
 
18
  ---
19
 
 
21
 
22
  ### 1. PyPI ๋˜๋Š” GitHub ์„ค์น˜๋ฅผ ํ†ตํ•œ ์‚ฌ์šฉ (๊ถŒ์žฅ)
23
  - PyPI ๋˜๋Š” GitHub๋ฅผ ํ†ตํ•ด ์„ค์น˜ํ•˜๋ฉด ๋”์šฑ ๋” ํŽธ๋ฆฌํ•˜๊ฒŒ ์‚ฌ์šฉํ•˜์‹ค ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
24
+ - [KSSDS GitHub repository](https://github.com/ggomarobot/KSSDS)์—์„œ ์„ค์ • ๋ฐ ์‚ฌ์šฉ ๋ฐฉ๋ฒ•์„ ํ™•์ธํ•˜์„ธ์š”.
25
+
26
  ### 2. Hugging Face Hub ๋„ค์ดํ‹ฐ๋ธŒ ๋ฐฉ์‹ ์‚ฌ์šฉ
27
 
28
+ Hugging Face Hub์—์„œ ๋ชจ๋ธ๊ณผ T5 ์ธ์ฝ”๋”๋ฅผ ์ง์ ‘ ๋‹ค์šด๋กœ๋“œํ•˜์—ฌ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
29
+ ์•„๋ž˜๋Š” ๋ฌธ์žฅ ๋ถ„๋ฆฌ๋ฅผ ์ˆ˜ํ–‰ํ•˜๋Š” ์ „์ฒด ํŒŒ์ดํ”„๋ผ์ธ์˜ ์˜ˆ์ œ ์ฝ”๋“œ์ž…๋‹ˆ๋‹ค.
30
 
31
  ```python
32
  # ํ•„์š”ํ•œ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ ๋ฐ ๋ชจ๋“ˆ ๋ถˆ๋Ÿฌ์˜ค๊ธฐ
 
198
 
199
  ```
200
 
201
+ ## KSSDS_NO_LF ๋ชจ๋ธ ์†Œ๊ฐœ
202
 
203
+ **KSSDS_NO_LF**๋Š” ablation study๋ฅผ ์œ„ํ•ด Length Filter๋ฅผ ์ ์šฉํ•˜์ง€ ์•Š๊ณ  ํ•™์Šต๋œ ๋ชจ๋ธ์ž…๋‹ˆ๋‹ค.
204
+ ์ด ๋ชจ๋ธ์€ ๊ธด ํ…์ŠคํŠธ๋ฅผ ์ฒ˜๋ฆฌํ•  ๋•Œ ์ƒ๋Œ€์ ์œผ๋กœ ๋œ ์„ธ๋ถ€์ ์ธ ๋ฌธ์žฅ ๋ถ„๋ฆฌ๋ฅผ ์ˆ˜ํ–‰ํ•ฉ๋‹ˆ๋‹ค.
205
+
206
+ ์‚ฌ์šฉ ๋ฐฉ์‹์€ ๋ณธ ๋ชจ๋ธ๊ณผ ๋™์ผํ•˜์ง€๋งŒ
207
+ ์—ฐ๊ตฌ ๋ฐ ๋น„๊ต ๋ชฉ์  ์™ธ์—๋Š” ์‹ค์ œ ์‚ฌ์šฉ์— ์ ํ•ฉํ•˜์ง€ ์•Š์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
208
+
209
+ ### ์‚ฌ์šฉ ๋ฐฉ๋ฒ•
210
+ ์œ„์˜ [Hugging Face Hub ๋„ค์ดํ‹ฐ๋ธŒ ๋ฐฉ์‹ ์‚ฌ์šฉ](#2-hugging-face-hub-๋„ค์ดํ‹ฐ๋ธŒ-๋ฐฉ์‹-์‚ฌ์šฉ) ์ฝ”๋“œ ์Šค๋‹ˆํŽซ์—์„œ `model_name`์„ `"ggomarobot/KSSDS_NO_LF"`๋กœ ๊ต์ฒดํ•˜์—ฌ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค:
211
+ ```python
212
+ # Load tokenizer and model
213
+ model_name = "ggomarobot/KSSDS_NO_LF"
214
+ tokenizer = AutoTokenizer.from_pretrained(model_name)
215
+ model = T5ForTokenClassification.from_pretrained(model_name)
216
+ ```