docuflow / README.md
woongmin's picture
[Delta-02] HF Spaces README frontmatter μΆ”κ°€
1dd2838
|
Raw History Blame Contribute Delete
11.5 kB
---
title: DocuFlow
emoji: πŸ“„
colorFrom: blue
colorTo: green
sdk: docker
app_port: 7860
---
<br>
<br>
> Pi Lab(파이랩)의 λͺ¨λ“  κ³Όμ œλŠ” μ‹€μ œ κΈ°μ—…μ—μ„œ μ£Όμ–΄μ§€λŠ” κ³Όμ œμ— κΈ°λ°˜ν•˜μ—¬ μ œμž‘λ˜μ—ˆμœΌλ©°, μ €μž‘κΆŒλ²•μ˜ 보호λ₯Ό λ°›μŠ΅λ‹ˆλ‹€.
<br>
<br>
# Project DocuFlow: Intelligent Document Agent
<br>
<p align="center">
<img src="/assets/docuflow.jpg" width="100%">
</p>
<br>
**🧠 The Future of Document Processing**
_πŸ“ HQ: 42 Silicon Valley Blvd, Palo Alto, CA_
<br>
> "Read less, Understand more."
<br>
**DocuFlow AI**λŠ” μ „ μ„Έκ³„μ˜ λΉ„μ •ν˜• λ¬Έμ„œ 데이터λ₯Ό μ§€λŠ₯ν˜• 지식 베이슀둜 μ „ν™˜ν•˜λŠ” μ‹€λ¦¬μ½˜λ°Έλ¦¬ 기반의 AI μŠ€νƒ€νŠΈμ—…μž…λ‹ˆλ‹€. μš°λ¦¬λŠ” λ‹¨μˆœνžˆ λ¬Έμ„œλ₯Ό μ €μž₯ν•˜λŠ” 것을 λ„˜μ–΄, λ¬Έμ„œμ™€ λŒ€ν™”ν•˜κ³  μˆ¨κ²¨μ§„ 톡찰λ ₯을 μ¦‰μ‹œ μ°Ύμ•„λ‚΄λŠ” κ²½ν—˜μ„ μ œκ³΅ν•©λ‹ˆλ‹€.
그리고 였늘, λ‹Ήμ‹  **Alex**λŠ” AI μ—”μ§€λ‹ˆμ–΄λ‘œ νŒ€μ— ν•©λ₯˜ν–ˆμŠ΅λ‹ˆλ‹€.
<br>
<br>
## πŸ’¬ [Slack] `#docuflow-engineering`
### πŸ§‘β€πŸ’Ό Sarah (PM)
> κ³ κ°μ‚¬λ“€λ‘œλΆ€ν„° ν”Όλ“œλ°±μ΄ μ™”λŠ”λ°, κΈ°μ‘΄ 챗봇이 λ¬Έμ„œμ— μ—†λŠ” λ‚΄μš©μ„ μ§€μ–΄λ‚΄μ„œ λ‹΅λ³€ν•œλ‹€κ³  ν•΄μš”(Hallucination).
> 이번 MVP의 핡심은 **μ •ν™•μ„±**μž…λ‹ˆλ‹€. μ‚¬μš©μžκ°€ μ—…λ‘œλ“œν•œ PDF λ¬Έμ„œ λ‚΄μ—μ„œλ§Œ κ·Όκ±°λ₯Ό μ°Ύμ•„ λ‹΅λ³€ν•˜λŠ” **κ³ μ„±λŠ₯ RAG(검색 증강 생성) νŒŒμ΄ν”„λΌμΈ**이 λ°˜λ“œμ‹œ ν•„μš”ν•΄μš”.
<br>
### πŸ‘¨β€πŸ’» Dave (CTO)
> Alex, 이번 과제의 핡심은 LLM을 λ‹¨μˆœνžˆ ν˜ΈμΆœν•˜λŠ” 게 μ•„λ‹ˆλΌ, **'μ–΄λ–»κ²Œ λ¬Έμ„œλ₯Ό κ²€μƒ‰ν•˜λŠλƒ'** 에 λ‹¬λ €μžˆλ„€.
> λ‹¨μˆœνžˆ APIλ₯Ό μ—°κ²°ν•˜λŠ” μˆ˜μ€€μ„ λ„˜μ–΄, **λŒ€λŸ‰μ˜ λ¬Έμ„œλ₯Ό μ²˜λ¦¬ν•˜λŠ” μ„±λŠ₯**κ³Ό **μ •ν™•ν•œ 정보λ₯Ό μ°Ύμ•„λ‚΄λŠ” ν’ˆμ§ˆ**κΉŒμ§€ κ³ λ―Όν•΄μ•Ό ν•΄.
> 2μ£Ό λ™μ•ˆ 주말 집쀑 μ‹€μŠ΅κ³Ό 평일 μ€€λΉ„λ₯Ό λ°˜λ³΅ν•˜λ©°, 우리 μ„œλΉ„μŠ€μ˜ μ½”μ–΄ 엔진을 μ™„μ„±ν•΄μ£Όκ²Œ.
<br>
<br>
## 🧩 ν”„λ‘œμ νŠΈ λͺ©ν‘œ
**DocuFlow Agent**λŠ” RAG(Retrieval-Augmented Generation) κΈ°μˆ μ„ ν™œμš©ν•˜μ—¬ PDF λ¬Έμ„œμ˜ 지식을 AIμ—κ²Œ μ£Όμž…ν•˜λŠ” μ§€λŠ₯ν˜• 검색 μ‹œμŠ€ν…œμž…λ‹ˆλ‹€.
λ³Έ ν”„λ‘œμ νŠΈλŠ” 2μ£Ό 운영(주말 νŽ˜μ–΄ν”„λ‘œκ·Έλž˜λ° + 평일 μ€€λΉ„/쑰사)을 톡해, 빈 껍데기뿐인 μ„œλ²„λ₯Ό **μƒμš© μˆ˜μ€€μ˜ RAG μ—”μ§„**으둜 λ°œμ „μ‹œν‚€λŠ” 것을 λͺ©ν‘œλ‘œ ν•©λ‹ˆλ‹€.
<br>
<br>
## βš™οΈ ν™˜κ²½ μ„€μ •
μ•„λž˜ 1~6단계λ₯Ό μˆœμ„œλŒ€λ‘œ μ™„λ£Œν•΄μ•Ό ν”„λ‘œμ νŠΈλ₯Ό μ‹€ν–‰ν•  수 μžˆμŠ΅λ‹ˆλ‹€.
### 1. pyenv μ„€μΉ˜ (졜초 1회, macOS)
pyenvκ°€ 이미 μ„€μΉ˜λ˜μ–΄ μžˆλ‹€λ©΄ 이 단계λ₯Ό κ±΄λ„ˆλ›°μ„Έμš”.
```bash
# Homebrew둜 pyenv μ„€μΉ˜
brew install pyenv
# μ…Έ 섀정에 pyenv μ΄ˆκΈ°ν™” μΆ”κ°€ (zsh κΈ°μ€€)
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc
echo 'eval "$(pyenv init -)"' >> ~/.zshrc
# μ„€μ • 반영
source ~/.zshrc
# μ„€μΉ˜ 확인 (버전 λ²ˆν˜Έκ°€ 좜λ ₯되면 정상)
pyenv --version
```
---
### 2. Python 3.11 및 μ˜μ‘΄μ„± μ„€μΉ˜
```bash
# Python 3.11 μ„€μΉ˜ 및 μ„€μ •
pyenv install 3.11
pyenv local 3.11
# μ„€μΉ˜ 확인 (3.11.x κ°€ 좜λ ₯되면 정상)
python --version
# pipenv μ„€μΉ˜ (졜초 1회)
pip install pipenv
# ν”„λ‘œμ νŠΈ μ˜μ‘΄μ„± μ„€μΉ˜
pipenv install
```
---
### 3. Ollama μ„€μΉ˜ 및 λͺ¨λΈ λ‹€μš΄λ‘œλ“œ
이 ν”„λ‘œμ νŠΈλŠ” 기본적으둜 둜컬 Ollama μ„œλ²„λ₯Ό μ‚¬μš©ν•©λ‹ˆλ‹€.
**macOS μ„€μΉ˜:**
```bash
# Homebrew둜 μ„€μΉ˜
brew install --cask ollama
# λ˜λŠ” 곡식 μ›Ήμ‚¬μ΄νŠΈμ—μ„œ λ‹€μš΄λ‘œλ“œ
# https://ollama.com/download
```
**μ„œλ²„ μ‹€ν–‰:**
μ„€μΉ˜ ν›„ **Ollama 앱을 μ‹€ν–‰**ν•˜λ©΄ 메뉴 바에 μ•„μ΄μ½˜μ΄ ν‘œμ‹œλ˜κ³  μ„œλ²„κ°€ μžλ™μœΌλ‘œ μ‹œμž‘λ©λ‹ˆλ‹€.
> μ•± λ°©μ‹μœΌλ‘œ μ„€μΉ˜ν•œ 경우 `ollama serve`λ₯Ό λ³„λ„λ‘œ μ‹€ν–‰ν•˜μ§€ μ•Šμ•„λ„ λ©λ‹ˆλ‹€. 앱이 μ„œλ²„λ₯Ό μžλ™μœΌλ‘œ κ΄€λ¦¬ν•©λ‹ˆλ‹€.
**λͺ¨λΈ λ‹€μš΄λ‘œλ“œ (μ„œλ²„ μ‹€ν–‰ ν›„ μ‹€ν–‰):**
```bash
# μž„λ² λ”© λͺ¨λΈ (μ•½ 274MB)
ollama pull nomic-embed-text
# μ±„νŒ… λͺ¨λΈ (μ•½ 4.9GB, λ„€νŠΈμ›Œν¬ ν™˜κ²½μ— 따라 10λΆ„ 이상 μ†Œμš”)
ollama pull llama3.1
```
> ⚠️ `llama3.1` λ‹€μš΄λ‘œλ“œλŠ” μ‹œκ°„μ΄ 많이 μ†Œμš”λ©λ‹ˆλ‹€.
> 주말 μ„Έμ…˜ 당일이 μ•„λ‹Œ **사전에 λ°˜λ“œμ‹œ μ™„λ£Œ**ν•΄λ‘μ„Έμš”.
**μ„œλ²„ μƒνƒœ 확인:**
```bash
curl http://localhost:11434/api/tags
```
λͺ¨λΈ λͺ©λ‘μ΄ JSON ν˜•νƒœλ‘œ 좜λ ₯되면 μ •μƒμž…λ‹ˆλ‹€.
> Ollama μ„œλ²„λŠ” **ν”„λ‘œμ νŠΈ μ‹€ν–‰ μ€‘μ—λŠ” 계속 μΌœλ‘” μƒνƒœλ₯Ό μœ μ§€**ν•΄μ•Ό ν•©λ‹ˆλ‹€. μ„œλ²„κ°€ κΊΌμ§„ μƒνƒœμ—μ„œλŠ” μž„λ² λ”©/μ±„νŒ… μš”μ²­μ΄ μ‹€νŒ¨ν•©λ‹ˆλ‹€.
**μ„œλ²„ μ’…λ£Œ (μ‹€μŠ΅ μ’…λ£Œ ν›„):**
- **메뉴 λ°”** μƒλ‹¨μ˜ Ollama μ•„μ΄μ½˜ 클릭 β†’ **Quit Ollama**
- λ˜λŠ” ν„°λ―Έλ„μ—μ„œ `killall Ollama`
---
### 4. Supabase ν”„λ‘œμ νŠΈ 생성 및 ν™˜κ²½ λ³€μˆ˜ μ„€μ • (.env)
**Supabase ν”„λ‘œμ νŠΈ 생성:**
1. [supabase.com](https://supabase.com)에 μ ‘μ†ν•˜μ—¬ 계정을 λ§Œλ“€κ³  μƒˆ ν”„λ‘œμ νŠΈλ₯Ό μƒμ„±ν•˜μ„Έμš”.
2. ν”„λ‘œμ νŠΈ 생성 ν›„ μ•„λž˜ κ²½λ‘œμ—μ„œ 두 값을 볡사해 λ‘‘λ‹ˆλ‹€.
- **Settings β†’ General β†’ Project ID** 확인 β†’ `https://<project-id>.supabase.co` ν˜•νƒœλ‘œ `SUPABASE_URL`에 μž…λ ₯
- **Settings β†’ API Keys β†’ Legacy API keys β†’ `anon` ν‚€** β†’ `SUPABASE_KEY`에 μž…λ ₯
**ν™˜κ²½ λ³€μˆ˜ μ„€μ •:**
```bash
cp .env.example .env
# .env νŒŒμΌμ„ μ—΄μ–΄ λ³΅μ‚¬ν•œ 값을 μ±„μš°μ„Έμš”
```
```ini
# κΈ°λ³Έ: 둜컬 Ollama μ‚¬μš©
PROVIDER="ollama"
OLLAMA_BASE="http://localhost:11434"
OLLAMA_EMBED_MODEL="nomic-embed-text"
OLLAMA_CHAT_MODEL="llama3.1"
# 곡톡
SUPABASE_URL="https://your-project-id.supabase.co" # Settings β†’ General β†’ Project ID
SUPABASE_KEY="eyJ..." # Settings β†’ API Keys β†’ Legacy API keys β†’ anon
# 선택: OpenAI μ‚¬μš© μ‹œ
# PROVIDER="openai"
# OPENAI_API_KEY="sk-..."
# OPENAI_EMBEDDING_MODEL="text-embedding-3-small"
# OPENAI_CHAT_MODEL="gpt-4o-mini"
```
---
### 5. λ°μ΄ν„°λ² μ΄μŠ€(Supabase) ꡬ성
Supabase λŒ€μ‹œλ³΄λ“œμ˜ **SQL Editor**μ—μ„œ μ•„λž˜ 쿼리λ₯Ό μ‹€ν–‰ν•˜μ„Έμš”.
<details>
<summary><b>πŸ“œ SQL 쿼리 (κΈ°λ³Έ: Ollama 768차원)</b></summary>
<br>
```sql
-- 1. 벑터 ν™•μž₯ ν™œμ„±ν™”
CREATE EXTENSION IF NOT EXISTS vector;
-- 2. λ¬Έμ„œ 청크 ν…Œμ΄λΈ” 생성
CREATE TABLE IF NOT EXISTS document_chunks (
id BIGSERIAL PRIMARY KEY,
document_id TEXT NOT NULL,
content TEXT NOT NULL,
embedding vector(768), -- nomic-embed-text 차원
chunk_index INTEGER NOT NULL,
metadata JSONB DEFAULT '{}',
created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);
-- 3. 검색 ν•¨μˆ˜(RPC) 생성
CREATE OR REPLACE FUNCTION match_documents (
query_embedding vector(768),
match_count int,
p_document_id text
) RETURNS TABLE (
id BIGINT,
document_id TEXT,
content TEXT,
similarity FLOAT
)
LANGUAGE plpgsql
AS $$
BEGIN
RETURN QUERY
SELECT
dc.id,
dc.document_id,
dc.content,
1 - (dc.embedding <=> query_embedding) as similarity
FROM document_chunks AS dc
WHERE dc.document_id = p_document_id
ORDER BY dc.embedding <=> query_embedding
LIMIT match_count;
END;
$$;
-- 4. 인덱슀 생성 (HNSW 방식)
CREATE INDEX ON document_chunks USING hnsw (embedding vector_cosine_ops);
```
</details>
<details>
<summary><b>πŸ“œ λŒ€μ•ˆ: OpenAI 1536차원 μŠ€ν‚€λ§ˆ</b></summary>
<br>
```sql
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE IF NOT EXISTS document_chunks (
id BIGSERIAL PRIMARY KEY,
document_id TEXT NOT NULL,
content TEXT NOT NULL,
embedding vector(1536), -- text-embedding-3-small 차원
chunk_index INTEGER NOT NULL,
metadata JSONB DEFAULT '{}',
created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);
CREATE OR REPLACE FUNCTION match_documents (
query_embedding vector(1536),
match_count int,
p_document_id text
) RETURNS TABLE (
id BIGINT,
document_id TEXT,
content TEXT,
similarity FLOAT
)
LANGUAGE plpgsql
AS $$
BEGIN
RETURN QUERY
SELECT
dc.id,
dc.document_id,
dc.content,
1 - (dc.embedding <=> query_embedding) as similarity
FROM document_chunks AS dc
WHERE dc.document_id = p_document_id
ORDER BY dc.embedding <=> query_embedding
LIMIT match_count;
END;
$$;
CREATE INDEX ON document_chunks USING hnsw (embedding vector_cosine_ops);
```
</details>
μ‹€ν–‰ ν›„ Supabase λŒ€μ‹œλ³΄λ“œμ˜ **Table Editor**μ—μ„œ `document_chunks` ν…Œμ΄λΈ”μ΄ μƒμ„±λ˜μ—ˆλŠ”μ§€ ν™•μΈν•˜μ„Έμš”.
---
### 6. μ„œλ²„ μ‹€ν–‰
```bash
pipenv run uvicorn app.main:app --reload
```
μ‹€ν–‰ ν›„ λΈŒλΌμš°μ €μ—μ„œ `http://127.0.0.1:8000` μ ‘μ†μœΌλ‘œ UIλ₯Ό ν™•μΈν•˜μ„Έμš”.
---
### 7. μ½”λ“œ 포맀터 μ„€μ • (선택)
컀밋 μ „ μžλ™μœΌλ‘œ μ½”λ“œ μŠ€νƒ€μΌμ„ μ •λ¦¬ν•˜λŠ” `black` 포맀터λ₯Ό pre-commit hook으둜 μ„€μ •ν•  수 μžˆμŠ΅λ‹ˆλ‹€.
```bash
# dev μ˜μ‘΄μ„± μ„€μΉ˜ (pre-commit, black 포함)
pipenv install --dev
# git hook 등둝 (졜초 1회)
pipenv run pre-commit install
```
등둝 ν›„μ—λŠ” `git commit` μ‹œ `black`이 μžλ™ μ‹€ν–‰λ©λ‹ˆλ‹€. μ½”λ“œ μŠ€νƒ€μΌμ΄ μˆ˜μ •λœ 경우 νŒŒμΌμ„ λ‹€μ‹œ `git add` ν•œ λ’€ μ»€λ°‹ν•˜λ©΄ λ©λ‹ˆλ‹€.
> 이 단계λ₯Ό κ±΄λ„ˆλ›°μ–΄λ„ ν”„λ‘œμ νŠΈ μ‹€ν–‰μ—λŠ” 영ν–₯이 μ—†μŠ΅λ‹ˆλ‹€.
<br>
<br>
## πŸ—“οΈ 2μ£Ό μ§„ν–‰ ꡬ쑰
ν™˜κ²½ 섀정을 μ™„λ£Œν•œ ν›„, μ•„λž˜ μˆœμ„œλŒ€λ‘œ 단계별 λ¬Έμ„œλ₯Ό 따라 μ§„ν–‰ν•˜μ„Έμš”.
- 주말(ν† /일, 9:00-18:00): νŽ˜μ–΄ν”„λ‘œκ·Έλž˜λ°μœΌλ‘œ κ΅¬ν˜„/검증
- 평일: 사전 μ€€λΉ„, 쑰사, μ‹€ν—˜ 기둝, μ˜μ‚¬κ²°μ • 정리
| 단계 | κΈ°κ°„ | λ¬Έμ„œ |
| ------------- | ------------------ | ------------------------------------------------------------ |
| **Prework** | Week 1 μ „ 평일 | [`docs/01-prework/README.md`](./docs/01-prework/README.md) |
| **Weekend-1** | Week 1 주말 | [`docs/02-weekend1/README.md`](./docs/02-weekend1/README.md) |
| **Midweek** | Week 1~2 사이 평일 | [`docs/03-midweek/README.md`](./docs/03-midweek/README.md) |
| **Weekend-2** | Week 2 주말 | [`docs/04-weekend2/README.md`](./docs/04-weekend2/README.md) |
<br>
<br>
## πŸ“š μ°Έκ³  ν•™μŠ΅ 자료
### ν”„λ‘œμ νŠΈ λ‚΄λΆ€ λ¬Έμ„œ
- **[RAG μ‹œμŠ€ν…œ 기술 심측 뢄석](./docs/00-shared/reference.md)**: RAG μ•„ν‚€ν…μ²˜μ™€ 기술 μŠ€νƒ μ „λ°˜. **ν•„λ…ν•˜μ„Έμš”.**
- **[배포 κ°€μ΄λ“œ](./docs/04-weekend2/deployment.md)**: 배포 상세 κ°€μ΄λ“œ(심화/μ°Έκ³ )μž…λ‹ˆλ‹€.
### μ™ΈλΆ€ μ°Έκ³  자료
- **[What is RAG? (IBM)](https://www.ibm.com/topics/retrieval-augmented-generation)**: RAG κ°œλ…μ„ κ·Έλ¦Όκ³Ό ν•¨κ»˜ μ„€λͺ…ν•˜λŠ” μž…λ¬Έ μžλ£Œμž…λ‹ˆλ‹€.
- **[RAG from Scratch (LangChain YouTube)](https://www.youtube.com/playlist?list=PLfaIDFEXuae2LXbO1_PKyVJiQ23ZztA0x)**: LangChain 곡식 RAG κ΅¬ν˜„ μ˜μƒ μ‹œλ¦¬μ¦ˆμž…λ‹ˆλ‹€ (μ˜μ–΄).
- **[pgvector GitHub](https://github.com/pgvector/pgvector)**: `<=>` μ—°μ‚°μžμ™€ HNSW 인덱슀 μž‘λ™ 방식을 ν™•μΈν•˜μ„Έμš”.
- **[Supabase Vector κ°€μ΄λ“œ](https://supabase.com/docs/guides/ai)**: PostgreSQL을 벑터 λ°μ΄ν„°λ² μ΄μŠ€λ‘œ μ‚¬μš©ν•˜λŠ” λ°©λ²•μž…λ‹ˆλ‹€.
- **[LangChain RecursiveCharacterTextSplitter](https://reference.langchain.com/python/langchain-text-splitters/character/RecursiveCharacterTextSplitter)**: ν…μŠ€νŠΈ μ²­ν‚Ή μ „λž΅μ— λŒ€ν•œ Python 곡식 λ ˆνΌλŸ°μŠ€μž…λ‹ˆλ‹€.
- **[OpenAI Embeddings API](https://platform.openai.com/docs/guides/embeddings)**: ν…μŠ€νŠΈλ₯Ό 숫자둜 λ³€ν™˜ν•˜λŠ” 원리λ₯Ό 이해해야 ν•©λ‹ˆλ‹€.
- **[Vector Similarity Explained (Pinecone)](https://www.pinecone.io/learn/vector-similarity/)**: μœ μ‚¬λ„ μΈ‘μ • 방식 비ꡐ μžλ£Œμž…λ‹ˆλ‹€.