--- title: DocuFlow emoji: πŸ“„ colorFrom: blue colorTo: green sdk: docker app_port: 7860 ---

> Pi Lab(파이랩)의 λͺ¨λ“  κ³Όμ œλŠ” μ‹€μ œ κΈ°μ—…μ—μ„œ μ£Όμ–΄μ§€λŠ” κ³Όμ œμ— κΈ°λ°˜ν•˜μ—¬ μ œμž‘λ˜μ—ˆμœΌλ©°, μ €μž‘κΆŒλ²•μ˜ 보호λ₯Ό λ°›μŠ΅λ‹ˆλ‹€.

# Project DocuFlow: Intelligent Document Agent


**🧠 The Future of Document Processing** _πŸ“ HQ: 42 Silicon Valley Blvd, Palo Alto, CA_
> "Read less, Understand more."
**DocuFlow AI**λŠ” μ „ μ„Έκ³„μ˜ λΉ„μ •ν˜• λ¬Έμ„œ 데이터λ₯Ό μ§€λŠ₯ν˜• 지식 베이슀둜 μ „ν™˜ν•˜λŠ” μ‹€λ¦¬μ½˜λ°Έλ¦¬ 기반의 AI μŠ€νƒ€νŠΈμ—…μž…λ‹ˆλ‹€. μš°λ¦¬λŠ” λ‹¨μˆœνžˆ λ¬Έμ„œλ₯Ό μ €μž₯ν•˜λŠ” 것을 λ„˜μ–΄, λ¬Έμ„œμ™€ λŒ€ν™”ν•˜κ³  μˆ¨κ²¨μ§„ 톡찰λ ₯을 μ¦‰μ‹œ μ°Ύμ•„λ‚΄λŠ” κ²½ν—˜μ„ μ œκ³΅ν•©λ‹ˆλ‹€. 그리고 였늘, λ‹Ήμ‹  **Alex**λŠ” AI μ—”μ§€λ‹ˆμ–΄λ‘œ νŒ€μ— ν•©λ₯˜ν–ˆμŠ΅λ‹ˆλ‹€.

## πŸ’¬ [Slack] `#docuflow-engineering` ### πŸ§‘β€πŸ’Ό Sarah (PM) > κ³ κ°μ‚¬λ“€λ‘œλΆ€ν„° ν”Όλ“œλ°±μ΄ μ™”λŠ”λ°, κΈ°μ‘΄ 챗봇이 λ¬Έμ„œμ— μ—†λŠ” λ‚΄μš©μ„ μ§€μ–΄λ‚΄μ„œ λ‹΅λ³€ν•œλ‹€κ³  ν•΄μš”(Hallucination). > 이번 MVP의 핡심은 **μ •ν™•μ„±**μž…λ‹ˆλ‹€. μ‚¬μš©μžκ°€ μ—…λ‘œλ“œν•œ PDF λ¬Έμ„œ λ‚΄μ—μ„œλ§Œ κ·Όκ±°λ₯Ό μ°Ύμ•„ λ‹΅λ³€ν•˜λŠ” **κ³ μ„±λŠ₯ RAG(검색 증강 생성) νŒŒμ΄ν”„λΌμΈ**이 λ°˜λ“œμ‹œ ν•„μš”ν•΄μš”.
### πŸ‘¨β€πŸ’» Dave (CTO) > Alex, 이번 과제의 핡심은 LLM을 λ‹¨μˆœνžˆ ν˜ΈμΆœν•˜λŠ” 게 μ•„λ‹ˆλΌ, **'μ–΄λ–»κ²Œ λ¬Έμ„œλ₯Ό κ²€μƒ‰ν•˜λŠλƒ'** 에 λ‹¬λ €μžˆλ„€. > λ‹¨μˆœνžˆ APIλ₯Ό μ—°κ²°ν•˜λŠ” μˆ˜μ€€μ„ λ„˜μ–΄, **λŒ€λŸ‰μ˜ λ¬Έμ„œλ₯Ό μ²˜λ¦¬ν•˜λŠ” μ„±λŠ₯**κ³Ό **μ •ν™•ν•œ 정보λ₯Ό μ°Ύμ•„λ‚΄λŠ” ν’ˆμ§ˆ**κΉŒμ§€ κ³ λ―Όν•΄μ•Ό ν•΄. > 2μ£Ό λ™μ•ˆ 주말 집쀑 μ‹€μŠ΅κ³Ό 평일 μ€€λΉ„λ₯Ό λ°˜λ³΅ν•˜λ©°, 우리 μ„œλΉ„μŠ€μ˜ μ½”μ–΄ 엔진을 μ™„μ„±ν•΄μ£Όκ²Œ.

## 🧩 ν”„λ‘œμ νŠΈ λͺ©ν‘œ **DocuFlow Agent**λŠ” RAG(Retrieval-Augmented Generation) κΈ°μˆ μ„ ν™œμš©ν•˜μ—¬ PDF λ¬Έμ„œμ˜ 지식을 AIμ—κ²Œ μ£Όμž…ν•˜λŠ” μ§€λŠ₯ν˜• 검색 μ‹œμŠ€ν…œμž…λ‹ˆλ‹€. λ³Έ ν”„λ‘œμ νŠΈλŠ” 2μ£Ό 운영(주말 νŽ˜μ–΄ν”„λ‘œκ·Έλž˜λ° + 평일 μ€€λΉ„/쑰사)을 톡해, 빈 껍데기뿐인 μ„œλ²„λ₯Ό **μƒμš© μˆ˜μ€€μ˜ RAG μ—”μ§„**으둜 λ°œμ „μ‹œν‚€λŠ” 것을 λͺ©ν‘œλ‘œ ν•©λ‹ˆλ‹€.

## βš™οΈ ν™˜κ²½ μ„€μ • μ•„λž˜ 1~6단계λ₯Ό μˆœμ„œλŒ€λ‘œ μ™„λ£Œν•΄μ•Ό ν”„λ‘œμ νŠΈλ₯Ό μ‹€ν–‰ν•  수 μžˆμŠ΅λ‹ˆλ‹€. ### 1. pyenv μ„€μΉ˜ (졜초 1회, macOS) pyenvκ°€ 이미 μ„€μΉ˜λ˜μ–΄ μžˆλ‹€λ©΄ 이 단계λ₯Ό κ±΄λ„ˆλ›°μ„Έμš”. ```bash # Homebrew둜 pyenv μ„€μΉ˜ brew install pyenv # μ…Έ 섀정에 pyenv μ΄ˆκΈ°ν™” μΆ”κ°€ (zsh κΈ°μ€€) echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc echo 'eval "$(pyenv init -)"' >> ~/.zshrc # μ„€μ • 반영 source ~/.zshrc # μ„€μΉ˜ 확인 (버전 λ²ˆν˜Έκ°€ 좜λ ₯되면 정상) pyenv --version ``` --- ### 2. Python 3.11 및 μ˜μ‘΄μ„± μ„€μΉ˜ ```bash # Python 3.11 μ„€μΉ˜ 및 μ„€μ • pyenv install 3.11 pyenv local 3.11 # μ„€μΉ˜ 확인 (3.11.x κ°€ 좜λ ₯되면 정상) python --version # pipenv μ„€μΉ˜ (졜초 1회) pip install pipenv # ν”„λ‘œμ νŠΈ μ˜μ‘΄μ„± μ„€μΉ˜ pipenv install ``` --- ### 3. Ollama μ„€μΉ˜ 및 λͺ¨λΈ λ‹€μš΄λ‘œλ“œ 이 ν”„λ‘œμ νŠΈλŠ” 기본적으둜 둜컬 Ollama μ„œλ²„λ₯Ό μ‚¬μš©ν•©λ‹ˆλ‹€. **macOS μ„€μΉ˜:** ```bash # Homebrew둜 μ„€μΉ˜ brew install --cask ollama # λ˜λŠ” 곡식 μ›Ήμ‚¬μ΄νŠΈμ—μ„œ λ‹€μš΄λ‘œλ“œ # https://ollama.com/download ``` **μ„œλ²„ μ‹€ν–‰:** μ„€μΉ˜ ν›„ **Ollama 앱을 μ‹€ν–‰**ν•˜λ©΄ 메뉴 바에 μ•„μ΄μ½˜μ΄ ν‘œμ‹œλ˜κ³  μ„œλ²„κ°€ μžλ™μœΌλ‘œ μ‹œμž‘λ©λ‹ˆλ‹€. > μ•± λ°©μ‹μœΌλ‘œ μ„€μΉ˜ν•œ 경우 `ollama serve`λ₯Ό λ³„λ„λ‘œ μ‹€ν–‰ν•˜μ§€ μ•Šμ•„λ„ λ©λ‹ˆλ‹€. 앱이 μ„œλ²„λ₯Ό μžλ™μœΌλ‘œ κ΄€λ¦¬ν•©λ‹ˆλ‹€. **λͺ¨λΈ λ‹€μš΄λ‘œλ“œ (μ„œλ²„ μ‹€ν–‰ ν›„ μ‹€ν–‰):** ```bash # μž„λ² λ”© λͺ¨λΈ (μ•½ 274MB) ollama pull nomic-embed-text # μ±„νŒ… λͺ¨λΈ (μ•½ 4.9GB, λ„€νŠΈμ›Œν¬ ν™˜κ²½μ— 따라 10λΆ„ 이상 μ†Œμš”) ollama pull llama3.1 ``` > ⚠️ `llama3.1` λ‹€μš΄λ‘œλ“œλŠ” μ‹œκ°„μ΄ 많이 μ†Œμš”λ©λ‹ˆλ‹€. > 주말 μ„Έμ…˜ 당일이 μ•„λ‹Œ **사전에 λ°˜λ“œμ‹œ μ™„λ£Œ**ν•΄λ‘μ„Έμš”. **μ„œλ²„ μƒνƒœ 확인:** ```bash curl http://localhost:11434/api/tags ``` λͺ¨λΈ λͺ©λ‘μ΄ JSON ν˜•νƒœλ‘œ 좜λ ₯되면 μ •μƒμž…λ‹ˆλ‹€. > Ollama μ„œλ²„λŠ” **ν”„λ‘œμ νŠΈ μ‹€ν–‰ μ€‘μ—λŠ” 계속 μΌœλ‘” μƒνƒœλ₯Ό μœ μ§€**ν•΄μ•Ό ν•©λ‹ˆλ‹€. μ„œλ²„κ°€ κΊΌμ§„ μƒνƒœμ—μ„œλŠ” μž„λ² λ”©/μ±„νŒ… μš”μ²­μ΄ μ‹€νŒ¨ν•©λ‹ˆλ‹€. **μ„œλ²„ μ’…λ£Œ (μ‹€μŠ΅ μ’…λ£Œ ν›„):** - **메뉴 λ°”** μƒλ‹¨μ˜ Ollama μ•„μ΄μ½˜ 클릭 β†’ **Quit Ollama** - λ˜λŠ” ν„°λ―Έλ„μ—μ„œ `killall Ollama` --- ### 4. Supabase ν”„λ‘œμ νŠΈ 생성 및 ν™˜κ²½ λ³€μˆ˜ μ„€μ • (.env) **Supabase ν”„λ‘œμ νŠΈ 생성:** 1. [supabase.com](https://supabase.com)에 μ ‘μ†ν•˜μ—¬ 계정을 λ§Œλ“€κ³  μƒˆ ν”„λ‘œμ νŠΈλ₯Ό μƒμ„±ν•˜μ„Έμš”. 2. ν”„λ‘œμ νŠΈ 생성 ν›„ μ•„λž˜ κ²½λ‘œμ—μ„œ 두 값을 볡사해 λ‘‘λ‹ˆλ‹€. - **Settings β†’ General β†’ Project ID** 확인 β†’ `https://.supabase.co` ν˜•νƒœλ‘œ `SUPABASE_URL`에 μž…λ ₯ - **Settings β†’ API Keys β†’ Legacy API keys β†’ `anon` ν‚€** β†’ `SUPABASE_KEY`에 μž…λ ₯ **ν™˜κ²½ λ³€μˆ˜ μ„€μ •:** ```bash cp .env.example .env # .env νŒŒμΌμ„ μ—΄μ–΄ λ³΅μ‚¬ν•œ 값을 μ±„μš°μ„Έμš” ``` ```ini # κΈ°λ³Έ: 둜컬 Ollama μ‚¬μš© PROVIDER="ollama" OLLAMA_BASE="http://localhost:11434" OLLAMA_EMBED_MODEL="nomic-embed-text" OLLAMA_CHAT_MODEL="llama3.1" # 곡톡 SUPABASE_URL="https://your-project-id.supabase.co" # Settings β†’ General β†’ Project ID SUPABASE_KEY="eyJ..." # Settings β†’ API Keys β†’ Legacy API keys β†’ anon # 선택: OpenAI μ‚¬μš© μ‹œ # PROVIDER="openai" # OPENAI_API_KEY="sk-..." # OPENAI_EMBEDDING_MODEL="text-embedding-3-small" # OPENAI_CHAT_MODEL="gpt-4o-mini" ``` --- ### 5. λ°μ΄ν„°λ² μ΄μŠ€(Supabase) ꡬ성 Supabase λŒ€μ‹œλ³΄λ“œμ˜ **SQL Editor**μ—μ„œ μ•„λž˜ 쿼리λ₯Ό μ‹€ν–‰ν•˜μ„Έμš”.
πŸ“œ SQL 쿼리 (κΈ°λ³Έ: Ollama 768차원)
```sql -- 1. 벑터 ν™•μž₯ ν™œμ„±ν™” CREATE EXTENSION IF NOT EXISTS vector; -- 2. λ¬Έμ„œ 청크 ν…Œμ΄λΈ” 생성 CREATE TABLE IF NOT EXISTS document_chunks ( id BIGSERIAL PRIMARY KEY, document_id TEXT NOT NULL, content TEXT NOT NULL, embedding vector(768), -- nomic-embed-text 차원 chunk_index INTEGER NOT NULL, metadata JSONB DEFAULT '{}', created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP ); -- 3. 검색 ν•¨μˆ˜(RPC) 생성 CREATE OR REPLACE FUNCTION match_documents ( query_embedding vector(768), match_count int, p_document_id text ) RETURNS TABLE ( id BIGINT, document_id TEXT, content TEXT, similarity FLOAT ) LANGUAGE plpgsql AS $$ BEGIN RETURN QUERY SELECT dc.id, dc.document_id, dc.content, 1 - (dc.embedding <=> query_embedding) as similarity FROM document_chunks AS dc WHERE dc.document_id = p_document_id ORDER BY dc.embedding <=> query_embedding LIMIT match_count; END; $$; -- 4. 인덱슀 생성 (HNSW 방식) CREATE INDEX ON document_chunks USING hnsw (embedding vector_cosine_ops); ```
πŸ“œ λŒ€μ•ˆ: OpenAI 1536차원 μŠ€ν‚€λ§ˆ
```sql CREATE EXTENSION IF NOT EXISTS vector; CREATE TABLE IF NOT EXISTS document_chunks ( id BIGSERIAL PRIMARY KEY, document_id TEXT NOT NULL, content TEXT NOT NULL, embedding vector(1536), -- text-embedding-3-small 차원 chunk_index INTEGER NOT NULL, metadata JSONB DEFAULT '{}', created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP ); CREATE OR REPLACE FUNCTION match_documents ( query_embedding vector(1536), match_count int, p_document_id text ) RETURNS TABLE ( id BIGINT, document_id TEXT, content TEXT, similarity FLOAT ) LANGUAGE plpgsql AS $$ BEGIN RETURN QUERY SELECT dc.id, dc.document_id, dc.content, 1 - (dc.embedding <=> query_embedding) as similarity FROM document_chunks AS dc WHERE dc.document_id = p_document_id ORDER BY dc.embedding <=> query_embedding LIMIT match_count; END; $$; CREATE INDEX ON document_chunks USING hnsw (embedding vector_cosine_ops); ```
μ‹€ν–‰ ν›„ Supabase λŒ€μ‹œλ³΄λ“œμ˜ **Table Editor**μ—μ„œ `document_chunks` ν…Œμ΄λΈ”μ΄ μƒμ„±λ˜μ—ˆλŠ”μ§€ ν™•μΈν•˜μ„Έμš”. --- ### 6. μ„œλ²„ μ‹€ν–‰ ```bash pipenv run uvicorn app.main:app --reload ``` μ‹€ν–‰ ν›„ λΈŒλΌμš°μ €μ—μ„œ `http://127.0.0.1:8000` μ ‘μ†μœΌλ‘œ UIλ₯Ό ν™•μΈν•˜μ„Έμš”. --- ### 7. μ½”λ“œ 포맀터 μ„€μ • (선택) 컀밋 μ „ μžλ™μœΌλ‘œ μ½”λ“œ μŠ€νƒ€μΌμ„ μ •λ¦¬ν•˜λŠ” `black` 포맀터λ₯Ό pre-commit hook으둜 μ„€μ •ν•  수 μžˆμŠ΅λ‹ˆλ‹€. ```bash # dev μ˜μ‘΄μ„± μ„€μΉ˜ (pre-commit, black 포함) pipenv install --dev # git hook 등둝 (졜초 1회) pipenv run pre-commit install ``` 등둝 ν›„μ—λŠ” `git commit` μ‹œ `black`이 μžλ™ μ‹€ν–‰λ©λ‹ˆλ‹€. μ½”λ“œ μŠ€νƒ€μΌμ΄ μˆ˜μ •λœ 경우 νŒŒμΌμ„ λ‹€μ‹œ `git add` ν•œ λ’€ μ»€λ°‹ν•˜λ©΄ λ©λ‹ˆλ‹€. > 이 단계λ₯Ό κ±΄λ„ˆλ›°μ–΄λ„ ν”„λ‘œμ νŠΈ μ‹€ν–‰μ—λŠ” 영ν–₯이 μ—†μŠ΅λ‹ˆλ‹€.

## πŸ—“οΈ 2μ£Ό μ§„ν–‰ ꡬ쑰 ν™˜κ²½ 섀정을 μ™„λ£Œν•œ ν›„, μ•„λž˜ μˆœμ„œλŒ€λ‘œ 단계별 λ¬Έμ„œλ₯Ό 따라 μ§„ν–‰ν•˜μ„Έμš”. - 주말(ν† /일, 9:00-18:00): νŽ˜μ–΄ν”„λ‘œκ·Έλž˜λ°μœΌλ‘œ κ΅¬ν˜„/검증 - 평일: 사전 μ€€λΉ„, 쑰사, μ‹€ν—˜ 기둝, μ˜μ‚¬κ²°μ • 정리 | 단계 | κΈ°κ°„ | λ¬Έμ„œ | | ------------- | ------------------ | ------------------------------------------------------------ | | **Prework** | Week 1 μ „ 평일 | [`docs/01-prework/README.md`](./docs/01-prework/README.md) | | **Weekend-1** | Week 1 주말 | [`docs/02-weekend1/README.md`](./docs/02-weekend1/README.md) | | **Midweek** | Week 1~2 사이 평일 | [`docs/03-midweek/README.md`](./docs/03-midweek/README.md) | | **Weekend-2** | Week 2 주말 | [`docs/04-weekend2/README.md`](./docs/04-weekend2/README.md) |

## πŸ“š μ°Έκ³  ν•™μŠ΅ 자료 ### ν”„λ‘œμ νŠΈ λ‚΄λΆ€ λ¬Έμ„œ - **[RAG μ‹œμŠ€ν…œ 기술 심측 뢄석](./docs/00-shared/reference.md)**: RAG μ•„ν‚€ν…μ²˜μ™€ 기술 μŠ€νƒ μ „λ°˜. **ν•„λ…ν•˜μ„Έμš”.** - **[배포 κ°€μ΄λ“œ](./docs/04-weekend2/deployment.md)**: 배포 상세 κ°€μ΄λ“œ(심화/μ°Έκ³ )μž…λ‹ˆλ‹€. ### μ™ΈλΆ€ μ°Έκ³  자료 - **[What is RAG? (IBM)](https://www.ibm.com/topics/retrieval-augmented-generation)**: RAG κ°œλ…μ„ κ·Έλ¦Όκ³Ό ν•¨κ»˜ μ„€λͺ…ν•˜λŠ” μž…λ¬Έ μžλ£Œμž…λ‹ˆλ‹€. - **[RAG from Scratch (LangChain YouTube)](https://www.youtube.com/playlist?list=PLfaIDFEXuae2LXbO1_PKyVJiQ23ZztA0x)**: LangChain 곡식 RAG κ΅¬ν˜„ μ˜μƒ μ‹œλ¦¬μ¦ˆμž…λ‹ˆλ‹€ (μ˜μ–΄). - **[pgvector GitHub](https://github.com/pgvector/pgvector)**: `<=>` μ—°μ‚°μžμ™€ HNSW 인덱슀 μž‘λ™ 방식을 ν™•μΈν•˜μ„Έμš”. - **[Supabase Vector κ°€μ΄λ“œ](https://supabase.com/docs/guides/ai)**: PostgreSQL을 벑터 λ°μ΄ν„°λ² μ΄μŠ€λ‘œ μ‚¬μš©ν•˜λŠ” λ°©λ²•μž…λ‹ˆλ‹€. - **[LangChain RecursiveCharacterTextSplitter](https://reference.langchain.com/python/langchain-text-splitters/character/RecursiveCharacterTextSplitter)**: ν…μŠ€νŠΈ μ²­ν‚Ή μ „λž΅μ— λŒ€ν•œ Python 곡식 λ ˆνΌλŸ°μŠ€μž…λ‹ˆλ‹€. - **[OpenAI Embeddings API](https://platform.openai.com/docs/guides/embeddings)**: ν…μŠ€νŠΈλ₯Ό 숫자둜 λ³€ν™˜ν•˜λŠ” 원리λ₯Ό 이해해야 ν•©λ‹ˆλ‹€. - **[Vector Similarity Explained (Pinecone)](https://www.pinecone.io/learn/vector-similarity/)**: μœ μ‚¬λ„ μΈ‘μ • 방식 비ꡐ μžλ£Œμž…λ‹ˆλ‹€.