docuflow / README.md
woongmin's picture
[Delta-02] HF Spaces README frontmatter μΆ”κ°€
1dd2838
|
Raw History Blame Contribute Delete
11.5 kB
metadata
title: DocuFlow
emoji: πŸ“„
colorFrom: blue
colorTo: green
sdk: docker
app_port: 7860


Pi Lab(파이랩)의 λͺ¨λ“  κ³Όμ œλŠ” μ‹€μ œ κΈ°μ—…μ—μ„œ μ£Όμ–΄μ§€λŠ” κ³Όμ œμ— κΈ°λ°˜ν•˜μ—¬ μ œμž‘λ˜μ—ˆμœΌλ©°, μ €μž‘κΆŒλ²•μ˜ 보호λ₯Ό λ°›μŠ΅λ‹ˆλ‹€.



Project DocuFlow: Intelligent Document Agent



🧠 The Future of Document Processing

πŸ“ HQ: 42 Silicon Valley Blvd, Palo Alto, CA


"Read less, Understand more."


DocuFlow AIλŠ” μ „ μ„Έκ³„μ˜ λΉ„μ •ν˜• λ¬Έμ„œ 데이터λ₯Ό μ§€λŠ₯ν˜• 지식 베이슀둜 μ „ν™˜ν•˜λŠ” μ‹€λ¦¬μ½˜λ°Έλ¦¬ 기반의 AI μŠ€νƒ€νŠΈμ—…μž…λ‹ˆλ‹€. μš°λ¦¬λŠ” λ‹¨μˆœνžˆ λ¬Έμ„œλ₯Ό μ €μž₯ν•˜λŠ” 것을 λ„˜μ–΄, λ¬Έμ„œμ™€ λŒ€ν™”ν•˜κ³  μˆ¨κ²¨μ§„ 톡찰λ ₯을 μ¦‰μ‹œ μ°Ύμ•„λ‚΄λŠ” κ²½ν—˜μ„ μ œκ³΅ν•©λ‹ˆλ‹€.

그리고 였늘, λ‹Ήμ‹  AlexλŠ” AI μ—”μ§€λ‹ˆμ–΄λ‘œ νŒ€μ— ν•©λ₯˜ν–ˆμŠ΅λ‹ˆλ‹€.



πŸ’¬ [Slack] #docuflow-engineering

πŸ§‘β€πŸ’Ό Sarah (PM)

κ³ κ°μ‚¬λ“€λ‘œλΆ€ν„° ν”Όλ“œλ°±μ΄ μ™”λŠ”λ°, κΈ°μ‘΄ 챗봇이 λ¬Έμ„œμ— μ—†λŠ” λ‚΄μš©μ„ μ§€μ–΄λ‚΄μ„œ λ‹΅λ³€ν•œλ‹€κ³  ν•΄μš”(Hallucination). 이번 MVP의 핡심은 μ •ν™•μ„±μž…λ‹ˆλ‹€. μ‚¬μš©μžκ°€ μ—…λ‘œλ“œν•œ PDF λ¬Έμ„œ λ‚΄μ—μ„œλ§Œ κ·Όκ±°λ₯Ό μ°Ύμ•„ λ‹΅λ³€ν•˜λŠ” κ³ μ„±λŠ₯ RAG(검색 증강 생성) νŒŒμ΄ν”„λΌμΈμ΄ λ°˜λ“œμ‹œ ν•„μš”ν•΄μš”.


πŸ‘¨β€πŸ’» Dave (CTO)

Alex, 이번 과제의 핡심은 LLM을 λ‹¨μˆœνžˆ ν˜ΈμΆœν•˜λŠ” 게 μ•„λ‹ˆλΌ, 'μ–΄λ–»κ²Œ λ¬Έμ„œλ₯Ό κ²€μƒ‰ν•˜λŠλƒ' 에 λ‹¬λ €μžˆλ„€. λ‹¨μˆœνžˆ APIλ₯Ό μ—°κ²°ν•˜λŠ” μˆ˜μ€€μ„ λ„˜μ–΄, λŒ€λŸ‰μ˜ λ¬Έμ„œλ₯Ό μ²˜λ¦¬ν•˜λŠ” μ„±λŠ₯κ³Ό μ •ν™•ν•œ 정보λ₯Ό μ°Ύμ•„λ‚΄λŠ” ν’ˆμ§ˆκΉŒμ§€ κ³ λ―Όν•΄μ•Ό ν•΄. 2μ£Ό λ™μ•ˆ 주말 집쀑 μ‹€μŠ΅κ³Ό 평일 μ€€λΉ„λ₯Ό λ°˜λ³΅ν•˜λ©°, 우리 μ„œλΉ„μŠ€μ˜ μ½”μ–΄ 엔진을 μ™„μ„±ν•΄μ£Όκ²Œ.



🧩 ν”„λ‘œμ νŠΈ λͺ©ν‘œ

DocuFlow AgentλŠ” RAG(Retrieval-Augmented Generation) κΈ°μˆ μ„ ν™œμš©ν•˜μ—¬ PDF λ¬Έμ„œμ˜ 지식을 AIμ—κ²Œ μ£Όμž…ν•˜λŠ” μ§€λŠ₯ν˜• 검색 μ‹œμŠ€ν…œμž…λ‹ˆλ‹€.

λ³Έ ν”„λ‘œμ νŠΈλŠ” 2μ£Ό 운영(주말 νŽ˜μ–΄ν”„λ‘œκ·Έλž˜λ° + 평일 μ€€λΉ„/쑰사)을 톡해, 빈 껍데기뿐인 μ„œλ²„λ₯Ό μƒμš© μˆ˜μ€€μ˜ RAG μ—”μ§„μœΌλ‘œ λ°œμ „μ‹œν‚€λŠ” 것을 λͺ©ν‘œλ‘œ ν•©λ‹ˆλ‹€.



βš™οΈ ν™˜κ²½ μ„€μ •

μ•„λž˜ 1~6단계λ₯Ό μˆœμ„œλŒ€λ‘œ μ™„λ£Œν•΄μ•Ό ν”„λ‘œμ νŠΈλ₯Ό μ‹€ν–‰ν•  수 μžˆμŠ΅λ‹ˆλ‹€.

1. pyenv μ„€μΉ˜ (졜초 1회, macOS)

pyenvκ°€ 이미 μ„€μΉ˜λ˜μ–΄ μžˆλ‹€λ©΄ 이 단계λ₯Ό κ±΄λ„ˆλ›°μ„Έμš”.

# Homebrew둜 pyenv μ„€μΉ˜
brew install pyenv

# μ…Έ 섀정에 pyenv μ΄ˆκΈ°ν™” μΆ”κ°€ (zsh κΈ°μ€€)
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc
echo 'eval "$(pyenv init -)"' >> ~/.zshrc

# μ„€μ • 반영
source ~/.zshrc

# μ„€μΉ˜ 확인 (버전 λ²ˆν˜Έκ°€ 좜λ ₯되면 정상)
pyenv --version

2. Python 3.11 및 μ˜μ‘΄μ„± μ„€μΉ˜

# Python 3.11 μ„€μΉ˜ 및 μ„€μ •
pyenv install 3.11
pyenv local 3.11

# μ„€μΉ˜ 확인 (3.11.x κ°€ 좜λ ₯되면 정상)
python --version

# pipenv μ„€μΉ˜ (졜초 1회)
pip install pipenv

# ν”„λ‘œμ νŠΈ μ˜μ‘΄μ„± μ„€μΉ˜
pipenv install

3. Ollama μ„€μΉ˜ 및 λͺ¨λΈ λ‹€μš΄λ‘œλ“œ

이 ν”„λ‘œμ νŠΈλŠ” 기본적으둜 둜컬 Ollama μ„œλ²„λ₯Ό μ‚¬μš©ν•©λ‹ˆλ‹€.

macOS μ„€μΉ˜:

# Homebrew둜 μ„€μΉ˜
brew install --cask ollama

# λ˜λŠ” 곡식 μ›Ήμ‚¬μ΄νŠΈμ—μ„œ λ‹€μš΄λ‘œλ“œ
# https://ollama.com/download

μ„œλ²„ μ‹€ν–‰:

μ„€μΉ˜ ν›„ Ollama 앱을 μ‹€ν–‰ν•˜λ©΄ 메뉴 바에 μ•„μ΄μ½˜μ΄ ν‘œμ‹œλ˜κ³  μ„œλ²„κ°€ μžλ™μœΌλ‘œ μ‹œμž‘λ©λ‹ˆλ‹€.

μ•± λ°©μ‹μœΌλ‘œ μ„€μΉ˜ν•œ 경우 ollama serveλ₯Ό λ³„λ„λ‘œ μ‹€ν–‰ν•˜μ§€ μ•Šμ•„λ„ λ©λ‹ˆλ‹€. 앱이 μ„œλ²„λ₯Ό μžλ™μœΌλ‘œ κ΄€λ¦¬ν•©λ‹ˆλ‹€.

λͺ¨λΈ λ‹€μš΄λ‘œλ“œ (μ„œλ²„ μ‹€ν–‰ ν›„ μ‹€ν–‰):

# μž„λ² λ”© λͺ¨λΈ (μ•½ 274MB)
ollama pull nomic-embed-text

# μ±„νŒ… λͺ¨λΈ (μ•½ 4.9GB, λ„€νŠΈμ›Œν¬ ν™˜κ²½μ— 따라 10λΆ„ 이상 μ†Œμš”)
ollama pull llama3.1

⚠️ llama3.1 λ‹€μš΄λ‘œλ“œλŠ” μ‹œκ°„μ΄ 많이 μ†Œμš”λ©λ‹ˆλ‹€. 주말 μ„Έμ…˜ 당일이 μ•„λ‹Œ 사전에 λ°˜λ“œμ‹œ μ™„λ£Œν•΄λ‘μ„Έμš”.

μ„œλ²„ μƒνƒœ 확인:

curl http://localhost:11434/api/tags

λͺ¨λΈ λͺ©λ‘μ΄ JSON ν˜•νƒœλ‘œ 좜λ ₯되면 μ •μƒμž…λ‹ˆλ‹€.

Ollama μ„œλ²„λŠ” ν”„λ‘œμ νŠΈ μ‹€ν–‰ μ€‘μ—λŠ” 계속 μΌœλ‘” μƒνƒœλ₯Ό μœ μ§€ν•΄μ•Ό ν•©λ‹ˆλ‹€. μ„œλ²„κ°€ κΊΌμ§„ μƒνƒœμ—μ„œλŠ” μž„λ² λ”©/μ±„νŒ… μš”μ²­μ΄ μ‹€νŒ¨ν•©λ‹ˆλ‹€.

μ„œλ²„ μ’…λ£Œ (μ‹€μŠ΅ μ’…λ£Œ ν›„):

  • 메뉴 λ°” μƒλ‹¨μ˜ Ollama μ•„μ΄μ½˜ 클릭 β†’ Quit Ollama
  • λ˜λŠ” ν„°λ―Έλ„μ—μ„œ killall Ollama

4. Supabase ν”„λ‘œμ νŠΈ 생성 및 ν™˜κ²½ λ³€μˆ˜ μ„€μ • (.env)

Supabase ν”„λ‘œμ νŠΈ 생성:

  1. supabase.com에 μ ‘μ†ν•˜μ—¬ 계정을 λ§Œλ“€κ³  μƒˆ ν”„λ‘œμ νŠΈλ₯Ό μƒμ„±ν•˜μ„Έμš”.
  2. ν”„λ‘œμ νŠΈ 생성 ν›„ μ•„λž˜ κ²½λ‘œμ—μ„œ 두 값을 볡사해 λ‘‘λ‹ˆλ‹€.
    • Settings β†’ General β†’ Project ID 확인 β†’ https://<project-id>.supabase.co ν˜•νƒœλ‘œ SUPABASE_URL에 μž…λ ₯
    • Settings β†’ API Keys β†’ Legacy API keys β†’ anon ν‚€ β†’ SUPABASE_KEY에 μž…λ ₯

ν™˜κ²½ λ³€μˆ˜ μ„€μ •:

cp .env.example .env
# .env νŒŒμΌμ„ μ—΄μ–΄ λ³΅μ‚¬ν•œ 값을 μ±„μš°μ„Έμš”
# κΈ°λ³Έ: 둜컬 Ollama μ‚¬μš©
PROVIDER="ollama"
OLLAMA_BASE="http://localhost:11434"
OLLAMA_EMBED_MODEL="nomic-embed-text"
OLLAMA_CHAT_MODEL="llama3.1"

# 곡톡
SUPABASE_URL="https://your-project-id.supabase.co"  # Settings β†’ General β†’ Project ID
SUPABASE_KEY="eyJ..."                               # Settings β†’ API Keys β†’ Legacy API keys β†’ anon

# 선택: OpenAI μ‚¬μš© μ‹œ
# PROVIDER="openai"
# OPENAI_API_KEY="sk-..."
# OPENAI_EMBEDDING_MODEL="text-embedding-3-small"
# OPENAI_CHAT_MODEL="gpt-4o-mini"

5. λ°μ΄ν„°λ² μ΄μŠ€(Supabase) ꡬ성

Supabase λŒ€μ‹œλ³΄λ“œμ˜ SQL Editorμ—μ„œ μ•„λž˜ 쿼리λ₯Ό μ‹€ν–‰ν•˜μ„Έμš”.

πŸ“œ SQL 쿼리 (κΈ°λ³Έ: Ollama 768차원)
-- 1. 벑터 ν™•μž₯ ν™œμ„±ν™”
CREATE EXTENSION IF NOT EXISTS vector;

-- 2. λ¬Έμ„œ 청크 ν…Œμ΄λΈ” 생성
CREATE TABLE IF NOT EXISTS document_chunks (
    id BIGSERIAL PRIMARY KEY,
    document_id TEXT NOT NULL,
    content TEXT NOT NULL,
    embedding vector(768), -- nomic-embed-text 차원
    chunk_index INTEGER NOT NULL,
    metadata JSONB DEFAULT '{}',
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);

-- 3. 검색 ν•¨μˆ˜(RPC) 생성
CREATE OR REPLACE FUNCTION match_documents (
    query_embedding vector(768),
    match_count int,
    p_document_id text
) RETURNS TABLE (
    id BIGINT,
    document_id TEXT,
    content TEXT,
    similarity FLOAT
)
LANGUAGE plpgsql
AS $$
BEGIN
    RETURN QUERY
    SELECT
        dc.id,
        dc.document_id,
        dc.content,
        1 - (dc.embedding <=> query_embedding) as similarity
    FROM document_chunks AS dc
    WHERE dc.document_id = p_document_id
    ORDER BY dc.embedding <=> query_embedding
    LIMIT match_count;
END;
;βˆ’βˆ’4.μΈλ±μŠ€μƒμ„±(HNSW방식)CREATEINDEXONdocumentchunksUSINGhnsw(embeddingvectorcosineops);β€˜β€˜β€˜</details><details><summary><b>πŸ“œλŒ€μ•ˆ:OpenAI1536μ°¨μ›μŠ€ν‚€λ§ˆ</b></summary><br>β€˜β€˜β€˜sqlCREATEEXTENSIONIFNOTEXISTSvector;CREATETABLEIFNOTEXISTSdocumentchunks(idBIGSERIALPRIMARYKEY,documentidTEXTNOTNULL,contentTEXTNOTNULL,embeddingvector(1536),βˆ’βˆ’textβˆ’embeddingβˆ’3βˆ’small차원chunkindexINTEGERNOTNULL,metadataJSONBDEFAULTβ€²β€²,createdatTIMESTAMPWITHTIMEZONEDEFAULTCURRENTTIMESTAMP);CREATEORREPLACEFUNCTIONmatchdocuments(queryembeddingvector(1536),matchcountint,pdocumentidtext)RETURNSTABLE(idBIGINT,documentidTEXT,contentTEXT,similarityFLOAT)LANGUAGEplpgsqlAS;

-- 4. 인덱슀 생성 (HNSW 방식)
CREATE INDEX ON document_chunks USING hnsw (embedding vector_cosine_ops);
```

</details>

<details>
<summary><b>πŸ“œ λŒ€μ•ˆ: OpenAI 1536차원 μŠ€ν‚€λ§ˆ</b></summary>
<br>

```sql
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE IF NOT EXISTS document_chunks (
    id BIGSERIAL PRIMARY KEY,
    document_id TEXT NOT NULL,
    content TEXT NOT NULL,
    embedding vector(1536), -- text-embedding-3-small 차원
    chunk_index INTEGER NOT NULL,
    metadata JSONB DEFAULT '{}',
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);

CREATE OR REPLACE FUNCTION match_documents (
    query_embedding vector(1536),
    match_count int,
    p_document_id text
) RETURNS TABLE (
    id BIGINT,
    document_id TEXT,
    content TEXT,
    similarity FLOAT
)
LANGUAGE plpgsql
AS 
BEGIN
    RETURN QUERY
    SELECT
        dc.id,
        dc.document_id,
        dc.content,
        1 - (dc.embedding <=> query_embedding) as similarity
    FROM document_chunks AS dc
    WHERE dc.document_id = p_document_id
    ORDER BY dc.embedding <=> query_embedding
    LIMIT match_count;
END;
$$;

CREATE INDEX ON document_chunks USING hnsw (embedding vector_cosine_ops);

μ‹€ν–‰ ν›„ Supabase λŒ€μ‹œλ³΄λ“œμ˜ Table Editorμ—μ„œ document_chunks ν…Œμ΄λΈ”μ΄ μƒμ„±λ˜μ—ˆλŠ”μ§€ ν™•μΈν•˜μ„Έμš”.


6. μ„œλ²„ μ‹€ν–‰

pipenv run uvicorn app.main:app --reload

μ‹€ν–‰ ν›„ λΈŒλΌμš°μ €μ—μ„œ http://127.0.0.1:8000 μ ‘μ†μœΌλ‘œ UIλ₯Ό ν™•μΈν•˜μ„Έμš”.


7. μ½”λ“œ 포맀터 μ„€μ • (선택)

컀밋 μ „ μžλ™μœΌλ‘œ μ½”λ“œ μŠ€νƒ€μΌμ„ μ •λ¦¬ν•˜λŠ” black 포맀터λ₯Ό pre-commit hook으둜 μ„€μ •ν•  수 μžˆμŠ΅λ‹ˆλ‹€.

# dev μ˜μ‘΄μ„± μ„€μΉ˜ (pre-commit, black 포함)
pipenv install --dev

# git hook 등둝 (졜초 1회)
pipenv run pre-commit install

등둝 ν›„μ—λŠ” git commit μ‹œ black이 μžλ™ μ‹€ν–‰λ©λ‹ˆλ‹€. μ½”λ“œ μŠ€νƒ€μΌμ΄ μˆ˜μ •λœ 경우 νŒŒμΌμ„ λ‹€μ‹œ git add ν•œ λ’€ μ»€λ°‹ν•˜λ©΄ λ©λ‹ˆλ‹€.

이 단계λ₯Ό κ±΄λ„ˆλ›°μ–΄λ„ ν”„λ‘œμ νŠΈ μ‹€ν–‰μ—λŠ” 영ν–₯이 μ—†μŠ΅λ‹ˆλ‹€.



πŸ—“οΈ 2μ£Ό μ§„ν–‰ ꡬ쑰

ν™˜κ²½ 섀정을 μ™„λ£Œν•œ ν›„, μ•„λž˜ μˆœμ„œλŒ€λ‘œ 단계별 λ¬Έμ„œλ₯Ό 따라 μ§„ν–‰ν•˜μ„Έμš”.

  • 주말(ν† /일, 9:00-18:00): νŽ˜μ–΄ν”„λ‘œκ·Έλž˜λ°μœΌλ‘œ κ΅¬ν˜„/검증
  • 평일: 사전 μ€€λΉ„, 쑰사, μ‹€ν—˜ 기둝, μ˜μ‚¬κ²°μ • 정리
단계 κΈ°κ°„ λ¬Έμ„œ
Prework Week 1 μ „ 평일 docs/01-prework/README.md
Weekend-1 Week 1 주말 docs/02-weekend1/README.md
Midweek Week 1~2 사이 평일 docs/03-midweek/README.md
Weekend-2 Week 2 주말 docs/04-weekend2/README.md


πŸ“š μ°Έκ³  ν•™μŠ΅ 자료

ν”„λ‘œμ νŠΈ λ‚΄λΆ€ λ¬Έμ„œ

μ™ΈλΆ€ μ°Έκ³  자료

  • What is RAG? (IBM): RAG κ°œλ…μ„ κ·Έλ¦Όκ³Ό ν•¨κ»˜ μ„€λͺ…ν•˜λŠ” μž…λ¬Έ μžλ£Œμž…λ‹ˆλ‹€.
  • RAG from Scratch (LangChain YouTube): LangChain 곡식 RAG κ΅¬ν˜„ μ˜μƒ μ‹œλ¦¬μ¦ˆμž…λ‹ˆλ‹€ (μ˜μ–΄).
  • pgvector GitHub: <=> μ—°μ‚°μžμ™€ HNSW 인덱슀 μž‘λ™ 방식을 ν™•μΈν•˜μ„Έμš”.
  • Supabase Vector κ°€μ΄λ“œ: PostgreSQL을 벑터 λ°μ΄ν„°λ² μ΄μŠ€λ‘œ μ‚¬μš©ν•˜λŠ” λ°©λ²•μž…λ‹ˆλ‹€.
  • LangChain RecursiveCharacterTextSplitter: ν…μŠ€νŠΈ μ²­ν‚Ή μ „λž΅μ— λŒ€ν•œ Python 곡식 λ ˆνΌλŸ°μŠ€μž…λ‹ˆλ‹€.
  • OpenAI Embeddings API: ν…μŠ€νŠΈλ₯Ό 숫자둜 λ³€ν™˜ν•˜λŠ” 원리λ₯Ό 이해해야 ν•©λ‹ˆλ‹€.
  • Vector Similarity Explained (Pinecone): μœ μ‚¬λ„ μΈ‘μ • 방식 비ꡐ μžλ£Œμž…λ‹ˆλ‹€.