Spaces:
Runtime error
Runtime error
|
Download README.md from progressive-intelligence-lab/docuflow: direct link, hf CLI and curl.
- Browser
- Download file 11.5 kB
-
https://huggingface.co/spaces/progressive-intelligence-lab/docuflow/resolve/main/README.md
- Command line
-
hf download hf://spaces/progressive-intelligence-lab/docuflow/README.md
-
curl -L -o README.md https://huggingface.co/spaces/progressive-intelligence-lab/docuflow/resolve/main/README.md
11.5 kB
| title: DocuFlow | |
| emoji: π | |
| colorFrom: blue | |
| colorTo: green | |
| sdk: docker | |
| app_port: 7860 | |
| <br> | |
| <br> | |
| > Pi Lab(νμ΄λ©)μ λͺ¨λ κ³Όμ λ μ€μ κΈ°μ μμ μ£Όμ΄μ§λ κ³Όμ μ κΈ°λ°νμ¬ μ μλμμΌλ©°, μ μκΆλ²μ 보νΈλ₯Ό λ°μ΅λλ€. | |
| <br> | |
| <br> | |
| # Project DocuFlow: Intelligent Document Agent | |
| <br> | |
| <p align="center"> | |
| <img src="/assets/docuflow.jpg" width="100%"> | |
| </p> | |
| <br> | |
| **π§ The Future of Document Processing** | |
| _π HQ: 42 Silicon Valley Blvd, Palo Alto, CA_ | |
| <br> | |
| > "Read less, Understand more." | |
| <br> | |
| **DocuFlow AI**λ μ μΈκ³μ λΉμ ν λ¬Έμ λ°μ΄ν°λ₯Ό μ§λ₯ν μ§μ λ² μ΄μ€λ‘ μ ννλ μ€λ¦¬μ½λ°Έλ¦¬ κΈ°λ°μ AI μ€ννΈμ μ λλ€. μ°λ¦¬λ λ¨μν λ¬Έμλ₯Ό μ μ₯νλ κ²μ λμ΄, λ¬Έμμ λννκ³ μ¨κ²¨μ§ ν΅μ°°λ ₯μ μ¦μ μ°Ύμλ΄λ κ²½νμ μ 곡ν©λλ€. | |
| κ·Έλ¦¬κ³ μ€λ, λΉμ **Alex**λ AI μμ§λμ΄λ‘ νμ ν©λ₯νμ΅λλ€. | |
| <br> | |
| <br> | |
| ## π¬ [Slack] `#docuflow-engineering` | |
| ### π§βπΌ Sarah (PM) | |
| > κ³ κ°μ¬λ€λ‘λΆν° νΌλλ°±μ΄ μλλ°, κΈ°μ‘΄ μ±λ΄μ΄ λ¬Έμμ μλ λ΄μ©μ μ§μ΄λ΄μ λ΅λ³νλ€κ³ ν΄μ(Hallucination). | |
| > μ΄λ² MVPμ ν΅μ¬μ **μ νμ±**μ λλ€. μ¬μ©μκ° μ λ‘λν PDF λ¬Έμ λ΄μμλ§ κ·Όκ±°λ₯Ό μ°Ύμ λ΅λ³νλ **κ³ μ±λ₯ RAG(κ²μ μ¦κ° μμ±) νμ΄νλΌμΈ**μ΄ λ°λμ νμν΄μ. | |
| <br> | |
| ### π¨βπ» Dave (CTO) | |
| > Alex, μ΄λ² κ³Όμ μ ν΅μ¬μ LLMμ λ¨μν νΈμΆνλ κ² μλλΌ, **'μ΄λ»κ² λ¬Έμλ₯Ό κ²μνλλ'** μ λ¬λ €μλ€. | |
| > λ¨μν APIλ₯Ό μ°κ²°νλ μμ€μ λμ΄, **λλμ λ¬Έμλ₯Ό μ²λ¦¬νλ μ±λ₯**κ³Ό **μ νν μ 보λ₯Ό μ°Ύμλ΄λ νμ§**κΉμ§ κ³ λ―Όν΄μΌ ν΄. | |
| > 2μ£Ό λμ μ£Όλ§ μ§μ€ μ€μ΅κ³Ό νμΌ μ€λΉλ₯Ό λ°λ³΅νλ©°, μ°λ¦¬ μλΉμ€μ μ½μ΄ μμ§μ μμ±ν΄μ£Όκ². | |
| <br> | |
| <br> | |
| ## π§© νλ‘μ νΈ λͺ©ν | |
| **DocuFlow Agent**λ RAG(Retrieval-Augmented Generation) κΈ°μ μ νμ©νμ¬ PDF λ¬Έμμ μ§μμ AIμκ² μ£Όμ νλ μ§λ₯ν κ²μ μμ€ν μ λλ€. | |
| λ³Έ νλ‘μ νΈλ 2μ£Ό μ΄μ(μ£Όλ§ νμ΄νλ‘κ·Έλλ° + νμΌ μ€λΉ/μ‘°μ¬)μ ν΅ν΄, λΉ κ»λ°κΈ°λΏμΈ μλ²λ₯Ό **μμ© μμ€μ RAG μμ§**μΌλ‘ λ°μ μν€λ κ²μ λͺ©νλ‘ ν©λλ€. | |
| <br> | |
| <br> | |
| ## βοΈ νκ²½ μ€μ | |
| μλ 1~6λ¨κ³λ₯Ό μμλλ‘ μλ£ν΄μΌ νλ‘μ νΈλ₯Ό μ€νν μ μμ΅λλ€. | |
| ### 1. pyenv μ€μΉ (μ΅μ΄ 1ν, macOS) | |
| pyenvκ° μ΄λ―Έ μ€μΉλμ΄ μλ€λ©΄ μ΄ λ¨κ³λ₯Ό 건λλ°μΈμ. | |
| ```bash | |
| # Homebrewλ‘ pyenv μ€μΉ | |
| brew install pyenv | |
| # μ Έ μ€μ μ pyenv μ΄κΈ°ν μΆκ° (zsh κΈ°μ€) | |
| echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc | |
| echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc | |
| echo 'eval "$(pyenv init -)"' >> ~/.zshrc | |
| # μ€μ λ°μ | |
| source ~/.zshrc | |
| # μ€μΉ νμΈ (λ²μ λ²νΈκ° μΆλ ₯λλ©΄ μ μ) | |
| pyenv --version | |
| ``` | |
| --- | |
| ### 2. Python 3.11 λ° μμ‘΄μ± μ€μΉ | |
| ```bash | |
| # Python 3.11 μ€μΉ λ° μ€μ | |
| pyenv install 3.11 | |
| pyenv local 3.11 | |
| # μ€μΉ νμΈ (3.11.x κ° μΆλ ₯λλ©΄ μ μ) | |
| python --version | |
| # pipenv μ€μΉ (μ΅μ΄ 1ν) | |
| pip install pipenv | |
| # νλ‘μ νΈ μμ‘΄μ± μ€μΉ | |
| pipenv install | |
| ``` | |
| --- | |
| ### 3. Ollama μ€μΉ λ° λͺ¨λΈ λ€μ΄λ‘λ | |
| μ΄ νλ‘μ νΈλ κΈ°λ³Έμ μΌλ‘ λ‘컬 Ollama μλ²λ₯Ό μ¬μ©ν©λλ€. | |
| **macOS μ€μΉ:** | |
| ```bash | |
| # Homebrewλ‘ μ€μΉ | |
| brew install --cask ollama | |
| # λλ 곡μ μΉμ¬μ΄νΈμμ λ€μ΄λ‘λ | |
| # https://ollama.com/download | |
| ``` | |
| **μλ² μ€ν:** | |
| μ€μΉ ν **Ollama μ±μ μ€ν**νλ©΄ λ©λ΄ λ°μ μμ΄μ½μ΄ νμλκ³ μλ²κ° μλμΌλ‘ μμλ©λλ€. | |
| > μ± λ°©μμΌλ‘ μ€μΉν κ²½μ° `ollama serve`λ₯Ό λ³λλ‘ μ€ννμ§ μμλ λ©λλ€. μ±μ΄ μλ²λ₯Ό μλμΌλ‘ κ΄λ¦¬ν©λλ€. | |
| **λͺ¨λΈ λ€μ΄λ‘λ (μλ² μ€ν ν μ€ν):** | |
| ```bash | |
| # μλ² λ© λͺ¨λΈ (μ½ 274MB) | |
| ollama pull nomic-embed-text | |
| # μ±ν λͺ¨λΈ (μ½ 4.9GB, λ€νΈμν¬ νκ²½μ λ°λΌ 10λΆ μ΄μ μμ) | |
| ollama pull llama3.1 | |
| ``` | |
| > β οΈ `llama3.1` λ€μ΄λ‘λλ μκ°μ΄ λ§μ΄ μμλ©λλ€. | |
| > μ£Όλ§ μΈμ λΉμΌμ΄ μλ **μ¬μ μ λ°λμ μλ£**ν΄λμΈμ. | |
| **μλ² μν νμΈ:** | |
| ```bash | |
| curl http://localhost:11434/api/tags | |
| ``` | |
| λͺ¨λΈ λͺ©λ‘μ΄ JSON ννλ‘ μΆλ ₯λλ©΄ μ μμ λλ€. | |
| > Ollama μλ²λ **νλ‘μ νΈ μ€ν μ€μλ κ³μ μΌλ μνλ₯Ό μ μ§**ν΄μΌ ν©λλ€. μλ²κ° κΊΌμ§ μνμμλ μλ² λ©/μ±ν μμ²μ΄ μ€ν¨ν©λλ€. | |
| **μλ² μ’ λ£ (μ€μ΅ μ’ λ£ ν):** | |
| - **λ©λ΄ λ°** μλ¨μ Ollama μμ΄μ½ ν΄λ¦ β **Quit Ollama** | |
| - λλ ν°λ―Έλμμ `killall Ollama` | |
| --- | |
| ### 4. Supabase νλ‘μ νΈ μμ± λ° νκ²½ λ³μ μ€μ (.env) | |
| **Supabase νλ‘μ νΈ μμ±:** | |
| 1. [supabase.com](https://supabase.com)μ μ μνμ¬ κ³μ μ λ§λ€κ³ μ νλ‘μ νΈλ₯Ό μμ±νμΈμ. | |
| 2. νλ‘μ νΈ μμ± ν μλ κ²½λ‘μμ λ κ°μ 볡μ¬ν΄ λ‘λλ€. | |
| - **Settings β General β Project ID** νμΈ β `https://<project-id>.supabase.co` ννλ‘ `SUPABASE_URL`μ μ λ ₯ | |
| - **Settings β API Keys β Legacy API keys β `anon` ν€** β `SUPABASE_KEY`μ μ λ ₯ | |
| **νκ²½ λ³μ μ€μ :** | |
| ```bash | |
| cp .env.example .env | |
| # .env νμΌμ μ΄μ΄ 볡μ¬ν κ°μ μ±μ°μΈμ | |
| ``` | |
| ```ini | |
| # κΈ°λ³Έ: λ‘컬 Ollama μ¬μ© | |
| PROVIDER="ollama" | |
| OLLAMA_BASE="http://localhost:11434" | |
| OLLAMA_EMBED_MODEL="nomic-embed-text" | |
| OLLAMA_CHAT_MODEL="llama3.1" | |
| # κ³΅ν΅ | |
| SUPABASE_URL="https://your-project-id.supabase.co" # Settings β General β Project ID | |
| SUPABASE_KEY="eyJ..." # Settings β API Keys β Legacy API keys β anon | |
| # μ ν: OpenAI μ¬μ© μ | |
| # PROVIDER="openai" | |
| # OPENAI_API_KEY="sk-..." | |
| # OPENAI_EMBEDDING_MODEL="text-embedding-3-small" | |
| # OPENAI_CHAT_MODEL="gpt-4o-mini" | |
| ``` | |
| --- | |
| ### 5. λ°μ΄ν°λ² μ΄μ€(Supabase) κ΅¬μ± | |
| Supabase λμ보λμ **SQL Editor**μμ μλ 쿼리λ₯Ό μ€ννμΈμ. | |
| <details> | |
| <summary><b>π SQL 쿼리 (κΈ°λ³Έ: Ollama 768μ°¨μ)</b></summary> | |
| <br> | |
| ```sql | |
| -- 1. λ²‘ν° νμ₯ νμ±ν | |
| CREATE EXTENSION IF NOT EXISTS vector; | |
| -- 2. λ¬Έμ μ²ν¬ ν μ΄λΈ μμ± | |
| CREATE TABLE IF NOT EXISTS document_chunks ( | |
| id BIGSERIAL PRIMARY KEY, | |
| document_id TEXT NOT NULL, | |
| content TEXT NOT NULL, | |
| embedding vector(768), -- nomic-embed-text μ°¨μ | |
| chunk_index INTEGER NOT NULL, | |
| metadata JSONB DEFAULT '{}', | |
| created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP | |
| ); | |
| -- 3. κ²μ ν¨μ(RPC) μμ± | |
| CREATE OR REPLACE FUNCTION match_documents ( | |
| query_embedding vector(768), | |
| match_count int, | |
| p_document_id text | |
| ) RETURNS TABLE ( | |
| id BIGINT, | |
| document_id TEXT, | |
| content TEXT, | |
| similarity FLOAT | |
| ) | |
| LANGUAGE plpgsql | |
| AS $$ | |
| BEGIN | |
| RETURN QUERY | |
| SELECT | |
| dc.id, | |
| dc.document_id, | |
| dc.content, | |
| 1 - (dc.embedding <=> query_embedding) as similarity | |
| FROM document_chunks AS dc | |
| WHERE dc.document_id = p_document_id | |
| ORDER BY dc.embedding <=> query_embedding | |
| LIMIT match_count; | |
| END; | |
| $$; | |
| -- 4. μΈλ±μ€ μμ± (HNSW λ°©μ) | |
| CREATE INDEX ON document_chunks USING hnsw (embedding vector_cosine_ops); | |
| ``` | |
| </details> | |
| <details> | |
| <summary><b>π λμ: OpenAI 1536μ°¨μ μ€ν€λ§</b></summary> | |
| <br> | |
| ```sql | |
| CREATE EXTENSION IF NOT EXISTS vector; | |
| CREATE TABLE IF NOT EXISTS document_chunks ( | |
| id BIGSERIAL PRIMARY KEY, | |
| document_id TEXT NOT NULL, | |
| content TEXT NOT NULL, | |
| embedding vector(1536), -- text-embedding-3-small μ°¨μ | |
| chunk_index INTEGER NOT NULL, | |
| metadata JSONB DEFAULT '{}', | |
| created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP | |
| ); | |
| CREATE OR REPLACE FUNCTION match_documents ( | |
| query_embedding vector(1536), | |
| match_count int, | |
| p_document_id text | |
| ) RETURNS TABLE ( | |
| id BIGINT, | |
| document_id TEXT, | |
| content TEXT, | |
| similarity FLOAT | |
| ) | |
| LANGUAGE plpgsql | |
| AS $$ | |
| BEGIN | |
| RETURN QUERY | |
| SELECT | |
| dc.id, | |
| dc.document_id, | |
| dc.content, | |
| 1 - (dc.embedding <=> query_embedding) as similarity | |
| FROM document_chunks AS dc | |
| WHERE dc.document_id = p_document_id | |
| ORDER BY dc.embedding <=> query_embedding | |
| LIMIT match_count; | |
| END; | |
| $$; | |
| CREATE INDEX ON document_chunks USING hnsw (embedding vector_cosine_ops); | |
| ``` | |
| </details> | |
| μ€ν ν Supabase λμ보λμ **Table Editor**μμ `document_chunks` ν μ΄λΈμ΄ μμ±λμλμ§ νμΈνμΈμ. | |
| --- | |
| ### 6. μλ² μ€ν | |
| ```bash | |
| pipenv run uvicorn app.main:app --reload | |
| ``` | |
| μ€ν ν λΈλΌμ°μ μμ `http://127.0.0.1:8000` μ μμΌλ‘ UIλ₯Ό νμΈνμΈμ. | |
| --- | |
| ### 7. μ½λ ν¬λ§€ν° μ€μ (μ ν) | |
| μ»€λ° μ μλμΌλ‘ μ½λ μ€νμΌμ μ 리νλ `black` ν¬λ§€ν°λ₯Ό pre-commit hookμΌλ‘ μ€μ ν μ μμ΅λλ€. | |
| ```bash | |
| # dev μμ‘΄μ± μ€μΉ (pre-commit, black ν¬ν¨) | |
| pipenv install --dev | |
| # git hook λ±λ‘ (μ΅μ΄ 1ν) | |
| pipenv run pre-commit install | |
| ``` | |
| λ±λ‘ νμλ `git commit` μ `black`μ΄ μλ μ€νλ©λλ€. μ½λ μ€νμΌμ΄ μμ λ κ²½μ° νμΌμ λ€μ `git add` ν λ€ μ»€λ°νλ©΄ λ©λλ€. | |
| > μ΄ λ¨κ³λ₯Ό 건λλ°μ΄λ νλ‘μ νΈ μ€νμλ μν₯μ΄ μμ΅λλ€. | |
| <br> | |
| <br> | |
| ## ποΈ 2μ£Ό μ§ν ꡬ쑰 | |
| νκ²½ μ€μ μ μλ£ν ν, μλ μμλλ‘ λ¨κ³λ³ λ¬Έμλ₯Ό λ°λΌ μ§ννμΈμ. | |
| - μ£Όλ§(ν /μΌ, 9:00-18:00): νμ΄νλ‘κ·Έλλ°μΌλ‘ ꡬν/κ²μ¦ | |
| - νμΌ: μ¬μ μ€λΉ, μ‘°μ¬, μ€ν κΈ°λ‘, μμ¬κ²°μ μ 리 | |
| | λ¨κ³ | κΈ°κ° | λ¬Έμ | | |
| | ------------- | ------------------ | ------------------------------------------------------------ | | |
| | **Prework** | Week 1 μ νμΌ | [`docs/01-prework/README.md`](./docs/01-prework/README.md) | | |
| | **Weekend-1** | Week 1 μ£Όλ§ | [`docs/02-weekend1/README.md`](./docs/02-weekend1/README.md) | | |
| | **Midweek** | Week 1~2 μ¬μ΄ νμΌ | [`docs/03-midweek/README.md`](./docs/03-midweek/README.md) | | |
| | **Weekend-2** | Week 2 μ£Όλ§ | [`docs/04-weekend2/README.md`](./docs/04-weekend2/README.md) | | |
| <br> | |
| <br> | |
| ## π μ°Έκ³ νμ΅ μλ£ | |
| ### νλ‘μ νΈ λ΄λΆ λ¬Έμ | |
| - **[RAG μμ€ν κΈ°μ μ¬μΈ΅ λΆμ](./docs/00-shared/reference.md)**: RAG μν€ν μ²μ κΈ°μ μ€ν μ λ°. **νλ νμΈμ.** | |
| - **[λ°°ν¬ κ°μ΄λ](./docs/04-weekend2/deployment.md)**: λ°°ν¬ μμΈ κ°μ΄λ(μ¬ν/μ°Έκ³ )μ λλ€. | |
| ### μΈλΆ μ°Έκ³ μλ£ | |
| - **[What is RAG? (IBM)](https://www.ibm.com/topics/retrieval-augmented-generation)**: RAG κ°λ μ κ·Έλ¦Όκ³Ό ν¨κ» μ€λͺ νλ μ λ¬Έ μλ£μ λλ€. | |
| - **[RAG from Scratch (LangChain YouTube)](https://www.youtube.com/playlist?list=PLfaIDFEXuae2LXbO1_PKyVJiQ23ZztA0x)**: LangChain 곡μ RAG ꡬν μμ μ리μ¦μ λλ€ (μμ΄). | |
| - **[pgvector GitHub](https://github.com/pgvector/pgvector)**: `<=>` μ°μ°μμ HNSW μΈλ±μ€ μλ λ°©μμ νμΈνμΈμ. | |
| - **[Supabase Vector κ°μ΄λ](https://supabase.com/docs/guides/ai)**: PostgreSQLμ λ²‘ν° λ°μ΄ν°λ² μ΄μ€λ‘ μ¬μ©νλ λ°©λ²μ λλ€. | |
| - **[LangChain RecursiveCharacterTextSplitter](https://reference.langchain.com/python/langchain-text-splitters/character/RecursiveCharacterTextSplitter)**: ν μ€νΈ μ²νΉ μ λ΅μ λν Python 곡μ λ νΌλ°μ€μ λλ€. | |
| - **[OpenAI Embeddings API](https://platform.openai.com/docs/guides/embeddings)**: ν μ€νΈλ₯Ό μ«μλ‘ λ³ννλ μ리λ₯Ό μ΄ν΄ν΄μΌ ν©λλ€. | |
| - **[Vector Similarity Explained (Pinecone)](https://www.pinecone.io/learn/vector-similarity/)**: μ μ¬λ μΈ‘μ λ°©μ λΉκ΅ μλ£μ λλ€. | |